Data Capture: Produce standardized CSV files

[bastianilso]

The R Shiny JS data capture will be part of an ecosystem, and have to adhere to some standardized ways in which we collect data in the HMI lab. This means:

• CSV files must be generated the same way, as our other data capture systems does it (e.g. LoggingManager for Unity). The convention is to use semi-colon ; as delimiter.
• CSV files must as a minimum include columns named: SessionID (a unique ID for the session), Timestamp (in the format yyyy-mm-dd hh:mm:ss.mmm ), Email (by default with the value anonymous), Framecount (default value: NA). These columns are assumed to be existing by the R Logging Loader.
• CSV files must be end with the extension *.csv.
• By convention we encode CSV filename as: "log" + <timestamp> + <datatype>. For example log2022-03-29 11-54-12.4138Event.csv. There are three established datatypes:
  • Event datatype: This is data created based on events (fx mouseclick, mousehover etc).
  • Meta datatype. This is metadata (date, configuration variables,..)
  • Sample datatype. This is data which is continuously collected x times per second. (fx mouse movement).

More requirements may appear as we iterate on it.

Testing whether the CSV files adhere to the requirements can be done by placing the CSV file in a folder called e.g. data and downloading the R Logging Loader and load the data from there in an R script:

source("utils/loadrawdata.R")
D <- LoadFromDirectory("data")

If it produces a datasat called D, then it succeeded:

Otherwise, feel free to paste an error into this issue and we can try to take a look what's going on.

EDIT: Just to be clear, @bilal-62210 I would like you to address these parts in a separate pull request, than the current one. We should merge the current one. The convention is that a pull request should address one problem/issue.

[bilal-62210]

@bastianilso when i run the r loading order the result is this : And this is my csv file: log2022-9-28 15-16-30.479Event.csv Can you help me please?

[hendrikknoche]

OK, this looks like basic debugging. Have you verified that the column 'Timestamp' exists in the file? what about 'SessionID'

[bilal-62210]

@hendrikknoche As you can see below: 'Timestamp' and 'SessionID' exist, this is Why i don't understand. I've used 'loadFromDirectory' following bastian's instructions and i think there is no read.csv function in the R logging loader :

[bastianilso]

@bilal-62210 you should not need to use absolute paths to load the data. why not use relative paths like i showed in the example? you can use getwd() in the R terminal to see what folder is the current working directory.

[hendrikknoche]

@hendrikknoche As you can see below: 'Timestamp' and 'SessionID' exist, this is Why i don't understand. I've used 'loadFromDirectory' following bastian's instructions and i think there is no read.csv function in the R logging loader :

OK, this is difficult to reproduce for me because the code cannot work on my end given that there's no utils folder in the repo currently.

the tidyverse library you need to load in R like this: library(tidyverse)

have a look here about differences between source() and library() https://stackoverflow.com/questions/55122477/what-is-the-difference-between-library-require-and-source-in-r

[hendrikknoche]

the`

@hendrikknoche As you can see below: 'Timestamp' and 'SessionID' exist, this is Why i don't understand. I've used 'loadFromDirectory' following bastian's instructions and i think there is no read.csv function in the R logging loader :

OK, this is difficult to reproduce for me because the code cannot work on my end given that there's no utils folder in the repo currently.

the tidyverse library you need to load in R like this: library(tidyverse)

have a look here about differences between source() and library() https://stackoverflow.com/questions/55122477/what-is-the-difference-between-library-require-and-source-in-r

the source("utils/..data.R") executes all the code inside the sourced files and in this case makes the functions inside available. So you can call 'loadFromDirectory'.

[bilal-62210]

@bastianilso this is the csv file: log2022-9-28 15-16-30.479Event.csv

[bilal-62210]

@bastianilso It's good for the stardardized csv file. It works so i think it's good for this issue.

[bastianilso]

great @bilal-62210, good to see. you can either pull request this or push directly to the repo, up to you. In these initial stages where we prototype code pushing to master is fine for me.