Jupyter notebook to create plots and load the whole dataset in a time-aligned pandas DataFrame

[luisqtr]

Hi.

Thanks for putting this interesting dataset in place and making it public.

This PR is a a suggestion to include some plots that I generated. They may complement the description of the dataset for researchers who want to work with this dataset in the future. The folder CEAP-plots contains PNG images visualizing the individual files from the folders Annotations, Behavior, Physio and their subsequent variants Raw, Transformed, Frame.

Currently working on a python class to load the JSON files to structured pandas DataFrame. If you think it may be useful, I could create another PR when the class is finished to include these scripts too.

[TongXueBIT]

Hi Luis,

Thank you very much for your attention and proposal for a pull request.

Indeed these images can help make the data more intuitive, however showing too many plots may appear as a large generated data dump, especially since these plots are generated from the raw data (without data cleaning and time alignment).

We propose to include the script to generate these plots instead, as well as showing an example of each plot, in the CEAP-360VR-Dataset/CEAP-360VR/6_Scripts/XXX directory. Would you be able to request another pull request? We will be very glad to merge this request on GitHub.

Thank you again for your contributions.

Best wishes, Tong Xue (on behalf of co-authors)

[luisqtr]

Thanks for your response, Tong.

It took a while to go through all the Json files to decide on which variables were relevant for my problem and at which preprocessing level. That's why I thought the plots could help but you are totally right, it's a large data dump (~150MB) and it doesn't provide any meaningful insights.

I am still fixing some details on the Python class that loads the Json files into pandas dfs to facilitate the data analysis. Surely, I will update my PR with the code and a Jupyter example when it's ready. 😁

[abdoelali]

Thank you very much Luis, we look forward to incorporating your code and Jupyter notebook examples.

Best wishes, Abdallah

On Tue, 9 Aug 2022 at 14:30, Luis Quintero @.***> wrote:

Thanks for your response, Tong.

It took a while to go through all the Json files to decide on which variables were relevant for my problem and at which preprocessing level. That's why I thought the plots could help but you are totally right, it's a large data dump (~150MB) and it doesn't provide any meaningful insights.

I am still fixing some details on the Python class that loads the Json files into pandas dfs to facilitate the data analysis. Surely, I will update my PR with the code and a Jupyter example when it's ready. 😁

— Reply to this email directly, view it on GitHub https://github.com/cwi-dis/CEAP-360VR-Dataset/pull/1#issuecomment-1209320212, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSR7CM5EPEHWK7X2VBCLX3VYJFPFANCNFSM55MJAKYA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Abdallah El Ali https://abdoelali.com

[luisqtr]

Hi again,

I worked a bit more on the dataset:

• Removed the initial folder with plots and added a code to generate them instead.
• Added the DatasetCEAP class to load the files as pd.DataFrame depending on the participant, data type (Annotations, Behavior, Physio), and processing level (Raw, Transformed, Frame),
• The jupyter notebook in 6_Scripts/4_... shows how to use the class, and how to combine the whole dataset in a single dataframe with synced timestamps @30Hz, without missing values and with target class labels for classification tasks.

Hope it helps encouraging the use of the nice dataset you put together.

/Luis