Clean up data structures on sherlock/langcog server

brialorelle / headcam-algo

Repository for development/validation of algorithms to extract automated information from headcam data

2 stars 1 forks source link

Clean up data structures on sherlock/langcog server #1

Open brialorelle opened 4 years ago

brialorelle commented 4 years ago

Clean up scratch/home spaces on sherlock;

[x] Move franchack files to xs-face directory
[x] Get rid of old/duplicate folders and files
[x] Make clear which videos are the newer version of dataset linked to Databrary meta data (add Readme, rename folders)
[x] Duplicate jsons from lab server to sherlock or rerun OP as needed so we have all raw
[x] Clean copy of all files on langcog server with README
[x] Upload clean script for getting databrary videos to langcog server and repo
[x] Backup jsons from Y on langcog server

ketan0 commented 4 years ago

moved all openpose outputs to $GROUP_SCRATCH/Home_Headcam_new/outputs.

Raw jsons (tiny files) will appear in $GROUP_SCRATCH/Home_Headcam_new/outputs/openpose_raw_json.

Condensed jsons (video-level dataframes) will appear in $GROUP_SCRATCH/Home_Headcam_new/outputs/openpose_condensed.

Also, got rid of stuff inside Home_Headcam directory, as it only contained the partially-transferred OP outputs from langcog server from a few days ago (before we stopped doing that, deeming it too slow)

(Cannot remove Home_Headcam due to permissions)

ketan0 commented 4 years ago

Also, the clean download script is here. Can either be run as a script, or import the functions and use in a jupyter notebook.