search

spine-generic / data-multi-subject_DO-NOT-USE

Multi-subject data for the Spine Generic project
0 stars 1 forks source link

Files larger than 100MB #2

Closed jcohenadad closed 4 years ago

jcohenadad commented 4 years ago

this could explain #1:

julien-macbook:~/data/spineGeneric_multi $ ll sub-beijingGE04/anat/
total 260128
drwxr-xr-x  8 julien  staff        256 30 Jun 10:15 .
drwxr-xr-x  4 julien  staff        128 30 Jun 10:02 ..
-rw-r--r--  1 julien  staff       1451 19 Jun 09:47 sub-beijingGE04_T1w.json
-rw-r--r--  1 julien  staff  122306489 19 Jun 09:47 sub-beijingGE04_T1w.nii.gz   → PROBLEM
-rw-r--r--  1 julien  staff       1771 19 Jun 09:47 sub-beijingGE04_T2star.json
-rw-r--r--  1 julien  staff    4821177 19 Jun 09:47 sub-beijingGE04_T2star.nii.gz
-rw-r--r--  1 julien  staff       1483 19 Jun 09:47 sub-beijingGE04_T2w.json
-rw-r--r--  1 julien  staff    4854665 19 Jun 09:47 sub-beijingGE04_T2w.nii.gz
jcohenadad commented 4 years ago

question: what should we do?

jcohenadad commented 4 years ago

there are only 4 problematic files:

julien-macbook:~/data/spineGeneric_multi $ find sub* -type f -print0 | xargs -0 ls -la | awk '{print int($5/1000) " KB\t" $9}' | sort -n -r -k1
122306 KB   sub-beijingGE04/anat/sub-beijingGE04_T1w.nii.gz
119980 KB   sub-beijingGE03/anat/sub-beijingGE03_T1w.nii.gz
119521 KB   sub-beijingGE02/anat/sub-beijingGE02_T1w.nii.gz
118446 KB   sub-beijingGE01/anat/sub-beijingGE01_T1w.nii.gz
80034 KB    sub-brnoUhb02/anat/sub-brnoUhb02_T1w.nii.gz
79269 KB    sub-brnoUhb05/anat/sub-brnoUhb05_T1w.nii.gz
kousu commented 4 years ago

This is what git-lfs is meant for.

git lfs install
git lfs track beijingGE*/anat/*.nii.gz
git push
kousu commented 4 years ago

Of course we're going to run off the limits in git-lfs quickly unless we pay Github for the privilege.

jcohenadad commented 4 years ago

i think i'll simply downsample these 4 data

jcohenadad commented 4 years ago

i'm also uploading by smaller chunks

jcohenadad commented 4 years ago

problematic files were downsampled-- problem solved