mariehbourget
commented
3 years ago @kousu, @alexfoias, I would need some help with the data_axondeepseg_tem on git-annex server. Thanks!
Closed mariehbourget closed 3 years ago
mariehbourget
commented
3 years ago @kousu, @alexfoias, I would need some help with the data_axondeepseg_tem on git-annex server. Thanks!
kousu
commented
3 years ago @mariehbourget it sounds to me like the git annex get . step failed. Can you try it again and send some screenshots? That should help me diagnose it! Thanks.
mariehbourget
commented
3 years ago I cloned and downloaded the dataset again from scratch. All checksums seem ok (terminal output below) So maybe an error when the dataset was first uploaded?
All the image files in the folders derivatives/sub-nyuMouseXX/ ending with _seg-axon-manual.png are 1Kb but they should be around ~50-100Kb like the others:
Thank you for your help!
kousu
commented
3 years ago T'as raison. Something very strange is going on here.
Most of the .png files are fine, but some are not:
nguenther@data:~/datasets/data_axondeepseg_tem$ find derivatives -name "*.png" -exec file {} \;
derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0005_TEM_seg-myelin-manual.png: PNG image data, 3762 x 2286, 8-bit grayscale, non-interlaced
derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0002_TEM_seg-axonmyelin-manual.png: PNG image data, 3762 x 2286, 8-bit grayscale, non-interlaced
derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0004_TEM_seg-myelin-manual.png: PNG image data, 3762 x 2286, 8-bit grayscale, non-interlaced
derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0007_TEM_seg-myelin-manual.png: PNG image data, 3762 x 2286, 8-bit grayscale, non-interlaced
derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0003_TEM_seg-axonmyelin-manual.png: PNG image data, 3762 x 2286, 8-bit grayscale, non-interlaced
derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0008_TEM_seg-axonmyelin-manual.png: PNG image data, 3762 x 2286, 8-bit grayscale, non-interlaced
derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0007_TEM_seg-axonmyelin-manual.png: PNG image data, 3762 x 2286, 8-bit grayscale, non-interlaced
derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0007_TEM_seg-axon-manual.png: ASCII text, with no line terminators
derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0006_TEM_seg-myelin-manual.png: PNG image data, 3762 x 2286, 8-bit grayscale, non-interlaced
derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0001_TEM_seg-axonmyelin-manual.png: PNG image data, 3762 x 2286, 8-bit grayscale, non-interlaced
derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0006_TEM_seg-axonmyelin-manual.png: PNG image data, 3762 x 2286, 8-bit grayscale, non-interlaced
derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0003_TEM_seg-myelin-manual.png: PNG image data, 3762 x 2286, 8-bit grayscale, non-interlaced
derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0005_TEM_seg-axonmyelin-manual.png: PNG image data, 3762 x 2286, 8-bit grayscale, non-interlaced
...It seems to be limited to the "seg-axon-manual" files:
nguenther@data:~/datasets/data_axondeepseg_tem$ find . -name "*.png" -exec file {} \; | grep ASCII
./derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0007_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0008_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse15/microscopy/sub-nyuMouse15_sample-0001_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse32/microscopy/sub-nyuMouse32_sample-0001_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse32/microscopy/sub-nyuMouse32_sample-0007_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse32/microscopy/sub-nyuMouse32_sample-0009_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse32/microscopy/sub-nyuMouse32_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse32/microscopy/sub-nyuMouse32_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse32/microscopy/sub-nyuMouse32_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse32/microscopy/sub-nyuMouse32_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse32/microscopy/sub-nyuMouse32_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse30/microscopy/sub-nyuMouse30_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse30/microscopy/sub-nyuMouse30_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse30/microscopy/sub-nyuMouse30_sample-0001_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse30/microscopy/sub-nyuMouse30_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse30/microscopy/sub-nyuMouse30_sample-0009_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse30/microscopy/sub-nyuMouse30_sample-0008_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse30/microscopy/sub-nyuMouse30_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse30/microscopy/sub-nyuMouse30_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse10/microscopy/sub-nyuMouse10_sample-0001_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse10/microscopy/sub-nyuMouse10_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse10/microscopy/sub-nyuMouse10_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse10/microscopy/sub-nyuMouse10_sample-0007_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse10/microscopy/sub-nyuMouse10_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse10/microscopy/sub-nyuMouse10_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse10/microscopy/sub-nyuMouse10_sample-0008_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse10/microscopy/sub-nyuMouse10_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse35/microscopy/sub-nyuMouse35_sample-0007_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse35/microscopy/sub-nyuMouse35_sample-0009_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse35/microscopy/sub-nyuMouse35_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse35/microscopy/sub-nyuMouse35_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse35/microscopy/sub-nyuMouse35_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse35/microscopy/sub-nyuMouse35_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse35/microscopy/sub-nyuMouse35_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse35/microscopy/sub-nyuMouse35_sample-0001_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse14/microscopy/sub-nyuMouse14_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse14/microscopy/sub-nyuMouse14_sample-0008_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse14/microscopy/sub-nyuMouse14_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse14/microscopy/sub-nyuMouse14_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse14/microscopy/sub-nyuMouse14_sample-0001_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse14/microscopy/sub-nyuMouse14_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse14/microscopy/sub-nyuMouse14_sample-0007_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse14/microscopy/sub-nyuMouse14_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse28/microscopy/sub-nyuMouse28_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse28/microscopy/sub-nyuMouse28_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse28/microscopy/sub-nyuMouse28_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse28/microscopy/sub-nyuMouse28_sample-0008_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse28/microscopy/sub-nyuMouse28_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse28/microscopy/sub-nyuMouse28_sample-0009_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse28/microscopy/sub-nyuMouse28_sample-0001_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse28/microscopy/sub-nyuMouse28_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse13/microscopy/sub-nyuMouse13_sample-0001_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse13/microscopy/sub-nyuMouse13_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse13/microscopy/sub-nyuMouse13_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse13/microscopy/sub-nyuMouse13_sample-0008_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse13/microscopy/sub-nyuMouse13_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse13/microscopy/sub-nyuMouse13_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse13/microscopy/sub-nyuMouse13_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse13/microscopy/sub-nyuMouse13_sample-0007_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse33/microscopy/sub-nyuMouse33_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse33/microscopy/sub-nyuMouse33_sample-0009_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse33/microscopy/sub-nyuMouse33_sample-0001_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse33/microscopy/sub-nyuMouse33_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse33/microscopy/sub-nyuMouse33_sample-0008_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse33/microscopy/sub-nyuMouse33_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse33/microscopy/sub-nyuMouse33_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse33/microscopy/sub-nyuMouse33_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0004_TEM_seg-axon-manual.png: ASCII text
./derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0001_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0007_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0008_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse09/microscopy/sub-nyuMouse09_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse09/microscopy/sub-nyuMouse09_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse09/microscopy/sub-nyuMouse09_sample-0007_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse09/microscopy/sub-nyuMouse09_sample-0008_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse09/microscopy/sub-nyuMouse09_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse09/microscopy/sub-nyuMouse09_sample-0001_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse09/microscopy/sub-nyuMouse09_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse09/microscopy/sub-nyuMouse09_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse34/microscopy/sub-nyuMouse34_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse34/microscopy/sub-nyuMouse34_sample-0009_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse34/microscopy/sub-nyuMouse34_sample-0001_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse34/microscopy/sub-nyuMouse34_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse34/microscopy/sub-nyuMouse34_sample-0008_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse34/microscopy/sub-nyuMouse34_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse34/microscopy/sub-nyuMouse34_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse34/microscopy/sub-nyuMouse34_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse12/microscopy/sub-nyuMouse12_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse12/microscopy/sub-nyuMouse12_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse12/microscopy/sub-nyuMouse12_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse12/microscopy/sub-nyuMouse12_sample-0008_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse12/microscopy/sub-nyuMouse12_sample-0009_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse12/microscopy/sub-nyuMouse12_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse12/microscopy/sub-nyuMouse12_sample-0010_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse12/microscopy/sub-nyuMouse12_sample-0011_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse29/microscopy/sub-nyuMouse29_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse29/microscopy/sub-nyuMouse29_sample-0001_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse29/microscopy/sub-nyuMouse29_sample-0009_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse29/microscopy/sub-nyuMouse29_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse29/microscopy/sub-nyuMouse29_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse29/microscopy/sub-nyuMouse29_sample-0007_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse29/microscopy/sub-nyuMouse29_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse29/microscopy/sub-nyuMouse29_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse07/microscopy/sub-nyuMouse07_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse07/microscopy/sub-nyuMouse07_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse07/microscopy/sub-nyuMouse07_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse07/microscopy/sub-nyuMouse07_sample-0008_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse07/microscopy/sub-nyuMouse07_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse07/microscopy/sub-nyuMouse07_sample-0001_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse07/microscopy/sub-nyuMouse07_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse07/microscopy/sub-nyuMouse07_sample-0007_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse36/microscopy/sub-nyuMouse36_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse36/microscopy/sub-nyuMouse36_sample-0008_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse36/microscopy/sub-nyuMouse36_sample-0007_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse36/microscopy/sub-nyuMouse36_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse36/microscopy/sub-nyuMouse36_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse36/microscopy/sub-nyuMouse36_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse36/microscopy/sub-nyuMouse36_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse36/microscopy/sub-nyuMouse36_sample-0001_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse31/microscopy/sub-nyuMouse31_sample-0001_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse31/microscopy/sub-nyuMouse31_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse31/microscopy/sub-nyuMouse31_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse31/microscopy/sub-nyuMouse31_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse31/microscopy/sub-nyuMouse31_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse31/microscopy/sub-nyuMouse31_sample-0007_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse31/microscopy/sub-nyuMouse31_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse31/microscopy/sub-nyuMouse31_sample-0009_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse11/microscopy/sub-nyuMouse11_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse11/microscopy/sub-nyuMouse11_sample-0007_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse11/microscopy/sub-nyuMouse11_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse11/microscopy/sub-nyuMouse11_sample-0008_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse11/microscopy/sub-nyuMouse11_sample-0001_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse11/microscopy/sub-nyuMouse11_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse11/microscopy/sub-nyuMouse11_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse11/microscopy/sub-nyuMouse11_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse27/microscopy/sub-nyuMouse27_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse27/microscopy/sub-nyuMouse27_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse27/microscopy/sub-nyuMouse27_sample-0007_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse27/microscopy/sub-nyuMouse27_sample-0008_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse27/microscopy/sub-nyuMouse27_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse27/microscopy/sub-nyuMouse27_sample-0001_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse27/microscopy/sub-nyuMouse27_sample-0002_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse27/microscopy/sub-nyuMouse27_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse25/microscopy/sub-nyuMouse25_sample-0001_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse25/microscopy/sub-nyuMouse25_sample-0003_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse25/microscopy/sub-nyuMouse25_sample-0004_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse25/microscopy/sub-nyuMouse25_sample-0008_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse25/microscopy/sub-nyuMouse25_sample-0006_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./derivatives/labels/sub-nyuMouse25/microscopy/sub-nyuMouse25_sample-0005_TEM_seg-axon-manual.png: ASCII text, with no line terminators
./.git/annex/objects/x8/k4/SHA256E-s8--e4f04418c61a764aef99d3e12b30db8a86b9955878af458cba46b16c6f18bec4.png/SHA256E-s8--e4f04418c61a764aef99d3e12b30db8a86b9955878af458cba46b16c6f18bec4.png: ASCII text
./.git/annex/objects/jJ/X1/SHA256E-s7--7ef1f17b4bc27508b960c25d83ac2c205705ef213b78e2490f7aa97988fb7c9f.png/SHA256E-s7--7ef1f17b4bc27508b960c25d83ac2c205705ef213b78e2490f7aa97988fb7c9f.png: ASCII text, with no line terminatorsmoreover their contents are all the same:
nguenther@data:~/datasets/data_axondeepseg_tem$ find derivatives -name "*seg-axon-manual.png" -exec cat {} \; -exec echo \;
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236
0.00236These files were copied in via
and
so they must have been incorrect in the uncurated dataset too.
mariehbourget
commented
3 years ago 0.00236 I recognize this value! (I should have check the content in text before, sorry about that).
This is the content of a file called pixel_size_in_micrometers.txt that are present in our ADS dataset but not in the BIDS one. Thanks, I'll check what is happening in the uncurated dataset.
kousu
commented
3 years ago Yes I think that's right. I am looking at the uncurated dataset, which I believe is ~/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset: it seems that every subject has a pixel_size_in_micrometers containing that value (forgive them all being run together, they are missing \ns):
u108545@romane:~/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset$ cat */pixel_size_in_micrometer.txt
0.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.002360.00236But why did that file end up in the "seg-axon-manual" files? I reviewed the curation script and that bug didn't jump out at me. I just reread it and I still don't see what's going on.
The source dataset seems to contain the correct segmentations, though, at least, so we should be able to repair it:
u108545@romane:~/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset$ file 20160718_nyu_mouse_25_0001/mask_seg-axon-manual.png
20160718_nyu_mouse_25_0001/mask_seg-axon-manual.png: PNG image data, 3762 x 2286, 8-bit grayscale, non-interlacedYes I think that's right. I am looking at the uncurated dataset, which I believe is
~/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset:
That is indeed the uncurated dataset. There is one more image which is in another folder, see comment here in the original PR: https://github.com/neuropoly/data-management/pull/103#issue-944829263
But why did that file end up in the "seg-axon-manual" files? I reviewed the curation script and that bug didn't jump out at me. I just reread it and I still don't see what's going on.
Same for me, I don't see how that could happen with the script!
The source dataset seems to contain the correct segmentations, though, at least, so we should be able to repair it:
Yes, I checked a few and they seem OK.
kousu
commented
3 years ago I wonder if maybe all the files got mixed up: maybe the segmentations are the masks and the masks are the myelin and the myelin are the segmentations? If we re-run the curation script into the git repo we can
a. see if it causes any diffs b. if it doesn't, hunt down the bug, fix it, and re-run it to cause those diffs
mariehbourget
commented
3 years ago I wonder if maybe all the files got mixed up:
Good point, I checked a few other files. From what I see, there are no mixup for the other segmentations, "seg-axonmyelin-manual" and "seg-myelin-manual" are correct.
alexfoias
commented
3 years ago I discussed with @kousu about the issue. I will do some investigations this afternoon.
alexfoias
commented
3 years ago Fixed in af/fix_der.
@kousu or @mariehbourget would you me able to check if the problem is still there?
mariehbourget
commented
3 years ago Fixed in
af/fix_der.@kousu or @mariehbourget would you me able to check if the problem is still there?
I checked the branch and all files look good. Thank you @alexfoias!
kousu
commented
3 years ago @alexfoias, I need a clarification. I found two copies of this dataset, at:
/mnt/duke/projects/axondeepseg/raw_data/data_TEM/osf//mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/The latter seems to be a manually processed derivative of the former -- it has {mask_seg-axon-manual.png, mask_seg-myelin-manual.png}. However it is missing 20160718_nyu_mouse_26_0004/. You mentioned you handled this (https://github.com/neuropoly/data-management/pull/103#issue-944829263) by filling in the missing file from duke/projects/axondeepseg/20210708_default_datasets_mask_curation/default_TEM_model/data_test; Why did you need to do that?
The fill-in data-test folder has a README, whose first line is
20160718_nyu_mouse_26_0004
which makes it sound like it's got the correct files. However, unlike all the other samples, its image.png doesn't match the first osf/ copy:
u108545@joplin:~/data-management$ diff -ru /mnt/duke/projects/axondeepseg/raw_data/data_TEM/osf/20160718_nyu_mouse_26_0001/image.png /mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/20160718_nyu_mouse_26_0001/image.png
u108545@joplin:~/data-management$ diff -ru /mnt/duke/projects/axondeepseg/raw_data/data_TEM/osf/20160718_nyu_mouse_26_0002/image.png /mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/20160718_nyu_mouse_26_0002/image.png
u108545@joplin:~/data-management$ diff -ru /mnt/duke/projects/axondeepseg/raw_data/data_TEM/osf/20160718_nyu_mouse_26_0003/image.png /mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/20160718_nyu_mouse_26_0003/image.png
u108545@joplin:~/data-management$ diff -ru /mnt/duke/projects/axondeepseg/raw_data/data_TEM/osf/20160718_nyu_mouse_26_0004/image.png /mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/20160718_nyu_mouse_26_0004/image.png
diff: /mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/20160718_nyu_mouse_26_0004/image.png: No such file or directory
u108545@joplin:~/data-management$ diff -ru /mnt/duke/projects/axondeepseg/raw_data/data_TEM/osf/20160718_nyu_mouse_26_0004/image.png /mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/../default_TEM_model/data_test/image.png
Binary files /mnt/duke/projects/axondeepseg/raw_data/data_TEM/osf/20160718_nyu_mouse_26_0004/image.png and /mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/../default_TEM_model/data_test/image.png differ
u108545@joplin:~/data-management$ diff -ru /mnt/duke/projects/axondeepseg/raw_data/data_TEM/osf/20160718_nyu_mouse_26_0005/image.png /mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/20160718_nyu_mouse_26_0005/image.png
u108545@joplin:~/data-management$ diff -ru /mnt/duke/projects/axondeepseg/raw_data/data_TEM/osf/20160718_nyu_mouse_26_0006/image.png /mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/20160718_nyu_mouse_26_0006/image.png
u108545@joplin:~/data-management$ diff -ru /mnt/duke/projects/axondeepseg/raw_data/data_TEM/osf/20160718_nyu_mouse_26_0007/image.png /mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/20160718_nyu_mouse_26_0007/image.png
u108545@joplin:~/data-management$ diff -ru /mnt/duke/projects/axondeepseg/raw_data/data_TEM/osf/20160718_nyu_mouse_26_0008/image.png /mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/20160718_nyu_mouse_26_0008/image.png What does match is its mask.png:
u108545@joplin:~/data-management$ diff -ru /mnt/duke/projects/axondeepseg/raw_data/data_TEM/osf/20160718_nyu_mouse_26_0004/mask.png /mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/../default_TEM_model/data_test/mask.png
u108545@joplin:~/data-management$So what's going on with this sample? Should it be removed from the dataset?
kousu
commented
3 years ago I checked out the source dataset that's mentioned in
With that I was able to compare
/mnt/duke/histology/mouse/20160718_nyu_mouse/20160718_nyu_mouse_26/20160718_nyu_mouse_26_0004/20160718_nyu_mouse_26_0004.tif/mnt/duke/projects/axondeepseg/raw_data/data_TEM/osf/20160718_nyu_mouse_26_0004/image.png/mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/default_TEM_model/data_test/image.pngI'm not going to upload them here (I'm not sure if they're open data or not) but they are all the same image; the latter two are a crop of the .tif, and between the latter two it looks to me like they are in fact identical except that the osf/ copy has a 1-px wide border around it.
osf/ copy is 3762x2286 pxdata_test/ copy is 3760x2284 pxkousu
commented
3 years ago I took a look at the rest of the data and it seems clear: 3762x2286 is the size we want:
I propose: we do:
cp -r /mnt/joplin/mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/default_TEM_model/data_test /mnt/joplin/mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/20160718_nyu_mouse_26_0004/
cp /mnt/duke/projects/axondeepseg/raw_data/data_TEM/osf/20160718_nyu_mouse_26_0004/image.png /mnt/joplin/mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/20160718_nyu_mouse_26_0004/image.pngBut if there's a better more original source for 26/0004 point me at it.
Once that's done #127 will cleanly reproduce the curated dataset.
mariehbourget
commented
3 years ago @kousu, thank you for pointing that out, I have an explanation that will help.
The /mnt/duke/projects/axondeepseg/raw_data/data_TEM/osf/ dataset is matching the current ADS TEM dataset from 2019 on OSF here. However, on OSF, the image and mask 20160718_nyu_mouse_26_0004 were removed because of discrepancies in the size, as stated in the README.
The /mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/ is the same dataset with the addition of mask_seg-axon-manual.png and mask_seg-myelin-manual.png that we did in preparation for the bidsification.
The size of the image 20160718_nyu_mouse_26_0004 was corrected to match the size of the mask (in 2020), and add to the TEM model as a "data_test" file separated from the original dataset. On OSF, these image and mask are in default_model_TEM/data_test/image.png and mask.png here. This is why they were treated separately from the rest of the dataset.
In short, it is normal that the 20160718_nyu_mouse_26_0004 image and mask are 3760x2284 instead of 3762x2286 and we want to have them included in the bidsified dataset as is.
kousu
commented
3 years ago Thanks for chiming in @mariehbourget; two questions:
mask.png to have the same dimensions as everything else? It's the only such mask:/mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/ at least? I am attempting to make the script in #127 behave reliably, without as much manual steps to remember and right now running it on that folder produces a diff against what's on the git server.mariehbourget
commented
3 years ago 1: Wouldn't it have been simpler to correct
mask.pngto have the same dimensions as everything else? It's the only such mask
In general, it is easier/faster to remove lines from an image file rather then adding them. This is was we decided to do on this specific case. The size of the images between them are not very important, as long as each corresponding image and mask are the same.
2: If we do keep the mismatched size, can we at least copy it back into
/mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/at least? I am attemtping to make the script in #127 behave reliably, without as much manual steps to remember and right now running it on that folder produces adiffagainst what's on thegitserver.
Yes sure, that is a good point, you can copy the data_test in TEM_dataset as 20160718_nyu_mouse_26_0004, no problem.
kousu
commented
3 years ago 1: Wouldn't it have been simpler to correct
mask.pngto have the same dimensions as everything else? It's the only such maskIn general, it is easier/faster to remove lines from an image file rather then adding them. This is was we decided to do on this specific case. The size of the images between them are not very important, as long as each corresponding image and mask are the same.
Would you be against me taking care of resizing the three masks to the larger size? I can do it in MSPaint's Canvas Size tool.
mariehbourget
commented
3 years ago 1: Wouldn't it have been simpler to correct
mask.pngto have the same dimensions as everything else? It's the only such maskIn general, it is easier/faster to remove lines from an image file rather then adding them. This is was we decided to do on this specific case. The size of the images between them are not very important, as long as each corresponding image and mask are the same.
Would you be against me taking care of resizing the three masks to the larger size? I can do it in MSPaint's Canvas Size tool.
I'm not against it, but adding a line on each side of the mask would also require to label (black, gray or white) these lines to match the image. This is mainly why we chose to crop the image instead.
alexfoias
commented
3 years ago @kousu Here is the answer for https://github.com/neuropoly/data-management/issues/123#issuecomment-936619079 :
Please see the issue that was used to create the initial dataset. In there you will find information on what data were curated and where to locate them. https://github.com/neuropoly/data-management/issues/102
kousu
commented
3 years ago I'm not against it, but adding a line on each side of the mask would also require to label (black, gray or white) these lines to match the image. This is mainly why we chose to crop the image instead.
I took a look and many of the masks are already kind of shakey around the borders anyway, with straight lines extended where they probably shouldn't be, so I decided it wouldn't hurt anything to interpolate the missing border values by copying out the existing border one pixel more.
I put them on separate layers and dragged the first to be 1x1 pixel in from the top left of the canvas and then toggled its visibility on and off. I also tried setting the layer blend mode to "Difference" and got an entirely black image, except at the borders, which double-confirmed my assumption. I also redid this proof in numpy:
p115628@joplin:~/data-management-123$ ./verify-centering.py
okSo that confirms to me that to reconstruct the missing masks, I need to add a 1px border on all sides.
I restarted from /mnt/duke/projects/axondeepseg/raw_data/data_TEM/osf/20160718_nyu_mouse_26_0004/, which has the full-size image.png,
osf/ copy, were extracted by simply mapping grey -> white and white -> white with:axons = ((masks == 255)*255).astype(masks.dtype)
myelin = ((masks == 128)*255).astype(masks.dtype)To prove this:
osf/ folder to the TEM_dataset folder, and ran it:cp -r /mnt/duke/projects/axondeepseg/raw_data/data_TEM/osf/20160718_nyu_mouse_26_0004 /mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/20160718_nyu_mouse_26_0004
rm /mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/20160718_nyu_mouse_26_0004/Icon$'\r'
expand-mask.py /mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset/20160718_nyu_mouse_26_0004u108545@joplin:~/data_axondeepseg_tem$ python ../data-management/scripts/curate_data_axondeepseg_tem.py -d /mnt/duke/projects/axondeepseg/20210708_default_datasets_mask_curation/TEM_dataset -o .u108545@joplin:~/data_axondeepseg_tem$ git diff
diff --git a/README b/README
index 0c89edc..8a9b86b 100644
--- a/README
+++ b/README
@@ -1,8 +1,8 @@
- - TEM dataset for AxonDeepSeg (https://axondeepseg.readthedocs.io/)
- - 158 brain (splenium) samples from 20 mice with axon and myelin manual segmentation labels.
- - 20160718_nyu_mouse_25_0002 was omitted because it contained incomplete data in the folder smb://duke.neuro.polymtl.ca/projects/axondeepseg/raw_data/data_TEM/3_done/
- - Original source files are located in smb://duke.neuro.polymtl.ca/histology/mouse/20160718_nyu_mouse
- - Our original paper (Zaimi et al. 2018), the FOV was reported to be 6x9 um^2. This is because 1) these original values were reported in the original data reference (below), and 2) our images here are slightly cropped at the bottom relative to the original data in order to remove the scale bar.
- - Our original paper (Zaimi et al. 2018) reported the resolution as being 0.002 micrometer, which was (for an unknown reason) rounded in the paper from the true value of 0.00236 micrometer, as reported in the original data reference (below).
- - Reference for the origin of the dataset: Jelescu, I. O. et al. In vivo quantification of demyelination and recovery using compartment-specific diffusion MRI metrics validated by electron microscopy. Neuroimage 132, 104–114 (2016).
- - BIDS version 1.6.0 - Microscopy BEP031 version 0.0.4 (2021-07-13T15:14:00)
\ No newline at end of file
+- TEM dataset for AxonDeepSeg (https://axondeepseg.readthedocs.io/)
+- 158 brain (splenium) samples from 20 mice with axon and myelin manual segmentation labels.
+- 20160718_nyu_mouse_25_0002 was omitted because it contained incomplete data in the folder smb://duke.neuro.polymtl.ca/projects/axondeepseg/raw_data/data_TEM/3_done/
+- Original source files are located in smb://duke.neuro.polymtl.ca/histology/mouse/20160718_nyu_mouse
+- Our original paper (Zaimi et al. 2018), the FOV was reported to be 6x9 um^2. This is because 1) these original values were reported in the original data reference (below), and 2) our images here are slightly cropped at the bottom relative to the original data in order to remove the scale bar.
+- Our original paper (Zaimi et al. 2018) reported the resolution as being 0.002 micrometer, which was (for an unknown reason) rounded in the paper from the true value of 0.00236 micrometer, as reported in the original data reference (below).
+- Reference for the origin of the dataset: Jelescu, I. O. et al. In vivo quantification of demyelination and recovery using compartment-specific diffusion MRI metrics validated by electron microscopy. Neuroimage 132, 104–114 (2016).
+- BIDS version 1.6.0 - Microscopy BEP031 version 0.0.4 (2021-07-13T15:14:00)
diff --git a/derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0004_TEM_seg-axon-manual.png b/derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0004_TEM_seg-axon-manual.png
index 3cc16e3..88bc732 100755
--- a/derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0004_TEM_seg-axon-manual.png
+++ b/derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0004_TEM_seg-axon-manual.png
@@ -1 +1 @@
-/annex/objects/SHA256E-s49181--cd6328f6bf5e4b4415bc9ba93bd4c62eecf2871885b60b9fdfb83af226c53930.png
+/annex/objects/SHA256E-s65674--f2d9e1a43c97d9f53d2e471736f1fd3fc68862c49603ca4e0b24f454378fdaad.png
diff --git a/derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0004_TEM_seg-axonmyelin-manual.png b/derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0004_TEM_seg-axonmyelin-manual.png
index 6b09f16..0554c27 100755
--- a/derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0004_TEM_seg-axonmyelin-manual.png
+++ b/derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0004_TEM_seg-axonmyelin-manual.png
@@ -1 +1 @@
-/annex/objects/SHA256E-s118347--2cb2f00d59c7bd899aaf1881a821973bdb4b45a93d387d806e0a865ea19f44f3.png
+/annex/objects/SHA256E-s118388--8c72646792ad5d5281c98b05c3446a66487951e04142a9896770e3efc4c5ac68.png
diff --git a/derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0004_TEM_seg-myelin-manual.png b/derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0004_TEM_seg-myelin-manual.png
index ffed538..6f81e68 100755
--- a/derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0004_TEM_seg-myelin-manual.png
+++ b/derivatives/labels/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0004_TEM_seg-myelin-manual.png
@@ -1 +1 @@
-/annex/objects/SHA256E-s86049--b96c212c49f04ba534b8acc5d8a4b16bd52dc547fa39a2b17ad24f6dcaa6c6c3.png
+/annex/objects/SHA256E-s109815--3a8969ac3ccd1725616a78bfa4454ace86060bd2aeb3a189ff6677841c50dace.png
diff --git a/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0004_TEM.png b/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0004_TEM.png
index efbe7c4..ebf4a9c 100755
--- a/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0004_TEM.png
+++ b/sub-nyuMouse26/microscopy/sub-nyuMouse26_sample-0004_TEM.png
@@ -1 +1 @@
-/annex/objects/SHA256E-s6208809--acc352ab33da94300746215d0c8845076238ab55598c08fefe07eee23df91151.png
+/annex/objects/SHA256E-s6455059--62c6d3eda2211e0d12dba113b26be41c844bdcc1e2b685d81b381a153ef577b1.pngu108545@joplin:~/data_axondeepseg_tem$ find . -name .git -prune -o -name "*.png" | xargs file | cut -f 2- -d : | sed 's/^[[:space:]]*//' | sort | uniq -c
1 directory
632 PNG image data, 3762 x 2286, 8-bit grayscale, non-interlaced
I downloaded the dataset
data_axondeepseg_temfrom git annex using these instructions: https://github.com/neuropoly/data-management/blob/master/internal-server.md#downloadAll the files in the folders
derivatives/sub-nyuMouseXX/ending with_seg-axon-manual.pngare 1Kb and give me this error message when trying to open them:I don't know how to check if this is an issue with the dataset itself or with my download.
All the other files in the dataset seem OK. As a reference, this dataset was bidsified and upload to the server in PR #103.