Missing thumbnails and stories

adamjtaylor commented 3 years ago

Following manual review on 2021-11-03, the following files are missing either thumbnail or story.

Vanderbilt HE missing both thumbnail and story

[ ] Generated
[ ] In assets bucket
[ ] Fixed HTA11_866_20000010116110030010000009999.tif HTA11_83_20000010116110030010000009999.tif HTA11_6310_20000010116110030010000009999.tif HTA11_546_20000010116110030010000009999.tif HTA11_5363_20000010116110030010000009999.tif HTA11_5216_20000010116110030010000009999.tif HTA11_347_20000010116110030010000009999.tif HTA11_2235_20000010116110030010000009999.tif HTA11_2112_20000010116110030010000009999.tif HTA11_1391_20000010116110030010000009999.tif

OHSU CyCIF fix background in thumbnail

[ ] Generated
[ ] In assets bucket
[ ] Fixed BEMS350173_Scene-004.ome.tif BEMS350173_Scene-003.ome.tif BEMS350173_Scene-002.ome.tif BEMS340264_Scene-004.ome.tif BEMS340264_Scene-003.ome.tif BEMS340264_Scene-002.ome.tif BEMS267865_Scene-001.ome.tif

OHSU listed mIHC but actually H&E - missing thumbnail and regenerate minerva as h&E

[ ] Generated
[ ] In assets bucket
[ ] Fixed HTA9_1_BB_F.ome.tif HTA9_1_BB_L.ome.tif HTA9_1_BB_M.ome.tif HTA9_1_POST_M.ome.tif HTA9_1_POST_L_RESTAINED.ome.tif HTA9_1_POST_F.ome.tif HTA9_1_BA_M.ome.tif HTA9_1_BA_L.ome.tif HTA9_1_BA_F.ome.tif

WASHU IMC missing thumbnail

[x] Generated
[x] In assets bucket
[x] Fixed HT064B1_H9_A1_A4_ROI_02.ome.tiff HT064B1_H9_A1_A4_ROI_01.ome.tiff HT064B1_H7_A1_A4_ROI_03.ome.tiff HT064B1_H7_A1_A4_ROI_02.ome.tiff HT064B1_H7_A1_A4_ROI_01.ome.tiff HT064B1_H3_A1_A4_ROI_03.ome.tiff HT064B1_H3_A1_A4_ROI_02.ome.tiff HT064B1_H3_A1_A4_ROI_01.ome.tiff HT064B1_H1_A1_A4_ROI_04.ome.tiff HT064B1_H1_A1_A4_ROI_03.ome.tiff HT064B1_H1_A1_A4_ROI_02.ome.tiff HT064B1_H1_A1_A4_ROI_01.ome.tiff HT061P1_PC_A1_A4_ROI_03.ome.tiff HT061P1_PC_A1_A4_ROI_02.ome.tiff HT061P1_PC_A1_A4_ROI_01.ome.tiff HT061P1_PB_A1_A4_ROI_03.ome.tiff HT061P1_PB_A1_A4_ROI_02.ome.tiff HT061P1_PB_A1_A4_ROI_01.ome.tiff HT061P1_PA_A1_A4_ROI_03.ome.tiff HT061P1_PA_A1_A4_ROI_02.ome.tiff HT061P1_PA_A1_A4_ROI_01.ome.tiff HT060P1_REMAIN_ROI_04.ome.tiff HT060P1_REMAIN_ROI_03.ome.tiff HT060P1_REMAIN_ROI_02.ome.tiff HT060P1_REMAIN_ROI_01.ome.tiff HT060P1_PB_A1_A4_ROI_06.ome.tiff HT060P1_PB_A1_A4_ROI_05.ome.tiff HT060P1_PB_A1_A4_ROI_04.ome.tiff HT060P1_PB_A1_A4_ROI_02.ome.tiff HT060P1_PB_A1_A4_ROI_01.ome.tiff HT060P1_PA_A1_A4_ROI_03.ome.tiff HT060P1_PA_A1_A4_ROI_02.ome.tiff HT060P1_PA_A1_A4_ROI_01.ome.tiff HT056P1_PB_A1_A4_ROI_05.ome.tiff HT056P1_PB_A1_A4_ROI_04.ome.tiff HT056P1_PB_A1_A4_ROI_03.ome.tiff HT056P1_PB_A1_A4_ROI_02.ome.tiff HT056P1_PB_A1_A4_ROI_01.ome.tiff HT056P1_PA_A1_A4_ROI_03.ome.tiff HT056P1_PA_A1_A4_ROI_02.ome.tiff HT056P1_PA_A1_A4_ROI_01.ome.tiff

WASHU IMC need minerva stories

[x] Generated
[x] In assets bucket
[x] Fixed HTAN_HT122P1_S1H5_L1_L4_ROI_002.ome.tiff HTAN_HT122P1_S1H9_L1_L4_ROI_001.ome.tiff HTAN_HT122P1_S1H9_L1_L4_ROI_002.ome.tiff HTAN_HT123P1_S1H1_L1_L4_ROI_001.ome.tiff HTAN_HT123P1_S1H3_L1_L4_ROI_001.ome.tiff HTAN_HT123P1_S1H3_L1_L4_ROI_002.ome.tiff HTAN_HT124P1S1H2_L1_L4_ROI_001.ome.tiff HTAN_HT124P1_S1H1_L1_L4_ROI_001.ome.tiff HTAN_HT124P1_S1H4_L1_L4_ROI_003.ome.tiff HTAN_HT166P1_S1H2_L1_L4_ROI_001.ome.tiff HTAN_HT124P1_S1H4_L1_L4_ROI_005.ome.tiff HTAN_HT166P1_S1H5_L4_ROI_001.ome.tiff HTAN_HT166P1_S1H5_L4_ROI_002.ome.tiff HTAN_HT166P1_S1H5_L4_ROI_003.ome.tiff HTAN_HT166P1_S1H6_L1_L4_ROI_002.ome.tiff HTAN_HT168P1_S1H2_L1_L4_ROI_001.ome.tiff HTAN_HT168P1_S1H4_L1_L4_ROI_001.ome.tiff HTAN_HT168P1_S1H4_L1_L4_ROI_002.ome.tiff HTAN_HT168P1_S1H4_L1_L4_ROI_003.ome.tiff HTAN_HT122P1_S1H5_L1_L4_ROI_001.ome.tiff HTAN_HT122P1_S1H4_L1_L4_ROI_002.ome.tiff HTAN_HT122P1_S1H4_L1_L4_ROI_001.ome.tiff HTAN_HT122P1_S1H3_L1_L4_ROI_003.ome.tiff HTAN_HT122P1_S1H3_L1_L4_ROI_001.ome.tiff HT168P1_S1H3_L1_L4_ROI_003.ome.tiff HT168P1_S1H3_L1_L4_ROI_001.ome.tiff

WASHU IMC need thumbnails and minerva stories

[x] Generated
[x] In assets bucket
[x] Fixed HT168P1_S1H3_L1_L4_ROI_002.ome.tiff HTAN_HT122P1_S1H3_L1_L4_ROI_002.ome.tiff HTAN_HT123P1_S1H1_L1_L4_ROI_002.ome.tiff HTAN_HT123P1_S1H5_L1_L4_ROI_001.ome.tiff HTAN_HT123P1_S1H5_L1_L4_ROI_002.ome.tiff HTAN_HT124P1S1H2_L1_L4_ROI_002.ome.tiff HTAN_HT124P1_S1H1_L1_L4_ROI_002.ome.tiff HTAN_HT124P1_S1H4_L1_L4_ROI_002.ome.tiff HTAN_HT166P1_S1H2_L1_L4_ROI_002.ome.tiff HTAN_HT166P1_S1H6_L1_L4_ROI_001.ome.tiff HTAN_HT168P1_S1H2_L1_L4_ROI_002.ome.tiff

adamjtaylor commented 3 years ago

@hweej ... last min request... could you look why the washu assets listed above re not appearing in the mapping. they are now present in the assets bucket but when doing the mapping they still show as null

I think this is related to these basenames appearing in two separate folders as well as in an archive folder.

hweej commented 3 years ago

I'll take a look at it!

adamjtaylor commented 3 years ago

Vanderbilt files failed due to out of Java heap memory errors during conversion to OME-TIFF (pyramid required for both minerva rendering and thumbnails) eg.

2021-11-03 13:50:00,114 [pool-1-thread-1] ERROR c.g.bioformats2raw.Converter - Failure processing chunk; resolution=0 plane=0 xx=39936 yy=17408 zz=0 width=1024 height=1024 depth=1
java.lang.OutOfMemoryError: Java heap space

Increasing limit on AWSBatch machines called by Nextflow Tower not sufficient at the moment!

hweej commented 3 years ago

@adamjtaylor , it seems like a new imc_level_2 subdirectory was created when uploading the latest htan-assets.

# Extra imc_level_2
htan-assets/minerva_stories/htan-dcc-washu/imc_level_2/imc_level_2/batch_1_10072021/ 

# I fixed htan assets back to this
htan-assets/minerva_stories/htan-dcc-washu/imc_level_2/batch_1_10072021/

ncihtan / htan-portal