search

dandi / dandisets-healthstatus

Healthchecks of dandisets and support libraries (pynwb and matnwb)
0 stars 1 forks source link

Traverse assets via Archive API #81

Closed jwodder closed 2 months ago

jwodder commented 2 months ago

As discussed in #77.

To do:

codecov[bot] commented 2 months ago

Codecov Report

Attention: Patch coverage is 44.31373% with 142 lines in your changes missing coverage. Please review.

Project coverage is 60.50%. Comparing base (eef70d9) to head (af6090f).

Files Patch % Lines
code/src/healthstatus/checker.py 39.45% 85 Missing and 4 partials :warning:
code/src/healthstatus/adandi.py 57.35% 27 Missing and 2 partials :warning:
code/src/healthstatus/aioutil.py 27.58% 21 Missing :warning:
code/src/healthstatus/core.py 60.00% 2 Missing :warning:
code/src/healthstatus/util.py 66.66% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #81 +/- ## ========================================== - Coverage 60.59% 60.50% -0.10% ========================================== Files 10 11 +1 Lines 835 947 +112 Branches 191 212 +21 ========================================== + Hits 506 573 +67 - Misses 309 351 +42 - Partials 20 23 +3 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

jwodder commented 2 months ago

@yarikoptic FYI: I started a run with the new code in "random-outdated-asset-first" mode at 2:37 PM EDT. The script found all the Dandisets within two minutes. Currently, tests are being run on assets from Dandiset 000016 through 000036 (not all the ones in between, though), and 16 Dandisets have finished having their asset tested.

yarikoptic commented 2 months ago

I would check on drogon later on what it is busy with and either didn't stall again

jwodder commented 2 months ago

@yarikoptic I started another run yesterday at 4 PM, and it's still going now. I managed to parse out the following durations from the logfile for the tests that have completed so far:

```csv dandiset,asset,test,outcome,time_secs 000003,sub-YutaMouse44/sub-YutaMouse44_ses-YutaMouse44-151128_behavior+ecephys.nwb,matnwb_nwbRead,FAIL,1031.0 000003,sub-YutaMouse44/sub-YutaMouse44_ses-YutaMouse44-151128_behavior+ecephys.nwb,pynwb_open_load_ns,PASS,279.0 000004,sub-P60CS/sub-P60CS_ses-20181001_ecephys+image.nwb,matnwb_nwbRead,TIMEOUT,3601.0 000004,sub-P60CS/sub-P60CS_ses-20181001_ecephys+image.nwb,pynwb_open_load_ns,PASS,212.0 000005,sub-anm300172/sub-anm300172_ses-20150908_obj-n5zk5z_behavior+icephys+ogen.nwb,matnwb_nwbRead,FAIL,1039.0 000005,sub-anm300172/sub-anm300172_ses-20150908_obj-n5zk5z_behavior+icephys+ogen.nwb,pynwb_open_load_ns,PASS,212.0 000006,sub-anm369964/sub-anm369964_ses-20170321.nwb,matnwb_nwbRead,FAIL,1035.0 000006,sub-anm369964/sub-anm369964_ses-20170321.nwb,pynwb_open_load_ns,PASS,212.0 000007,sub-BAYLORNL12/sub-BAYLORNL12_ses-20170614T094441.nwb,matnwb_nwbRead,FAIL,1040.0 000007,sub-BAYLORNL12/sub-BAYLORNL12_ses-20170614T094441.nwb,pynwb_open_load_ns,PASS,212.0 000008,sub-mouse-BXLFB/sub-mouse-BXLFB_ses-20180103-sample-6_slice-20180103-slice-6_cell-20180103-sample-6_icephys.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000008,sub-mouse-BXLFB/sub-mouse-BXLFB_ses-20180103-sample-6_slice-20180103-slice-6_cell-20180103-sample-6_icephys.nwb,pynwb_open_load_ns,PASS,316.0 000009,sub-anm00237800/sub-anm00237800_ses-20170627T093542_ecephys.nwb,matnwb_nwbRead,FAIL,1040.0 000009,sub-anm00237800/sub-anm00237800_ses-20170627T093542_ecephys.nwb,pynwb_open_load_ns,PASS,212.0 000010,sub-216219/sub-216219_ses-20130815_obj-1eyeli4_behavior+ophys.nwb,matnwb_nwbRead,FAIL,1039.0 000010,sub-216219/sub-216219_ses-20130815_obj-1eyeli4_behavior+ophys.nwb,pynwb_open_load_ns,PASS,267.0 000011,sub-270340/sub-270340_ses-20150618_behavior+ecephys+ogen.nwb,matnwb_nwbRead,FAIL,1036.0 000011,sub-270340/sub-270340_ses-20150618_behavior+ecephys+ogen.nwb,pynwb_open_load_ns,PASS,196.0 000012,sub-2/sub-2_ses-18518010_icephys.nwb,matnwb_nwbRead,FAIL,1041.0 000012,sub-2/sub-2_ses-18518010_icephys.nwb,pynwb_open_load_ns,PASS,196.0 000013,sub-anm106213/sub-anm106213_ses-20100822_behavior+icephys.nwb,matnwb_nwbRead,TIMEOUT,3657.0 000013,sub-anm106213/sub-anm106213_ses-20100822_behavior+icephys.nwb,pynwb_open_load_ns,PASS,595.0 000015,sub-an064/sub-an064_ses-20150219_obj-18v1u0u.nwb,matnwb_nwbRead,TIMEOUT,3603.0 000015,sub-an064/sub-an064_ses-20150219_obj-18v1u0u.nwb,pynwb_open_load_ns,PASS,544.0 000016,sub-mouse2-fni17/sub-mouse2-fni17_ses-161024145847.nwb,matnwb_nwbRead,TIMEOUT,3803.0 000016,sub-mouse2-fni17/sub-mouse2-fni17_ses-161024145847.nwb,pynwb_open_load_ns,TIMEOUT,3601.0 000017,sub-Lederberg/sub-Lederberg_ses-20171209T120000.nwb,matnwb_nwbRead,TIMEOUT,3802.0 000017,sub-Lederberg/sub-Lederberg_ses-20171209T120000.nwb,pynwb_open_load_ns,PASS,890.0 000019,sub-EC2/sub-EC2_ses-EC2-B15.nwb,matnwb_nwbRead,TIMEOUT,3645.0 000019,sub-EC2/sub-EC2_ses-EC2-B15.nwb,pynwb_open_load_ns,PASS,396.0 000020,sub-670689759/sub-670689759_ses-672918159_icephys.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000020,sub-670689759/sub-670689759_ses-672918159_icephys.nwb,pynwb_open_load_ns,PASS,2758.0 000021,sub-707296975/sub-707296975_ses-721123822_probe-760213142_ecephys.nwb,matnwb_nwbRead,TIMEOUT,3657.0 000021,sub-707296975/sub-707296975_ses-721123822_probe-760213142_ecephys.nwb,pynwb_open_load_ns,PASS,590.0 000022,sub-811322604/sub-811322604_ses-829720705_probe-832129157_ecephys.nwb,matnwb_nwbRead,TIMEOUT,3605.0 000022,sub-811322604/sub-811322604_ses-829720705_probe-832129157_ecephys.nwb,pynwb_open_load_ns,PASS,537.0 000023,sub-758233315/sub-758233315_ses-758661907_icephys.nwb,matnwb_nwbRead,TIMEOUT,4110.0 000023,sub-758233315/sub-758233315_ses-758661907_icephys.nwb,pynwb_open_load_ns,TIMEOUT,3801.0 000025,001_140709EXP_A1.nwb,matnwb_nwbRead,TIMEOUT,4110.0 000025,001_140709EXP_A1.nwb,pynwb_open_load_ns,TIMEOUT,3690.0 000027,sub-RAT123/sub-RAT123.nwb,matnwb_nwbRead,TIMEOUT,3994.0 000027,sub-RAT123/sub-RAT123.nwb,pynwb_open_load_ns,PASS,1099.0 000028,sub-MEAREC-250neuron-Neuropixels/sub-MEAREC-250neuron-Neuropixels_ses-20200727T094620_ecephys.nwb,matnwb_nwbRead,TIMEOUT,3661.0 000028,sub-MEAREC-250neuron-Neuropixels/sub-MEAREC-250neuron-Neuropixels_ses-20200727T094620_ecephys.nwb,pynwb_open_load_ns,PASS,1810.0 000029,sub-monk-g/sub-monk-g.nwb,matnwb_nwbRead,TIMEOUT,3648.0 000029,sub-monk-g/sub-monk-g.nwb,pynwb_open_load_ns,PASS,1824.0 000034,sub-MEAREC-250neuron-Neuropixels/sub-MEAREC-250neuron-Neuropixels_ecephys.nwb,matnwb_nwbRead,TIMEOUT,3661.0 000034,sub-MEAREC-250neuron-Neuropixels/sub-MEAREC-250neuron-Neuropixels_ecephys.nwb,pynwb_open_load_ns,PASS,1705.0 000035,sub-mouse-INETJ/sub-mouse-INETJ_ses-20200407-sample-13_slice-20200407-slice-5_cell-20200407-sample-13_icephys.nwb,matnwb_nwbRead,TIMEOUT,3605.0 000035,sub-mouse-INETJ/sub-mouse-INETJ_ses-20200407-sample-13_slice-20200407-slice-5_cell-20200407-sample-13_icephys.nwb,pynwb_open_load_ns,TIMEOUT,4044.0 000036,sub-395481/sub-395481_ses-20180725T154856_behavior+image+ophys.nwb,matnwb_nwbRead,TIMEOUT,4109.0 000036,sub-395481/sub-395481_ses-20180725T154856_behavior+image+ophys.nwb,pynwb_open_load_ns,PASS,1670.0 000037,sub-433451/sub-433451_ses-824434038_obj-raw_behavior+image+ophys.nwb,matnwb_nwbRead,TIMEOUT,3962.0 000037,sub-433451/sub-433451_ses-824434038_obj-raw_behavior+image+ophys.nwb,pynwb_open_load_ns,TIMEOUT,4110.0 000039,sub-678530120/sub-678530120_ses-696742747_behavior+ophys.nwb,matnwb_nwbRead,TIMEOUT,3617.0 000039,sub-678530120/sub-678530120_ses-696742747_behavior+ophys.nwb,pynwb_open_load_ns,TIMEOUT,3715.0 000041,sub-BWRat19/sub-BWRat19_ses-BWRat19-032413_ecephys.nwb,matnwb_nwbRead,TIMEOUT,3664.0 000041,sub-BWRat19/sub-BWRat19_ses-BWRat19-032413_ecephys.nwb,pynwb_open_load_ns,TIMEOUT,4076.0 000043,sub-Q19-26-008/sub-Q19-26-008_ses-20190816T183141_icephys.nwb,matnwb_nwbRead,TIMEOUT,3667.0 000043,sub-Q19-26-008/sub-Q19-26-008_ses-20190816T183141_icephys.nwb,pynwb_open_load_ns,TIMEOUT,3615.0 000044,sub-Cicero/sub-Cicero_ses-Cicero-09102014_behavior+ecephys.nwb,matnwb_nwbRead,TIMEOUT,3611.0 000044,sub-Cicero/sub-Cicero_ses-Cicero-09102014_behavior+ecephys.nwb,pynwb_open_load_ns,PASS,2285.0 000045,sub-44eaff6b-1ede-422d-badc-2e82e866516b/sub-44eaff6b-1ede-422d-badc-2e82e866516b_ses-9b164387-e6a0-4f6c-be0f-dd08cc0671bb_behavior+image.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000045,sub-44eaff6b-1ede-422d-badc-2e82e866516b/sub-44eaff6b-1ede-422d-badc-2e82e866516b_ses-9b164387-e6a0-4f6c-be0f-dd08cc0671bb_behavior+image.nwb,pynwb_open_load_ns,TIMEOUT,3612.0 000048,sub-222549/sub-222549_ecephys+ophys.nwb,matnwb_nwbRead,TIMEOUT,3930.0 000048,sub-222549/sub-222549_ecephys+ophys.nwb,pynwb_open_load_ns,PASS,2913.0 000049,sub-664605504/sub-664605504_ses-687293842_behavior+ophys.nwb,matnwb_nwbRead,TIMEOUT,3601.0 000049,sub-664605504/sub-664605504_ses-687293842_behavior+ophys.nwb,pynwb_open_load_ns,PASS,3153.0 000050,sub-763958536/sub-763958536_ses-794944810_behavior+ophys.nwb,matnwb_nwbRead,TIMEOUT,6179.0 000050,sub-763958536/sub-763958536_ses-794944810_behavior+ophys.nwb,pynwb_open_load_ns,TIMEOUT,3961.0 000051,pons8-yo_16xdownsampled.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000051,pons8-yo_16xdownsampled.nwb,pynwb_open_load_ns,PASS,1618.0 000053,sub-Dorian/sub-Dorian_ses-20191111_obj-ybom15_behavior.nwb,matnwb_nwbRead,TIMEOUT,5102.0 000053,sub-Dorian/sub-Dorian_ses-20191111_obj-ybom15_behavior.nwb,pynwb_open_load_ns,PASS,1076.0 000054,sub-R2/sub-R2_ses-20190219T210000_behavior+ophys.nwb,matnwb_nwbRead,TIMEOUT,3715.0 000054,sub-R2/sub-R2_ses-20190219T210000_behavior+ophys.nwb,pynwb_open_load_ns,PASS,2161.0 000055,sub-08/sub-08_ses-4_behavior+ecephys.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000055,sub-08/sub-08_ses-4_behavior+ecephys.nwb,pynwb_open_load_ns,TIMEOUT,5103.0 000056,sub-Mouse25/sub-Mouse25_ses-Mouse25-140204_behavior+ecephys.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000056,sub-Mouse25/sub-Mouse25_ses-Mouse25-140204_behavior+ecephys.nwb,pynwb_open_load_ns,TIMEOUT,3781.0 000059,sub-MS21/sub-MS21_ses-Peter-MS21-180628-155921-concat_desc-raw_ecephys.nwb,matnwb_nwbRead,TIMEOUT,3879.0 000059,sub-MS21/sub-MS21_ses-Peter-MS21-180628-155921-concat_desc-raw_ecephys.nwb,pynwb_open_load_ns,TIMEOUT,3719.0 000060,sub-365942/sub-365942_ses-20170725_behavior+ecephys+ogen.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000060,sub-365942/sub-365942_ses-20170725_behavior+ecephys+ogen.nwb,pynwb_open_load_ns,TIMEOUT,3716.0 000061,sub-Rat11/sub-Rat11_ses-Rat11-20150316_ecephys+image.nwb,matnwb_nwbRead,TIMEOUT,3735.0 000061,sub-Rat11/sub-Rat11_ses-Rat11-20150316_ecephys+image.nwb,pynwb_open_load_ns,TIMEOUT,3641.0 000064,sub-001/sub-001.nwb,matnwb_nwbRead,TIMEOUT,3707.0 000064,sub-001/sub-001.nwb,pynwb_open_load_ns,TIMEOUT,4166.0 000065,sub-Kibbles/sub-Kibbles_behavior+ecephys.nwb,matnwb_nwbRead,TIMEOUT,3708.0 000065,sub-Kibbles/sub-Kibbles_behavior+ecephys.nwb,pynwb_open_load_ns,TIMEOUT,4077.0 000067,sub-EE/sub-EE_ses-EE-048_ecephys.nwb,matnwb_nwbRead,TIMEOUT,3992.0 000067,sub-EE/sub-EE_ses-EE-048_ecephys.nwb,pynwb_open_load_ns,TIMEOUT,3879.0 000068,sub-abcd/sub-abcd_icephys.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000068,sub-abcd/sub-abcd_icephys.nwb,pynwb_open_load_ns,TIMEOUT,3879.0 000070,sub-Jenkins/sub-Jenkins_ses-20090912_behavior+ecephys.nwb,matnwb_nwbRead,TIMEOUT,3992.0 000070,sub-Jenkins/sub-Jenkins_ses-20090912_behavior+ecephys.nwb,pynwb_open_load_ns,TIMEOUT,3879.0 000107,PS_Stimsets-20190906.nwb,matnwb_nwbRead,TIMEOUT,4591.0 000107,PS_Stimsets-20190906.nwb,pynwb_open_load_ns,TIMEOUT,3622.0 000109,sub-651940947/sub-651940947_ses-652079700_icephys.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000109,sub-651940947/sub-651940947_ses-652079700_icephys.nwb,pynwb_open_load_ns,TIMEOUT,3622.0 000114,sub-V118/sub-V118_ses-03212018_ecephys.nwb,matnwb_nwbRead,TIMEOUT,4591.0 000114,sub-V118/sub-V118_ses-03212018_ecephys.nwb,pynwb_open_load_ns,TIMEOUT,3619.0 000115,sub-jaq/sub-jaq_ses-jaq-18_behavior+ecephys.nwb,matnwb_nwbRead,TIMEOUT,3942.0 000115,sub-jaq/sub-jaq_ses-jaq-18_behavior+ecephys.nwb,pynwb_open_load_ns,TIMEOUT,3992.0 000117,sub-20210615001-2021-06-15-0004/sub-20210615001-2021-06-15-0004_icephys.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000117,sub-20210615001-2021-06-15-0004/sub-20210615001-2021-06-15-0004_icephys.nwb,pynwb_open_load_ns,TIMEOUT,3992.0 000122,sub-P4/sub-P4_ses-20200104T031322.nwb,matnwb_nwbRead,TIMEOUT,3942.0 000122,sub-P4/sub-P4_ses-20200104T031322.nwb,pynwb_open_load_ns,PASS,3383.0 000126,sub-1001658946/sub-1001658946_obj-831bdx_icephys.nwb,matnwb_nwbRead,TIMEOUT,3630.0 000126,sub-1001658946/sub-1001658946_obj-831bdx_icephys.nwb,pynwb_open_load_ns,TIMEOUT,5653.0 000127,sub-Han/sub-Han_desc-train_behavior+ecephys.nwb,matnwb_nwbRead,TIMEOUT,3616.0 000127,sub-Han/sub-Han_desc-train_behavior+ecephys.nwb,pynwb_open_load_ns,TIMEOUT,4334.0 000128,sub-Jenkins/sub-Jenkins_ses-full_desc-test_ecephys.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000128,sub-Jenkins/sub-Jenkins_ses-full_desc-test_ecephys.nwb,pynwb_open_load_ns,TIMEOUT,3942.0 000129,sub-Indy/sub-Indy_desc-test_ecephys.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000129,sub-Indy/sub-Indy_desc-test_ecephys.nwb,pynwb_open_load_ns,TIMEOUT,3942.0 000130,sub-Haydn/sub-Haydn_desc-train_ecephys.nwb,matnwb_nwbRead,TIMEOUT,10004.0 000130,sub-Haydn/sub-Haydn_desc-train_ecephys.nwb,pynwb_open_load_ns,PASS,1276.0 000138,sub-Jenkins/sub-Jenkins_ses-large_desc-test_ecephys.nwb,matnwb_nwbRead,TIMEOUT,10127.0 000138,sub-Jenkins/sub-Jenkins_ses-large_desc-test_ecephys.nwb,pynwb_open_load_ns,PASS,503.0 000139,sub-Jenkins/sub-Jenkins_ses-medium_desc-test_ecephys.nwb,matnwb_nwbRead,TIMEOUT,3601.0 000139,sub-Jenkins/sub-Jenkins_ses-medium_desc-test_ecephys.nwb,pynwb_open_load_ns,PASS,204.0 000140,sub-Jenkins/sub-Jenkins_ses-small_desc-test_ecephys.nwb,matnwb_nwbRead,TIMEOUT,10085.0 000140,sub-Jenkins/sub-Jenkins_ses-small_desc-test_ecephys.nwb,pynwb_open_load_ns,PASS,204.0 000142,sub-902323111/sub-902323111_ses-902449420_icephys.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000142,sub-902323111/sub-902323111_ses-902449420_icephys.nwb,pynwb_open_load_ns,TIMEOUT,3616.0 000147,sub-P1/sub-P1_ses-2018-10-15_ecephys.nwb,matnwb_nwbRead,TIMEOUT,8058.0 000147,sub-P1/sub-P1_ses-2018-10-15_ecephys.nwb,pynwb_open_load_ns,PASS,272.0 000148,sub-20210825001/sub-20210825001_icephys.nwb,matnwb_nwbRead,TIMEOUT,3726.0 000148,sub-20210825001/sub-20210825001_icephys.nwb,pynwb_open_load_ns,TIMEOUT,3600.0 000149,sub-9bebfe0b-082e-4d66-aca7-fae29317f708/sub-9bebfe0b-082e-4d66-aca7-fae29317f708_ses-4b7fbad4-f6de-43b4-9b15-c7c7ef44db4b_behavior+ecephys+image.nwb,matnwb_nwbRead,TIMEOUT,7525.0 000149,sub-9bebfe0b-082e-4d66-aca7-fae29317f708/sub-9bebfe0b-082e-4d66-aca7-fae29317f708_ses-4b7fbad4-f6de-43b4-9b15-c7c7ef44db4b_behavior+ecephys+image.nwb,pynwb_open_load_ns,TIMEOUT,6690.0 000165,sub-Dill/sub-Dill_ses-Dill02-HomeCage_behavior+ecephys.nwb,matnwb_nwbRead,TIMEOUT,3601.0 000165,sub-Dill/sub-Dill_ses-Dill02-HomeCage_behavior+ecephys.nwb,pynwb_open_load_ns,TIMEOUT,3600.0 000166,sub-YMV11/sub-YMV11_ses-YMV11-171208_ecephys.nwb,matnwb_nwbRead,TIMEOUT,3696.0 000166,sub-YMV11/sub-YMV11_ses-YMV11-171208_ecephys.nwb,pynwb_open_load_ns,TIMEOUT,3600.0 000167,sub-8/sub-8_ses-20200204T180138_ophys.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000167,sub-8/sub-8_ses-20200204T180138_ophys.nwb,pynwb_open_load_ns,TIMEOUT,3710.0 000168,jGCaMP8s/jGCaMP8s_ANM479120_cell01.nwb,matnwb_nwbRead,TIMEOUT,3629.0 000168,jGCaMP8s/jGCaMP8s_ANM479120_cell01.nwb,pynwb_open_load_ns,TIMEOUT,7525.0 000173,sub-BH266/sub-BH266_ses-20211118_obj-rll9dh.nwb,matnwb_nwbRead,TIMEOUT,3629.0 000173,sub-BH266/sub-BH266_ses-20211118_obj-rll9dh.nwb,pynwb_open_load_ns,TIMEOUT,7525.0 000206,sub-TIGRE296/sub-TIGRE296_behavior+ophys.nwb,matnwb_nwbRead,TIMEOUT,3629.0 000206,sub-TIGRE296/sub-TIGRE296_behavior+ophys.nwb,pynwb_open_load_ns,TIMEOUT,7525.0 000207,sub-3/sub-3_ses-3_ecephys.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000207,sub-3/sub-3_ses-3_ecephys.nwb,pynwb_open_load_ns,TIMEOUT,7525.0 000209,sub-770227957/sub-770227957_ses-770275397_icephys.nwb,matnwb_nwbRead,TIMEOUT,3629.0 000209,sub-770227957/sub-770227957_ses-770275397_icephys.nwb,pynwb_open_load_ns,TIMEOUT,7525.0 000212,sub-0-200-VT040574-kirmutant-fly#-18/sub-0-200-VT040574-kirmutant-fly#-18_behavior.nwb,matnwb_nwbRead,TIMEOUT,3629.0 000212,sub-0-200-VT040574-kirmutant-fly#-18/sub-0-200-VT040574-kirmutant-fly#-18_behavior.nwb,pynwb_open_load_ns,TIMEOUT,6994.0 000213,sub-DT8/sub-DT8_ses-20170305-468um-2088um-170305-135233_behavior+ecephys.nwb,matnwb_nwbRead,TIMEOUT,3787.0 000213,sub-DT8/sub-DT8_ses-20170305-468um-2088um-170305-135233_behavior+ecephys.nwb,pynwb_open_load_ns,TIMEOUT,3600.0 000218,sub-DT5/sub-DT5_ses-20161030-2052um-2232um-161030-111142_behavior+ecephys.nwb,matnwb_nwbRead,TIMEOUT,3657.0 000218,sub-DT5/sub-DT5_ses-20161030-2052um-2232um-161030-111142_behavior+ecephys.nwb,pynwb_open_load_ns,TIMEOUT,3787.0 000219,sub-210226-210307-1/sub-210226-210307-1_ses-210307-a000_behavior+image+ophys.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000219,sub-210226-210307-1/sub-210226-210307-1_ses-210307-a000_behavior+image+ophys.nwb,pynwb_open_load_ns,PASS,1036.0 000220,sub-20190207002/sub-20190207002_icephys.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000220,sub-20190207002/sub-20190207002_icephys.nwb,pynwb_open_load_ns,TIMEOUT,3651.0 000221,sub-HI209/sub-HI209_ses-HI209-080619_ecephys+ogen.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000221,sub-HI209/sub-HI209_ses-HI209-080619_ecephys+ogen.nwb,pynwb_open_load_ns,PASS,536.0 000223,sub-3278/sub-3278_ses-20210614T151114_ecephys+ophys.nwb,matnwb_nwbRead,TIMEOUT,3601.0 000223,sub-3278/sub-3278_ses-20210614T151114_ecephys+ophys.nwb,pynwb_open_load_ns,TIMEOUT,5051.0 000226,sub-KSt137/sub-KSt137_ses-20161017T120000_behavior.nwb,matnwb_nwbRead,TIMEOUT,3601.0 000226,sub-KSt137/sub-KSt137_ses-20161017T120000_behavior.nwb,pynwb_open_load_ns,TIMEOUT,5021.0 000228,sub-H18-28-022/sub-H18-28-022_obj-1nyyyz6_icephys.nwb,matnwb_nwbRead,TIMEOUT,3601.0 000228,sub-H18-28-022/sub-H18-28-022_obj-1nyyyz6_icephys.nwb,pynwb_open_load_ns,TIMEOUT,5021.0 000230,sub-70375/sub-70375_ses-20170318T185915_behavior+ecephys.nwb,matnwb_nwbRead,TIMEOUT,3601.0 000230,sub-70375/sub-70375_ses-20170318T185915_behavior+ecephys.nwb,pynwb_open_load_ns,TIMEOUT,5021.0 000231,sub-231CR/sub-231CR_ses-20190927T145645_behavior+ecephys+image.nwb,matnwb_nwbRead,TIMEOUT,3601.0 000231,sub-231CR/sub-231CR_ses-20190927T145645_behavior+ecephys+image.nwb,pynwb_open_load_ns,TIMEOUT,5019.0 000232,sub-YT083/sub-YT083_ses-20200123T164814_behavior+ecephys.nwb,matnwb_nwbRead,TIMEOUT,3601.0 000232,sub-YT083/sub-YT083_ses-20200123T164814_behavior+ecephys.nwb,pynwb_open_load_ns,TIMEOUT,5021.0 000233,sub-CGM32/sub-CGM32_ses-CGM32-576um-210218-085131_ecephys.nwb,matnwb_nwbRead,TIMEOUT,5079.0 000233,sub-CGM32/sub-CGM32_ses-CGM32-576um-210218-085131_ecephys.nwb,pynwb_open_load_ns,TIMEOUT,4497.0 000235,sub-Fish02-GCaMP-vlgut-FBv-5dpf-RandomWave/sub-Fish02-GCaMP-vlgut-FBv-5dpf-RandomWave_ses-20210818T173531_behavior+ophys.nwb,matnwb_nwbRead,TIMEOUT,5494.0 000235,sub-Fish02-GCaMP-vlgut-FBv-5dpf-RandomWave/sub-Fish02-GCaMP-vlgut-FBv-5dpf-RandomWave_ses-20210818T173531_behavior+ophys.nwb,pynwb_open_load_ns,TIMEOUT,3600.0 000236,sub-Fish14-GCaMP-vlgut-MBd-5dpf-RandomWave/sub-Fish14-GCaMP-vlgut-MBd-5dpf-RandomWave_ses-20210830T162840_behavior+ophys.nwb,matnwb_nwbRead,TIMEOUT,5490.0 000236,sub-Fish14-GCaMP-vlgut-MBd-5dpf-RandomWave/sub-Fish14-GCaMP-vlgut-MBd-5dpf-RandomWave_ses-20210830T162840_behavior+ophys.nwb,pynwb_open_load_ns,TIMEOUT,3600.0 000237,sub-Fish33-GCaMP-vlgut-HBd-6dpf-RandomWave/sub-Fish33-GCaMP-vlgut-HBd-6dpf-RandomWave_ses-20210921T100911_behavior+ophys.nwb,matnwb_nwbRead,TIMEOUT,3754.0 000237,sub-Fish33-GCaMP-vlgut-HBd-6dpf-RandomWave/sub-Fish33-GCaMP-vlgut-HBd-6dpf-RandomWave_ses-20210921T100911_behavior+ophys.nwb,pynwb_open_load_ns,TIMEOUT,5976.0 000238,sub-RSpinal-Random-Wave-04/sub-RSpinal-Random-Wave-04_ses-20211008T101450_behavior+ophys.nwb,matnwb_nwbRead,TIMEOUT,3833.0 000238,sub-RSpinal-Random-Wave-04/sub-RSpinal-Random-Wave-04_ses-20211008T101450_behavior+ophys.nwb,pynwb_open_load_ns,TIMEOUT,3620.0 000239,sub-MX200102/sub-MX200102_ses-20201027T141948.nwb,matnwb_nwbRead,TIMEOUT,3672.0 000239,sub-MX200102/sub-MX200102_ses-20201027T141948.nwb,pynwb_open_load_ns,PASS,2552.0 000244,sub-SLC11/sub-SLC11_ses-anest_ophys.nwb,matnwb_nwbRead,TIMEOUT,3833.0 000244,sub-SLC11/sub-SLC11_ses-anest_ophys.nwb,pynwb_open_load_ns,TIMEOUT,3620.0 000245,sub-20220106003/sub-20220106003_ses-20220106T193954_slice-20220106003_cell-20220106003_icephys.nwb,matnwb_nwbRead,TIMEOUT,3853.0 000245,sub-20220106003/sub-20220106003_ses-20220106T193954_slice-20220106003_cell-20220106003_icephys.nwb,pynwb_open_load_ns,TIMEOUT,3600.0 000246,sub-phpV9-13/sub-phpV9-13_ses-20230822_obj-1tz06dm_ophys.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000246,sub-phpV9-13/sub-phpV9-13_ses-20230822_obj-1tz06dm_ophys.nwb,pynwb_open_load_ns,PASS,2551.0 000247,sub-oviDNSS1-driving-GCaMP7f-&-CsChrimson-fly-ID#5012/sub-oviDNSS1-driving-GCaMP7f-&-CsChrimson-fly-ID#5012_ses-2019-09-06-0001-oviDNSS1-GCaMP7f&CsChrimson-manualstim-nwb.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000247,sub-oviDNSS1-driving-GCaMP7f-&-CsChrimson-fly-ID#5012/sub-oviDNSS1-driving-GCaMP7f-&-CsChrimson-fly-ID#5012_ses-2019-09-06-0001-oviDNSS1-GCaMP7f&CsChrimson-manualstim-nwb.nwb,pynwb_open_load_ns,PASS,2554.0 000248,sub-631510/sub-631510_ses-1196157974_probe-5_ecephys.nwb,matnwb_nwbRead,TIMEOUT,3670.0 000248,sub-631510/sub-631510_ses-1196157974_probe-5_ecephys.nwb,pynwb_open_load_ns,PASS,1082.0 000249,sub-NV103/sub-NV103_ses-20190205T104917_ophys.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000249,sub-NV103/sub-NV103_ses-20190205T104917_ophys.nwb,pynwb_open_load_ns,PASS,1082.0 000250,sub-CS-fly#-2/sub-CS-fly#-2_behavior.nwb,matnwb_nwbRead,TIMEOUT,3721.0 000250,sub-CS-fly#-2/sub-CS-fly#-2_behavior.nwb,pynwb_open_load_ns,PASS,1609.0 000251,sub-139/sub-139_ses-Ca-VS-MB-THREE-TP-7.nwb,matnwb_nwbRead,FAIL,2159.0 000251,sub-139/sub-139_ses-Ca-VS-MB-THREE-TP-7.nwb,pynwb_open_load_ns,PASS,1576.0 000252,sub-P1/sub-P1_ses-2015-08-21_ecephys.nwb,matnwb_nwbRead,FAIL,1149.0 000252,sub-P1/sub-P1_ses-2015-08-21_ecephys.nwb,pynwb_open_load_ns,PASS,1449.0 000253,sub-647836/sub-647836_ses-1227858756_probe-1_ecephys.nwb,matnwb_nwbRead,FAIL,1149.0 000253,sub-647836/sub-647836_ses-1227858756_probe-1_ecephys.nwb,pynwb_open_load_ns,PASS,1447.0 000288,sub-H19.03.316/sub-H19.03.316_ses-910408142_icephys.nwb,pynwb_open_load_ns,PASS,3029.0 000292,sub-19o22001/sub-19o22001_icephys.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000292,sub-19o22001/sub-19o22001_icephys.nwb,pynwb_open_load_ns,PASS,1495.0 000293,sub-X2016-03-03-2016-03-03-0100/sub-X2016-03-03-2016-03-03-0100_icephys.nwb,matnwb_nwbRead,TIMEOUT,3653.0 000293,sub-X2016-03-03-2016-03-03-0100/sub-X2016-03-03-2016-03-03-0100_icephys.nwb,pynwb_open_load_ns,PASS,2511.0 000294,sub-3436/sub-3436_ses-20211011T115848_ecephys+icephys.nwb,pynwb_open_load_ns,TIMEOUT,3619.0 000295,sub-20220512002/sub-20220512002_ses-20220512T172916_slice-20220512002_cell-20220512002_icephys.nwb,matnwb_nwbRead,TIMEOUT,3610.0 000295,sub-20220512002/sub-20220512002_ses-20220512T172916_slice-20220512002_cell-20220512002_icephys.nwb,pynwb_open_load_ns,PASS,1377.0 000296,sub-487974254071810552/sub-487974254071810552_ses-20190220T200205_ophys.nwb,matnwb_nwbRead,FAIL,1146.0 000296,sub-487974254071810552/sub-487974254071810552_ses-20190220T200205_ophys.nwb,pynwb_open_load_ns,PASS,150.0 000297,sub-X2020-01-28-2020-01-28-0029/sub-X2020-01-28-2020-01-28-0029_icephys.nwb,matnwb_nwbRead,TIMEOUT,3600.0 000297,sub-X2020-01-28-2020-01-28-0029/sub-X2020-01-28-2020-01-28-0029_icephys.nwb,pynwb_open_load_ns,PASS,463.0 000298,sub-Rat203/sub-Rat203_ecephys.nwb,matnwb_nwbRead,FAIL,346.0 000298,sub-Rat203/sub-Rat203_ecephys.nwb,pynwb_open_load_ns,PASS,92.0 000299,sub-Rat203/sub-Rat203_ecephys.nwb,matnwb_nwbRead,TIMEOUT,3679.0 000299,sub-Rat203/sub-Rat203_ecephys.nwb,pynwb_open_load_ns,PASS,93.0 000301,sub-M41/sub-M41_ses-20201127T144127_behavior+ecephys.nwb,matnwb_nwbRead,TIMEOUT,3640.0 000301,sub-M41/sub-M41_ses-20201127T144127_behavior+ecephys.nwb,pynwb_open_load_ns,PASS,213.0 000302,sub-ANM-0179/sub-ANM-0179_ses-20211121_behavior+ecephys.nwb,pynwb_open_load_ns,PASS,194.0 000338,sub-002/sub-002_ses-20160101.nwb,pynwb_open_load_ns,PASS,619.0 000339,sub-BH274/sub-BH274_ses-20210806T193524_ecephys.nwb,pynwb_open_load_ns,PASS,329.0 ```

Interestingly, despite the timeout being set to 3600 seconds, a number of tests ran for longer than that (up to over two hours for some). I'm guessing that those just stalled after they were killed at the one hour mark.

yarikoptic commented 2 months ago

For me it is interesting that if I look at top, I do not see any of test processes but rather busy (up to 100% CPU) mount.davfs, and without any recent logs on it

although generally it seems to log interesting interactions ![image](https://github.com/user-attachments/assets/08b76558-f6cf-4a5b-9f95-f9a4d5a67199)

so remains unclear what davfs2 is actually doing and why IO is actually quite slow (likely slower than our dataset fuse given that I do not even see python/matlab to appear in top)

jwodder commented 2 months ago

@yarikoptic Also, many of the failed MatNWB tests emitted the error:

Error using load
Unable to read MAT-file
/home/dandi/cronlib/dandisets-healthstatus/matnwb/namespaces/core.mat. File
might be corrupt.
yarikoptic commented 2 months ago

Error using load Unable to read MAT-file /home/dandi/cronlib/dandisets-healthstatus/matnwb/namespaces/core.mat. File might be corrupt.

heh, hinting that may be even read outs could not be trusted, or it just times out internally... who knows (no source for matlab) .

jwodder commented 2 months ago

@yarikoptic The test run of this PR's code completed after 2 days and 7 hours.

Since this PR has done what it set out to do and the remaining performance problems are unrelated to it, can this PR be merged now, with the remaining performance problems left for discussion in separate issues?