VUIIS / dax

Distributed Automation for XNAT
MIT License
25 stars 24 forks source link

Xnatdownload repeated crashes on large downloads #430

Open baxpr opened 1 year ago

baxpr commented 1 year ago

Been having a lot of problems with extended (hours-long) Xnatdownload requests failing entirely due to connection timeout or connection failure. Here's a sketch of a way to download an assessor resource much more reliably at the cost of more XNAT sessions being created:

import os

from dax import XnatUtils

with XnatUtils.get_interface() as xnat:
    assrs = xnat.list_project_assessors('ABCD')
assrs = [a for a in assrs if a['proctype']=='cersuit_v3']

for a in assrs:

    print(a['label'])

    outdir = (
        'cersuit_v3/' + 
        a['project_id'] + '-x-' + 
        a['subject_label'] + '-x-' + 
        a['session_label'] + '-x-' + 
        a['label']  
        )
    if os.path.exists(outdir):
        print('    WARNING: Outdir exists - skipping')
        continue
    else:
        os.mkdir(outdir)

    try:
        with XnatUtils.get_interface() as xnat:
            rsrc = xnat.select_assessor_resource(
                a['project_id'],
                a['subject_label'],
                a['session_label'],
                a['label'],
                'VOLS_NATIVE',
                )
            rsrc.get(outdir, extract=True)
    except:
        print('    WARNING: Download failed')
baxpr commented 1 year ago

I think the underlying issue is really that Xnatdownload doesn't handle or recover from intermittent connection failures.