check_new_sessions isn't archiving scans_to_review.csv under some conditions

sibis-platform / ncanda-data-integration

This is the Data Integration, MRI, and Bioinformatics Component of the National Consortium on Alcohol and NeuroDevelopment in Adolescence (NCANDA), funded by the NIAAA.

https://www.nitrc.org/projects/ncanda-datacore

BSD 3-Clause "New" or "Revised" License

4 stars 10 forks source link

check_new_sessions isn't archiving scans_to_review.csv under some conditions #396

Closed shippy closed 3 years ago

shippy commented 3 years ago

I think I see what's happening:

On 676, visual QC gets uploaded unconditionally
But the file only gets archived on 1174, which is invoked if and only if (a) args.qc_csv is non-empty, (b) there is at least one new questionable scan, (c) the file exists (...which is not the problem), (d) file is non-empty.

To resolve:

[x] Move the archival logic to 676, or to the visual_qc module
[x] Move the "is there anything to archive" logic, too

shippy commented 3 years ago

Additionally, Kilian requested that upload_visual_qc.csv should check that no other flag has been set on the file before it uploads the review flag, which is tantamount to gating this section of upload_visual_qc.py with a scan.get('quality') == "unknown".

I'm not sure that the gating shouldn't be positive (i.e. only don't upload if the scan is already marked unusable), so I'll double-check with @kipohl once available.

shippy commented 3 years ago

I've done the latter, which should be correct based on my notes, and filed a PR.