search

VUIIS / dax

Distributed Automation for XNAT
MIT License
25 stars 24 forks source link

Data Pipeline Flexibility with Special Characters #406

Closed andrewpapale closed 1 year ago

andrewpapale commented 1 year ago

Hi DAX Team,

I think this is a feature request, happy to code it myself with some pointers from the team. We have scan data on XNAT that contains special characters in the folder names ">". We pull this data from XNAT to our server and then we are backing it up on SharePoint, which does not allow this character in files/folders. Therefore, to achieve backup from our server, I need to rename folders with ">" on our server. No problem, then it syncs to SharePoint just fine.

However, DAX is too smart, it thinks those folders with no ">" have not been downloaded and is now re-downloading all our data so we now have two copies of our data on our server. I would basically like to know how to partial string match with DAX in order to take into account the change in character. Thank you for your time.

baxpr commented 1 year ago

This is with Xnatdownload, yes? The download resume feature of Xnatdownload is a best-effort to save time but is not intended to be super versatile. Filtering like that would have to be implemented from the ground up, and as a practical matter I doubt we'll ever get to it ourselves.

Best I can think of for a workaround is to code up something that will tweak the filenames in the download_report.csv to match the filename changes you are making on disk. That's pretty fussy unfortunately.

baxpr commented 1 year ago

Closing issue, "won't fix". But always open to discuss