Dataset download issue

[AntiLibrary5]

Hi, To download the data from: https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=93258287 following the instructions for restricted data, I signed and submitted the TCIA Restricted License Agreement to "help@cancerimagingarchive.net" and got the approval and downloaded the full manifest file TCIA_FDG-PET-CT-Lesions_v1.tcia. In Ubuntu 22, using NBIA data retriever, I run the following command:

/opt/nbia-data-retriever/nbia-data-retriever --cli <location>/<manifest file name>.tcia -d <parent location>/<download directory> -l <location>/<credential file> -v –f

where I created a credentials text file with username and password for TCIA account.

I get the following error stored in logs:

2023-08-20 12:59:31: INFOS: Using manifiest file: /media/varora/LaCie1/Datasets/Atlas/TCIA_FDG-PET-CT-Lesions_v1.tcia

2023-08-20 12:59:31: INFOS: Running with option: quiet = false; verbose = true; force = true

2023-08-20 12:59:31: INFOS: The type of data downloading is DICOM

2023-08-20 12:59:34: PRÉCIS: connecting server: https://public.cancerimagingarchive.net/nbia-api/services/v1/getDeniedSeries?userName=varora&format=csv&seriesList=1.3.6.1.4.1.14519.5.2.1.4219.6651.146136900202431607373455344501,1.3.6.1.4.1.14519.5.2.1.4219.6651.306383459115118250894914293876,1.3.6.1.4.1.14519.5.2.1.4219.6651.182694110205155645649641184770,1.3.6.1.4.1.14519.5.2.1.4219.6651.133578453901173938694448827319,1.3.6.1.4.1.14519.5.2.1.4219.6651.913380603297407488136321844830,1.3.6.1.4.1.14519.5.2.1.4219.6651.113796169149340866789231604813,1.3.6.1.4.1.14519.5.2.1.4219.6651.127912095193700175403024940093,1.3.6.1.4.1.14519.5.2.1.4219.6651.497063096551936002076692470693,1.3.6.1.4.1.14519.5.2.1.4219.6651.151896746116030282379254412823,1.3.6.1.4.1.14519.5.2.1.4219.6651.151532511484742028529393389495,1.3.6.1.4.1.14519.5.2.1.4219.6651.313152023416660761769565814535,1.3.6.1.4.1.14519.5.2.1.4219.6651.340214977928587094922740368418,1.3.6.1.4.1.14519.5.2.1.4219.6651.691945556059439587815323672556,1.3.6.1.4.1.14519.5.2.1.4219.6651.139507561280964517902282778014,1.3.6.1.4.1.14519.5.2.1.4219.6651.544791134209643999492781802579

2023-08-20 12:59:34: GRAVE: Failed to verify the access permission: java.lang.RuntimeException: Failed to validate the access permission: HTTP Error code : 500

2023-08-20 12:59:34: PRÉCIS: connecting server: https://public.cancerimagingarchive.net/nbia-api/services/v1/getDeniedSeries?userName=varora&format=csv&seriesList=1.3.6.1.4.1.14519.5.2.1.4219.6651.201203836389382194960900561337,1.3.6.1.4.1.14519.5.2.1.4219.6651.331726561677742619635617278581,1.3.6.1.4.1.14519.5.2.1.4219.6651.685145825998226094117238219181,1.3.6.1.4.1.14519.5.2.1.4219.6651.277639348431922319317937152931,1.3.6.1.4.1.14519.5.2.1.4219.6651.330088744877758372691803842232,1.3.6.1.4.1.14519.5.2.1.4219.6651.189521095776369562861890283515,1.3.6.1.4.1.14519.5.2.1.4219.6651.419767891745145439832970906166,1.3.6.1.4.1.14519.5.2.1.4219.6651.125265645176850228733886964714,1.3.6.1.4.1.14519.5.2.1.4219.6651.685312158373853509427026630526,1.3.6.1.4.1.14519.5.2.1.4219.6651.593353848660307315437305183810,1.3.6.1.4.1.14519.5.2.1.4219.6651.235484849649395128026078131045,1.3.6.1.4.1.14519.5.2.1.4219.6651.246595591547608881806205488885,1.3.6.1.4.1.14519.5.2.1.4219.6651.262586358581088125198298057487,1.3.6.1.4.1.14519.5.2.1.4219.6651.262875085424462314166764200442,1.3.6.1.4.1.14519.5.2.1.4219.6651.121172146471849680989100182901

2023-08-20 12:59:34: GRAVE: Failed to verify the access permission: java.lang.RuntimeException: Failed to validate the access permission: HTTP Error code : 500

2023-08-20 12:59:35: PRÉCIS: connecting server: https://public.cancerimagingarchive.net/nbia-api/services/v1/getDeniedSeries?userName=varora&format=csv&seriesList=1.3.6.1.4.1.14519.5.2.1.4219.6651.672999922549778318526020691011,1.3.6.1.4.1.14519.5.2.1.4219.6651.131448367855426076823217865738,1.3.6.1.4.1.14519.5.2.1.4219.6651.140824643058081559707839692085,1.3.6.1.4.1.14519.5.2.1.4219.6651.279442543211378829429098963054,1.3.6.1.4.1.14519.5.2.1.4219.6651.184208060209777209241690290085,1.3.6.1.4.1.14519.5.2.1.4219.6651.581558721638197352506561181767,1.3.6.1.4.1.14519.5.2.1.4219.6651.218952120440093022753341974503,1.3.6.1.4.1.14519.5.2.1.4219.6651.210163909385691548062581851543,1.3.6.1.4.1.14519.5.2.1.4219.6651.302097497514886099793093446983,1.3.6.1.4.1.14519.5.2.1.4219.6651.129987221039556543811558877407,1.3.6.1.4.1.14519.5.2.1.4219.6651.530790904803463349957785981979,1.3.6.1.4.1.14519.5.2.1.4219.6651.225589533585970784254977923134,1.3.6.1.4.1.14519.5.2.1.4219.6651.243982781039942768259194212076,1.3.6.1.4.1.14519.5.2.1.4219.6651.879477722437029865415700462577,1.3.6.1.4.1.14519.5.2.1.4219.6651.313672911677581977637636138363

2023-08-20 12:59:35: GRAVE: Failed to verify the access permission: java.lang.RuntimeException: Failed to validate the access permission: HTTP Error code : 500

Can you confirm if these are the right steps? How to resolve the issue? Also additional info in the man README would be helpful to all.

Thank you!

[alexanderjaus]

Hi AntiLibrary5,

Thank you very much for reaching out, I am experiencing the same problem at the moment. It would be best to check with TCIA officials. An alternative solution could be to download the images from the current AutoPET challenge in Nifti format https://autopet-ii.grand-challenge.org/dataset/ I'll leave this issue open for the moment since this would only be a temporary solution.

[AntiLibrary5]

Hi. Thanks for the response. I had contacted TCIA and they initially thought it was an issue of my credentials in their database but after correcting it, they had no idea. But suggested trying with API for restricted collections but I could not follow up:

https://wiki.cancerimagingarchive.net/display/Public/NBIA+Advanced+REST+API+Guide#NBIAAdvancedRESTAPIGuide-RequestingaTokentoUsewithRestrictedData

Thanks for the link, I'll use that. I am assuming the full dataset from TCIA = train+test set in the AutoPET challenge (access to only train set)? But its good enough. Thank you.

EDIT: Download speed is too slow when using the challenge link: http://www.midaslab.org/autoPET/data/nifti.zip

Best.

[alexanderjaus]

Hi, Thanks for sharing this. I gave the TCIA approach another try this morning and it seems to work right now. They probably fixed this issue on their side after you contacted them. So this may be another option to download the data

Regarding TCIA vs challenge data, I'd assume it is the same data and they have a private testset on which the evaluation is performed.

EDIT: The GUI NBIA data Retriever seems to work as well.

I'd close this issue for now, as the primary problem seems to be resolved at the moment.