search

Australian-Imaging-Service / pipelines

Scripts to generate analysis pipelines that can be run in XNAT's container service
Other
1 stars 3 forks source link

[STORY] Upload Dan's sample data (63.7GB) to XNAT to test the de-identification script and remove the dose report, report pdf, request form “encapsulated” in a DICOM. #304

Open anzhao opened 1 month ago

anzhao commented 1 month ago

Metadata

Epic: XNAT DATA Upload Feature: Feature Release: Required knowledge: mid-level

Description

User Stories are the primary means of expressing needed functionality. Since they focus on the user as the subject of interest, and not the system, user stories are value and customer-centric. Recommended form is the "user-voice form":

As a (user role), I want to (activity), so that (business value)

By using this format, the teams are guided to understand who is using the system, what they are doing with it, and why they are doing it.

Enabler Stories are written to support exploration, architecture, or infrastructure. Enabler stories may be expressed in technical rather than user-centric language. Enabler stories are demonstrated just like user stories, typically by showing the knowledge gained, artifacts produced, or the user interface, stub, or mock-up.

Acceptance Criteria

anzhao commented 1 month ago

Adopted shell script to upload all the Dan's data at one time

#!/bin/bash

# Set your XNAT credentials
XNAT_URL="https://dev.xnat.ais.sydney.edu.au"
USERNAME="antest"
PASSWORD="antest123"

# Only need to specify the project id
PROJECT_ID="zip-001"

# List of zip file names (modify as needed)
ZIP_FILES=("7145-4.zip" "1845.zip" "1780.zip" "1688.zip" "1644.zip" "1691.zip" "1593.zip" "1294.zip" "0534.zip" "1590.zip" "1087.zip" "1005.zip" "1022.zip" "0974.zip" "0712.zip" "0924.zip" "0504.zip" "0522.zip" "0346.zip" "0453.zip" "0083.zip" "0439.zip" "0413.zip" "0064.zip")

# Loop through the zip files and upload them
for ZIP_FILES in "${ZIP_FILES[@]}"; do
   curl -o /tmp/u.tmp -u $USERNAME:$PASSWORD -k -H 'Content-Type: application/zip' -X POST "$XNAT_URL/data/services/import?inbody=true&PROJECT_ID=$PROJECT_ID&overwrite=true" -T $ZIP_FILES
done
anzhao commented 1 month ago

Use the above script to upload Dan's sample data at one time.

Image

anzhao commented 1 month ago

Update the script to check if the file is exist on XNAT server. If Not, start the upload process. If Yes, check that if the previous upload is completed by comparing file size on the server and local computer. If the size is equal, upload is completed. If not, resume the upload from previous broken point by using "curl -C -url".

!/bin/bash

Set your XNAT credentials

XNAT_URL="https://dev.xnat.ais.sydney.edu.au" USERNAME="antest" PASSWORD="antest123"

Only need to specify the project id

PROJECT_ID="compress-001"

List of zip file names

ZIP_FILES=("7145-4.zip" "1845.zip" "1780.zip" "1688.zip" "1644.zip" "1691.zip" "1593.zip" "1294.zip" "0534.zip" "1590.zip" "1087.zip" "1005.zip" "1022.zip" "0974.zip" "0712.zip" "0924.zip" "0504.zip" "0522.zip" "0346.zip" "0453.zip" "0083.zip" "0439.zip" "0413.zip" "0064.zip")

Loop through the zip files and upload them

for ZIP_FILE in "${ZIP_FILES[@]}"; do

Check if the file exists on the server

FILE_EXISTS=$(curl -s -o /dev/null -w "%{http_code}" -u $USERNAME:$PASSWORD -k -X HEAD "$XNAT_URL/data/archive/projects/$PROJECT_ID/subjects/$ZIP_FILE")
if [ "$FILE_EXISTS" == "200" ]; then
    # File exists, check if upload is complete
    SERVER_SIZE=$(curl -s -u $USERNAME:$PASSWORD -k -X GET "$XNAT_URL/data/archive/projects/$PROJECT_ID/subjects/$ZIP_FILE" | grep -oP 'size="\K[^"]+')
    LOCAL_SIZE=$(stat -c %s $ZIP_FILE)
    if [ "$SERVER_SIZE" == "$LOCAL_SIZE" ]; then
        echo "File $ZIP_FILE already uploaded."
    else
        # Resume upload
        curl -o /tmp/u.tmp -u $USERNAME:$PASSWORD -k -H 'Content-Type: application/zip' -C - -X POST "$XNAT_URL/data/services/import?inbody=true&PROJECT_ID=$PROJECT_ID&overwrite=true" -T $ZIP_FILE
    fi
else
    # File does not exist, upload it
    curl -o /tmp/u.tmp -u $USERNAME:$PASSWORD -k -H 'Content-Type: application/zip' -X POST "$XNAT_URL/data/services/import?inbody=true&PROJECT_ID=$PROJECT_ID&overwrite=true" -T $ZIP_FILE
fi

done

anzhao commented 1 week ago

Our Dev server is able to accept the large file now, for instance a big Zip file of 63.7GB in a single upload. However, it is not received for some reason. I guess that the upload large zip file has reached the pre-archive staging space of XNAT, but it fails in the process of read DICOM metadata and XML session build phase. Image

anzhao commented 2 days ago

The Windows BAT file for Dan to run on his windows machine

@echo off setlocal enabledelayedexpansion

:: Set your XNAT credentials set XNAT_URL=https://dev.xnat.ais.sydney.edu.au set USERNAME=antest set PASSWORD=antest

:: Only need to specify the project id set PROJECT_ID=aildrtest-001

:: List of zip file names set ZIP_FILES=7145-4.zip 1845.zip 1780.zip 1688.zip 1644.zip 1691.zip 1593.zip 1294.zip 0534.zip 1590.zip 1087.zip 1005.zip 1022.zip 0974.zip 0712.zip 0924.zip 0504.zip 0522.zip 0346.zip 0453.zip 0083.zip 0439.zip 0413.zip 0064.zip

:: Loop through the zip files and upload them for %%Z in (%ZIP_FILES%) do ( :: Check if the file exists on the server set ZIP_FILE=%%Z set zip_file_without_extension=!ZIP_FILE:.zip=! for /f %%H in ('curl -s -o NUL -w "%%{http_code}" -u %USERNAME%:%PASSWORD% -k -X HEAD "%XNAT_URL%/data/archive/projects/%PROJECT_ID%/subjects/!zip_file_without_extension!"') do set FILE_EXISTS=%%H if "!FILE_EXISTS!"=="200" ( :: File exists, check if upload is complete for /f %%S in ('curl -s -u %USERNAME%:%PASSWORD% -k -X GET "%XNAT_URL%/data/archive/projects/%PROJECT_ID%/subjects/!zip_file_without_extension!" ^| findstr /R "size=\"[0-9]\""') do set SERVER_SIZE=%%S set SERVER_SIZE=!SERVER_SIZE:size="=! set SERVER_SIZE=!SERVER_SIZE:~0,-1! for /f "usebackq" %%L in (powershell -command "(Get-Item !ZIP_FILE!).length") do set LOCAL_SIZE=%%L if "!SERVER_SIZE!"=="!LOCAL_SIZE!" ( echo File !ZIP_FILE! already uploaded. ) else ( :: Resume upload curl -o NUL -u %USERNAME%:%PASSWORD% -k -H "Content-Type: application/zip" -C - -X POST "%XNAT_URL%/data/services/import?inbody=true&PROJECT_ID=%PROJECT_ID%&overwrite=true" -T !ZIP_FILE! ) ) else ( :: File does not exist, upload it curl -o NUL -u %USERNAME%:%PASSWORD% -k -H "Content-Type: application/zip" -X POST "%XNAT_URL%/data/services/import?inbody=true&PROJECT_ID=%PROJECT_ID%&overwrite=true" -T !ZIP_FILE! ) )

endlocal