ai-dock / stable-diffusion-webui

AUTOMATIC1111 (A1111) Stable Diffusion Web UI docker images for use in GPU cloud and local environments. Includes AI-Dock base for authentication and improved user experience.
Other
110 stars 62 forks source link

civitai and wget don't want to play nice together #28

Open disposeablehero opened 3 weeks ago

disposeablehero commented 3 weeks ago

Files from civitai links seems to be having issues with wget, including the one already in the default.sh in the lora array (though commented out). The command as written works just fine when manually entered on the command line so I'm somewhat at a loss.


--2024-09-07 04:09:55--  https://civitai.com/api/download/models/125843?type=Model&format=PickleTensor
Resolving civitai.com (civitai.com)... 104.22.19.237, 172.67.12.143, 104.22.18.237, ...
Connecting to civitai.com (civitai.com)|104.22.19.237|:443... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: https://civitai-delivery-worker-prod.5ac0637cfd0766c97916cefa3764fbdf.r2.cloudflarestorage.com/642113/model/4xUltrasharp.D8rD.pt?X-Amz-Expires=86400&response-content-disposition=attachment%3B%20filename%3D%224xUltrasharp_4xUltrasharpV10.pt%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=e01358d793ad6966166af8b3064953ad/20240907/us-east-1/s3/aws4_request&X-Amz-Date=20240907T040955Z&X-Amz-SignedHeaders=host&X-Amz-Signature=d177496bd45904020f73986f79f553bdb071c383a89df8486938654040327b43 [following]
--2024-09-07 04:09:55--  https://civitai-delivery-worker-prod.5ac0637cfd0766c97916cefa3764fbdf.r2.cloudflarestorage.com/642113/model/4xUltrasharp.D8rD.pt?X-Amz-Expires=86400&response-content-disposition=attachment%3B%20filename%3D%224xUltrasharp_4xUltrasharpV10.pt%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=e01358d793ad6966166af8b3064953ad/20240907/us-east-1/s3/aws4_request&X-Amz-Date=20240907T040955Z&X-Amz-SignedHeaders=host&X-Amz-Signature=d177496bd45904020f73986f79f553bdb071c383a89df8486938654040327b43
Resolving civitai-delivery-worker-prod.5ac0637cfd0766c97916cefa3764fbdf.r2.cloudflarestorage.com (civitai-delivery-worker-prod.5ac0637cfd0766c97916cefa3764fbdf.r2.cloudflarestorage.com)... 104.18.8.90, 104.18.9.90, 2606:4700::6812:85a, ...
Connecting to civitai-delivery-worker-prod.5ac0637cfd0766c97916cefa3764fbdf.r2.cloudflarestorage.com (civitai-delivery-worker-prod.5ac0637cfd0766c97916cefa3764fbdf.r2.cloudflarestorage.com)|104.18.8.90|:443... connected.
HTTP request sent, awaiting response... 400 Bad Request
The destination name is too long (384), reducing to 236
Incomplete or invalid multibyte sequence encountered
2024-09-07 04:09:55 ERROR 400: Bad Request.
disposeablehero commented 3 weeks ago

So it's seems to be content-disposition issues with wget and signed aws redirects after the server accepts the token. Since I'm lazy I just wrote a separate if statement for HF links, but I'm wondering if there's some super simple wget setting I overlooked (curl seems to have the same issues, content-disposition just names the downloaded file "?model-blah-blah"

function provisioning_download() {
    if [[ -n $HF_TOKEN && $1 =~ ^https://([a-zA-Z0-9_-]+\.)?huggingface\.co(/|$|\?) ]]; then
        auth_token="$HF_TOKEN"

        if [[ -n $auth_token ]];then
            wget --header="Authorization: Bearer $auth_token" -qnc --content-disposition --show-progress -e dotbytes="${3:-4M}" -P "$2" "$1"
        else
            wget -qnc --content-disposition --show-progress -e dotbytes="${3:-4M}" -P "$2" "$1"
        fi
    elif 
        [[ -n $CIVITAI_TOKEN && $1 =~ ^https://([a-zA-Z0-9_-]+\.)?civitai\.com(/|$|\?) ]]; then
        auth_token="$CIVITAI_TOKEN"

        if [[ -n $auth_token ]];then
            wget -qnc --content-disposition --show-progress -e dotbytes="${3:-4M}" -P "$2" "${1}&token=${auth_token}"
        else
            wget -qnc --content-disposition --show-progress -e dotbytes="${3:-4M}" -P "$2" "$1"
        fi
    fi
}
robballantyne commented 1 week ago

I'm switching all scripts to curl which does not have this issue.