Automating downloads from launchpad with CI

aepfli commented 3 years ago

I am not sure if this is even the right place to ask, if not, please let me know, where i should instead ask :)

I want to download the SAP Commerce artifacts via CI so i can easier distribute them with my co workers. We are using a Package Registry tool for storing most of our artifacts. To reduce the overhead, we thought it is a good ides to use our CI infrastructure to download via CURL and store it in the Package registry.

Sadly this seems to be not working, i assume it is related to our infrastructure within GCP. But if i use the same docker image locally for downloading, i have no issues at all downloading. But if i use it with the CI system, i am always redirected to a page via AkamaiGHost and my first request is a Temporarly Moved request.

I thought i ask here based on https://github.com/SAP/commerce-gradle-plugin/blob/master/docs/FAQ.md#faq - as this was my main orientation for the tooling. Maybe somebody else already had this problem, and found a solution.

Thank you

aepfli commented 3 years ago


BASIC_AUTH="$(echo -n "${SAP_USER_ID}:${SAP_PASSWORD}" | base64)"

curl -L -v -b cookies.txt  \
    -H "Authorization: Basic $BASIC_AUTH" \
    -o "download.file" \
    "https://softwaredownloads.sap.com/file/${SAP_FILE_ID}"

this is the magic i am using

mpern commented 3 years ago

Hm, that's strange.

I just ran a little test on my end using exactly the script you provided and I can download 2011.8 without a hitch.

Some food for thought:

Have you tested the particular S-User in the browser too?
Missing shell escaping?
Try using -u "$SAP_USER_ID:$SAP_PASSWORD" instead of building the Auth header manually. Maybe it helps?

A couple of redirects until the download actually starts are expected though.

SAP_USER_ID='S0123456789'
SAP_PASSWORD='<redacted>'
SAP_FILE_ID='0020000000820402021'

curl -L -v -b cookies.txt  \
    -u "$SAP_USER_ID:$SAP_PASSWORD" \
    -o "download.file" \
    "https://softwaredownloads.sap.com/file/${SAP_FILE_ID}"

aepfli commented 3 years ago

here is the strange thing, the basic auth header, the url everything is the same when i am inspecting the log closely. So i do not assume it has something to do with the Script or the data.

This is the log from locally:

> GET /file/0020000000820402021 HTTP/1.1
> Host: softwaredownloads.sap.com
> User-Agent: curl/7.61.1
> Accept: */*
> Authorization: Basic <stripped>

< HTTP/1.1 302 Found
< Date: Mon, 21 Jun 2021 13:46:35 GMT
* Added cookie JSESSIONID=<stripped> for domain softwaredownloads.sap.com, path /, expire 0
< Set-Cookie: JSESSIONID=<stripped> Path=/; Secure; HttpOnly
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Location: https://softwaredownloads.sap.com/?file=0020000000820402021&downloadId=05672df4-7d4d-4599-b9d1-fc9f35a9ada7
< DownloadId: 05672df4-7d4d-4599-b9d1-fc9f35a9ada7
< Content-Type: text/html;charset=utf-8
< Transfer-Encoding: chunked
< Server: SAP Software Download

compared to CI system:

> GET /file/0020000000820402021 HTTP/1.1
> Host: softwaredownloads.sap.com
> User-Agent: curl/7.61.1
> Accept: */*
> Authorization: Basic <stripped>

< HTTP/1.1 302 Moved Temporarily
< Server: AkamaiGHost
< Content-Length: 0
< Location: https://origin.softwaredownloads.sap.com/tokengen/?file=0020000000820402021
< Expires: Wed, 23 Jun 2021 08:48:43 GMT
< Cache-Control: max-age=0, no-cache, no-store
< Pragma: no-cache
< Date: Wed, 23 Jun 2021 08:48:43 GMT
< Connection: keep-alive
< Strict-Transport-Security: max-age=31536000

Might this be related to the machines hosted in GCP and a special treatment on Akamai? After the temporarly moved, i am getting forwarded to the token page (but without my auth header) and i get a form which points me to the SSO page.

<?xml version="1.0"?>
<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8"/>
            </head>
    <body onload="document.forms[0].submit()">
        <noscript>
            <p>
                <strong>Note:</strong> Since your browser does not support JavaScript,
                you must press the Continue button once to proceed.
            </p>
        </noscript>

        <form action="https://accounts.sap.com/saml2/idp/sso" method="post">
            <div>
                <input type="hidden" name="RelayState" value="https://origin.softwaredownloads.sap.com/tokengen/?file=0020000000820402021"/>                
                <input type="hidden" name="SAMLRequest" value="<stripped>"/>                

            </div>
            <noscript>
                <div>
                    <input type="submit" value="Continue"/>
                </div>
            </noscript>
        </form>
            </body>
</html>

i even tried to do the login via curl and mimic the redirect functionality, but that is not as easy as i hoped it will be. (after providing the j_username i get an error that it was not successful, and i should clean everything and start over)

Worst case i thought about tunneling my request through another machine, eg. mine to see if this works, to verify my GCP IP theory - but that is some effort, i am not sure i currently have time for :)

mpern commented 3 years ago

At that stage, your guess is as good as mine.

But maybe there is some IP filtering going on, I honestly don't know.

For me at takes 2 redirects until I end up at the file. Do you cache any cookies by chance?

aepfli commented 3 years ago

I hope that you maybe can tell me a fancy header to bypass this ;)

I will see if I find a workaround. And I will also add here my progress, if I make some (hopefully)

Thank you for your input.

On Fri, Jun 25, 2021, 12:32 Markus Perndorfer @.***> wrote:

At that stage, your guess is as good as mine.

But maybe there is some IP filtering going on, I honestly don't know.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SAP/commerce-gradle-plugin/issues/28#issuecomment-868404067, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACMGKQR34JSCBL4MM2ELSZ3TURLNFANCNFSM47FCVNIA .

aepfli commented 3 years ago

it seems like i managed to bypass this by first calling the token endpoint manually.

BASIC_AUTH="$(echo -n "${SAP_USER_ID}:${SAP_PASSWORD}" | base64)"

curl -L -v -b cookies.txt  -c cookie.txt \
    -H "Authorization: Basic $BASIC_AUTH" \
    -o "download.file" \
    "https://origin.softwaredownloads.sap.com/tokengen/?file=${SAP_FILE_ID}"

curl -L -v -b cookies.txt  -c cookie.txt \
    -H "Authorization: Basic $BASIC_AUTH" \
    -o "download.file" \
    "https://softwaredownloads.sap.com/file/${SAP_FILE_ID}"

aepfli commented 3 years ago

as i found i workarround i close this ticket - thank you @mpern for your support

mpern commented 3 years ago

Thank you for taking the time to find a workaround.

One other thing came to mind:

Try using --location-trusted (send Authorization header on every redirect) instead of -L and see if it changes the behaviour.

Background:

SAP / commerce-gradle-plugin

Automating downloads from launchpad with CI #28