StackStorm / st2-packages

StackStorm deb/rpm packages (automated docker build pipeline)
https://stackstorm.com/
27 stars 59 forks source link

Adding -L to all curl commands in scripts/ so that we follow redirects #647

Closed nmaludy closed 4 years ago

nmaludy commented 4 years ago

Debugging st2 e2e tests and ran across this error:

20200504T170709+0000 dpkg-deb: error: 'st2_3.3dev-1_amd64.deb' is not a debian format archive\n20200504T170709+0000 dpkg: error processing archive st2_3.3dev-1_amd64.deb (--install):\n20200504T170709+0000  subprocess dpkg-deb --control returned error exit status 2\n20200504T170709+0000 Errors were encountered while processing:\n20200504T170709+0000  st2_3.3dev-1_amd64.deb\n20200504T170709+0000 ############### ERROR ###############\n20200504T170709+0000 # Failed on Install st2 #\n20200504T170709+0000 #####################################"

I was able to reproduce it locally with st2 vagrant using the following (the build number came from the st2e2e console in the dev_build parameter):

$ REPO_TYPE=staging RELEASE=unstable DEV=st2/11285 BOX=generic/ubuntu1604 vagrant up

When i ran this command i got the same error from the e2e tests. When i logged into the vagrant box i noticed this:

# weird that size of the file is 485 bytes
vagrant@st2vagrant:~$ ls -l
-rw-rw-r-- 1 vagrant vagrant   485 May  4 17:24 st2_3.3dev-1_amd64.deb

# weird it's an ascii file
vagrant@st2vagrant:~$ file st2_3.3dev-1_amd64.deb 
st2_3.3dev-1_amd64.deb: ASCII text, with very long lines

# looks like it's a redirect message from curl
vagrant@st2vagrant:~$ cat st2_3.3dev-1_amd64.deb 
Redirecting to <https://circle-production-customer-artifacts.s3.amazonaws.com/picard/forks/552b8335f7d039bccaa2396e/87355177/nmaludy/st2/5eb03a756a6fa85be029969b-0-build/artifacts/packages/xenial/st2_3.3dev-1_amd64.deb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20200504T173033Z&X-Amz-SignedHeaders=host&X-Amz-Expires=60&X-Amz-Credential=AKIAJR3Q6CR467H7Z55A%2F20200504%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=9c5aef856c5bca2401f0e548662db4a7963eed9fbd922a5aee539e86c142d3cc>

# running curl manually
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 52.71.209.173...
* Connected to 11285-19051806-gh.circle-artifacts.com (52.71.209.173) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification SKIPPED
*        server certificate status verification SKIPPED
*        common name: *.circle-artifacts.com (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: CN=*.circle-artifacts.com
*        start date: Thu, 19 Sep 2019 00:00:00 GMT
*        expire date: Mon, 19 Oct 2020 12:00:00 GMT
*        issuer: C=US,O=Amazon,OU=Server CA 1B,CN=Amazon
*        compression: NULL
* ALPN, server did not agree to a protocol
> GET /0/packages/xenial/st2_3.3dev-1_amd64.deb HTTP/1.1
> Host: 11285-19051806-gh.circle-artifacts.com
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 302 Found
< Date: Mon, 04 May 2020 17:30:33 GMT
< Location: https://circle-production-customer-artifacts.s3.amazonaws.com/picard/forks/552b8335f7d039bccaa2396e/87355177/nmaludy/st2/5eb03a756a6fa85be029969b-0-build/artifacts/packages/xenial/st2_3.3dev-1_amd64.deb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20200504T173033Z&X-Amz-SignedHeaders=host&X-Amz-Expires=60&X-Amz-Credential=AKIAJR3Q6CR467H7Z55A%2F20200504%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=9c5aef856c5bca2401f0e548662db4a7963eed9fbd922a5aee539e86c142d3cc
< Server: nginx
< Set-Cookie: ring-session=o1BAjS9EDJO0LotvnI29stgxISlk90cCXYHdJnJ1Fbw%3D--MzKPTQg7hRvb3o6FD0NT3s1aGI0dvTQK6Q0tc10MLWw%3D;Path=/;HttpOnly;Expires=Tue, 04 May 2021 09:43:58 +0000;Max-Age=31536000;Secure
< X-CircleCI-Identity: artifacts-service-v1-6666757d5c-j5nc4
< X-Rate-Limit-Applied: yes
< X-Rate-Limit-Result: PASSED
< Content-Length: 485
< Connection: keep-alive
< 
{ [485 bytes data]
100   485  100   485    0     0   1122      0 --:--:-- --:--:-- --:--:--  1122
* Connection #0 to host 11285-19051806-gh.circle-artifacts.com left intact

######################################

I then noticed that curl wasn't being given the -L parameter to follow redirects... Adding the -L parameter into st2bootstrap-deb.sh fixed the problem.

######################################

After all of this here is the grep of the current curl commands from the scripts directory:

$ grep -r curl scripts/
scripts/build_python.sh:curl -sSL http://www.python.org/ftp/python/$pyver/Python-$pyver.tar.xz -o \
scripts/migrate_bintray_to_pkgcloud_trusty.sh:curl -sSL https://packagecloud.io/install/repositories/StackStorm/staging-stable/script.deb.sh | sudo bash
scripts/includes/common.sh:  PACKAGES_METADATA=$(curl -sSL -q https://circleci.com/api/v1.1/project/github/StackStorm/${DEV_BUILD}/artifacts)
scripts/st2bootstrap-el6.sh:  PACKAGES_METADATA=$(curl -sSL -q https://circleci.com/api/v1.1/project/github/StackStorm/${DEV_BUILD}/artifacts)
scripts/st2bootstrap-el6.sh:  sudo yum -y install curl rabbitmq-server
scripts/st2bootstrap-el6.sh:  curl -sL https://packagecloud.io/install/repositories/StackStorm/${REPO_PREFIX}${RELEASE}/script.rpm.sh | sudo bash
scripts/st2bootstrap-el6.sh:  curl -sL https://rpm.nodesource.com/setup_10.x | sudo -E bash -
scripts/st2bootstrap-el8.template.sh:  curl -sL https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/script.rpm.sh | sudo bash
scripts/st2bootstrap-el8.template.sh:  sudo yum -y install curl rabbitmq-server
scripts/st2bootstrap-el8.template.sh:  curl -sL https://packagecloud.io/install/repositories/StackStorm/${REPO_PREFIX}${RELEASE}/script.rpm.sh | sudo bash
scripts/st2bootstrap-el8.template.sh:  curl -sL https://rpm.nodesource.com/setup_10.x | sudo -E bash -
scripts/setup-vagrant.sh:sudo $INSTALL_CMD install -y git curl wget
scripts/setup-vagrant.sh:  sudo sh -c "curl -sL $DC_URL > $DC_BIN"
scripts/st2_bootstrap.sh:hash curl 2>/dev/null || { echo >&2 "'curl' is not installed. Aborting."; exit 1; }
scripts/st2_bootstrap.sh:CURLTEST=`curl --output /dev/null --silent --head --fail ${ST2BOOTSTRAP}`
scripts/st2_bootstrap.sh:    curl -sSL -k -o ${BOOTSTRAP_FILE} ${ST2BOOTSTRAP}
scripts/st2bootstrap-deb.template.sh:  # Note: gnupg-curl is needed to be able to use https transport when fetching keys
scripts/st2bootstrap-deb.template.sh:    sudo apt-get install -y gnupg-curl
scripts/st2bootstrap-deb.template.sh:  sudo apt-get install -y curl
scripts/st2bootstrap-deb.template.sh:  curl -sL https://packagecloud.io/install/repositories/StackStorm/${REPO_PREFIX}${RELEASE}/script.deb.sh | sudo bash
scripts/st2bootstrap-deb.template.sh:    curl -sSL -k -o ${PACKAGE_FILENAME} ${PACKAGE_URL}
scripts/st2bootstrap-deb.template.sh:    curl -sSL -k -o ${PACKAGE_FILENAME} ${PACKAGE_URL}
scripts/st2bootstrap-deb.template.sh:  curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
scripts/st2bootstrap-el6.template.sh:  sudo yum -y install curl rabbitmq-server
scripts/st2bootstrap-el6.template.sh:  curl -sL https://packagecloud.io/install/repositories/StackStorm/${REPO_PREFIX}${RELEASE}/script.rpm.sh | sudo bash
scripts/st2bootstrap-el6.template.sh:  curl -sL https://rpm.nodesource.com/setup_10.x | sudo -E bash -
scripts/st2bootstrap-el7.template.sh:  sudo yum -y install curl rabbitmq-server
scripts/st2bootstrap-el7.template.sh:  curl -sL https://packagecloud.io/install/repositories/StackStorm/${REPO_PREFIX}${RELEASE}/script.rpm.sh | sudo bash
scripts/st2bootstrap-el7.template.sh:  curl -sL https://rpm.nodesource.com/setup_10.x | sudo -E bash -
scripts/st2bootstrap-el8.sh:  PACKAGES_METADATA=$(curl -sSL -q https://circleci.com/api/v1.1/project/github/StackStorm/${DEV_BUILD}/artifacts)
scripts/st2bootstrap-el8.sh:  curl -sL https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/script.rpm.sh | sudo bash
scripts/st2bootstrap-el8.sh:  sudo yum -y install curl rabbitmq-server
scripts/st2bootstrap-el8.sh:  curl -sL https://packagecloud.io/install/repositories/StackStorm/${REPO_PREFIX}${RELEASE}/script.rpm.sh | sudo bash
scripts/st2bootstrap-el8.sh:  curl -sL https://rpm.nodesource.com/setup_10.x | sudo -E bash -
scripts/st2bootstrap-el7.sh:  PACKAGES_METADATA=$(curl -sSL -q https://circleci.com/api/v1.1/project/github/StackStorm/${DEV_BUILD}/artifacts)
scripts/st2bootstrap-el7.sh:  sudo yum -y install curl rabbitmq-server
scripts/st2bootstrap-el7.sh:  curl -sL https://packagecloud.io/install/repositories/StackStorm/${REPO_PREFIX}${RELEASE}/script.rpm.sh | sudo bash
scripts/st2bootstrap-el7.sh:  curl -sL https://rpm.nodesource.com/setup_10.x | sudo -E bash -
scripts/st2bootstrap-deb.sh:  PACKAGES_METADATA=$(curl -sSL -q https://circleci.com/api/v1.1/project/github/StackStorm/${DEV_BUILD}/artifacts)
scripts/st2bootstrap-deb.sh:  # Note: gnupg-curl is needed to be able to use https transport when fetching keys
scripts/st2bootstrap-deb.sh:    sudo apt-get install -y gnupg-curl
scripts/st2bootstrap-deb.sh:  sudo apt-get install -y curl
scripts/st2bootstrap-deb.sh:  curl -sL https://packagecloud.io/install/repositories/StackStorm/${REPO_PREFIX}${RELEASE}/script.deb.sh | sudo bash
scripts/st2bootstrap-deb.sh:    curl -sSL -k -o ${PACKAGE_FILENAME} ${PACKAGE_URL}
scripts/st2bootstrap-deb.sh:    curl -sSL -k -o ${PACKAGE_FILENAME} ${PACKAGE_URL}
scripts/st2bootstrap-deb.sh:  curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -