search

gazebo-tooling / release-tools

8 stars 9 forks source link

ign-msgs6-debbuilder job fails repeatedly to verify certificate when downloading tarball on optimus #414

Open scpeters opened 3 years ago

scpeters commented 3 years ago

The ubuntu/focal::armhf build of ignition-msgs6 has been failing consistently on the linux-optimus.focal-b9b55ab5 jenkins node.

I've proposed a change in https://github.com/ignition-tooling/release-tools/pull/413 that gives some additional debugging info indicating that it is failing to verify a certificate for osrf-distributions.s3.amazonaws.com

+ wget --quiet -O orig_tarball https://osrf-distributions.s3.amazonaws.com/ign-msgs/releases/ignition-msgs6-6.4.0.tar.bz2
+ echo rerunning wget without --quiet since it failed
rerunning wget without --quiet since it failed
+ wget -O orig_tarball https://osrf-distributions.s3.amazonaws.com/ign-msgs/releases/ignition-msgs6-6.4.0.tar.bz2
--2021-03-04 17:23:23--  https://osrf-distributions.s3.amazonaws.com/ign-msgs/releases/ignition-msgs6-6.4.0.tar.bz2
Resolving osrf-distributions.s3.amazonaws.com (osrf-distributions.s3.amazonaws.com)... 54.231.49.40
Connecting to osrf-distributions.s3.amazonaws.com (osrf-distributions.s3.amazonaws.com)|54.231.49.40|:443... connected.
ERROR: cannot verify osrf-distributions.s3.amazonaws.com's certificate, issued by ‘CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US’:
  Unable to locally verify the issuer's authority.
To connect to osrf-distributions.s3.amazonaws.com insecurely, use `--no-check-certificate'.
Build step 'Execute shell' marked build as failure
j-rivero commented 3 years ago

I think this is the first time I see it. It's an Amazon S3 certificate, weird that can not be revolve correctly by Docker containers. A possible workaround could be to relaunch the builds failing removing the S from httpS://osrf-distributions.s3.amazonaws.com/....

Failing build are in Focal and Sid, Sid is old and could be the case that the cert resolution is not valid there but I would not expect it from Focal.

scpeters commented 3 years ago

yeah, I have no idea what is happening here. could it be a squid cache issue?

scpeters commented 3 years ago

it just failed for ign-common3 focal:armhf as well

scpeters commented 3 years ago

it just failed for ign-common3 focal:armhf as well

the focal:armhf build of ign-common3 just succeeded on drogon. I believe this failure only occurs on the optimus machine

j-rivero commented 3 years ago

If I run tests in optimus base system, things seems fine:

jenkins@optimus:/tmp$  wget -O orig_tarball2 https://osrf-distributions.s3.amazonaws.com/ign-msgs/releases/ignition-msgs6-6.4.0.tar.bz2
--2021-03-09 19:02:40--  https://osrf-distributions.s3.amazonaws.com/ign-msgs/releases/ignition-msgs6-6.4.0.tar.bz2
Resolving osrf-distributions.s3.amazonaws.com (osrf-distributions.s3.amazonaws.com)... 52.216.226.144
Connecting to osrf-distributions.s3.amazonaws.com (osrf-distributions.s3.amazonaws.com)|52.216.226.144|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 312460 (305K) [application/x-bzip2]
Saving to: ‘orig_tarball2’

orig_tarball2                               100%[===========================================================================================>] 305.14K   993KB/s    in 0.3s    

2021-03-09 19:02:40 (993 KB/s) - ‘orig_tarball2’ saved [312460/312460]

Might be related to disk problems in optimus https://github.com/osrf/buildfarmer/issues/167

scpeters commented 3 years ago

another failure on optimus: https://build.osrfoundation.org/job/ign-gui3-debbuilder/58/console

scpeters commented 3 years ago

another failure on optimus: https://build.osrfoundation.org/job/ign-gui3-debbuilder/68/console

scpeters commented 3 years ago

a workaround was merged in #423