Closed austinpray closed 9 months ago
Same here. When running docker-compose with latest google SDK, showing the similiar message when pulling base image from gcr. The workaround by using 297.0.1 got it work.
Hi @austinpray
Looking at the runner env OPENSSL_1_1_1
seems to be installed. Does 299.0.0 work? I will try to see if something on the gcloud side may have changed.
related https://github.com/actions/virtual-environments/issues/1106
I tried a minimal repro here based on https://github.com/actions/virtual-environments/issues/1106 which seemed to work with 298
.
Same issue. 298 did not seem to work for me. 297.0.01 worked.
Confirming that this issue also exists in Ubuntu 20.04 with gcloud version 301.0.0.
To fix:
sudo apt install google-cloud-sdk=297.0.1-0
Confirming that the original sighting of this still happens on the latest runner: https://github.com/austinpray/kaori/runs/869998041?check_suite_focus=true#step:6:84
# github actions runner
Current runner version: '2.267.1'
Run GoogleCloudPlatform/github-actions/setup-gcloud@master
with:
project_id: ***
service_account_key: ***
export_default_credentials: true
version: latest
Results in gcloud version 301.0.0
Confirming that this issue also exists in Ubuntu 20.04 with gcloud version 301.0.0.
To fix:
sudo apt install google-cloud-sdk=297.0.1-0
This temporary workaround also works on Ubuntu Server 18.04.4.
What worked for us was to use the env var:
CLOUDSDK_PYTHON: /usr/bin/python
Ran into this error as well on Ubuntu 18.04.4 LTS
I get this error when, for example, I execute docker-compose build, but if I exectue this command with sudo it works.
I get this error when, for example, I execute docker-compose build
I also got the _OPENSSL_1_11 error when executing docker-compose build
with google-cloud-sdk
Debian package version 302.0.0-0
. Reverting to version 301.0.0-0
did not fix the problem, but reverting to 300.0.0-0
did.
System information:
# lsb_release --description
Description: Ubuntu 20.04 LTS
apt-get
.# docker-compose version
docker-compose version 1.26.2, build eefe0d31
docker-py version: 4.2.2
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019
@jtrh getting the same error as you while doing docker-compose build
. I tried prefixing it with CLOUDSDK_PYTHON=$(which python3)
didn't work either, same error.
System info:
$ gcloud -v
Google Cloud SDK 302.0.0
alpha 2020.07.17
beta 2020.07.17
bq 2.0.58
cloud-build-local
core 2020.07.17
gsutil 4.52
kubectl 1.15.11
pubsub-emulator 0.1.0
$ lsb_release --all
No LSB modules are available.
Distributor ID: LinuxMint
Description: Linux Mint 19.3 Tricia
Release: 19.3
Codename: tricia
$ docker-compose version
docker-compose version 1.24.1, build 4667896b
docker-py version: 3.7.3
CPython version: 3.6.8
OpenSSL version: OpenSSL 1.1.0j 20 Nov 2018
EDIT: this fixed it for me https://github.com/openssl/openssl/issues/5845#issuecomment-378601109
I was having this problem using the google/cloud-sdk
image on CircleCI.
My workaround was changing the docker-compose install method from the one recommended on the CircleCI docs to:
pip3 install docker-compose --upgrade
Hi @austinpray
Could you add a gcloud info
step with a build that errors out. I am unable to repro here with latest versions and 301.0.0
. If you have any tips on a reproduction, that would also be great.
@bharathkkb sure thing! here is the output
@bharathkkb I did notice that you were using docker pull
instead of docker-compose pull
Thanks @austinpray! I just saw the docker-compose
commands in your run.sh
:)
Thank you @sorliem. The solution you linked also worked for me (gcloud version 304).
Also remark that adding
LD_LIBRARY_PATH=/usr/local/lib;
in the command line didn't worked. Variable has to be exported.
I was able to reproduce this here. I was also able to fix it with the temporary workaround from https://github.com/openssl/openssl/issues/5845#issuecomment-378601109 suggest by @sorliem.
I will continue to look into a possible long term fix.
Thank you @sorliem. The solution you linked also worked for me (gcloud version 304).
Also remark that adding
LD_LIBRARY_PATH=/usr/local/lib;
in the command line didn't worked. Variable has to be exported. I have this very same problem after updating my Ubuntu 18.04 environment gcloud but when setting D_LIBRARY_PATH=/usr/local/lib I can run my docker-compose build commands OK. Thx!
I do not know if this is helpful if not excuse me: the same just happened to me after upgrading to 305.0.0. I am not sure what version I was running before but it was certainly 30x. Running Ubuntu 20.04 LTS. Exact error message below:
ERROR: gcloud failed to load: /tmp/_MEI3WIF2B/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /usr/lib/python3.8/lib-dynload/_ssl.cpython-38-x86_64-linux-gnu.so)
I got the same error on Ubuntu 18.04 and gcloud 304.0.0
export CLOUDSDK_PYTHON=/usr/bin/python
worked for me as well.
@jtrh getting the same error as you while doing
docker-compose build
. I tried prefixing it withCLOUDSDK_PYTHON=$(which python3)
didn't work either, same error.System info:
$ gcloud -v Google Cloud SDK 302.0.0 alpha 2020.07.17 beta 2020.07.17 bq 2.0.58 cloud-build-local core 2020.07.17 gsutil 4.52 kubectl 1.15.11 pubsub-emulator 0.1.0 $ lsb_release --all No LSB modules are available. Distributor ID: LinuxMint Description: Linux Mint 19.3 Tricia Release: 19.3 Codename: tricia $ docker-compose version docker-compose version 1.24.1, build 4667896b docker-py version: 3.7.3 CPython version: 3.6.8 OpenSSL version: OpenSSL 1.1.0j 20 Nov 2018
EDIT: this fixed it for me openssl/openssl#5845 (comment)
Thanks @sorliem Confirmed Workaround fix mine with:
gcloud -v Google Cloud SDK 307.0.0 bq 2.0.59 core 2020.08.21 gsutil 4.53
lsb_release --all Distributor ID: Ubuntu Description: Ubuntu 20.04.1 LTS Release: 20.04 Codename: focal
docker-compose version docker-compose version 1.26.2, build eefe0d31 docker-py version: 4.2.2 CPython version: 3.7.7 OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019
python --version Python 3.8.2
Why is LD_LIBRARY_PATH
set to /tmp/_MEI3WIF2B/
? Is this a bug with vendored python dependencies in docker-compose?
I can confirm that @bharathkkb workaround mentioned with the export LD_LIBRARY_PATH=/usr/local/lib
works for me !
By the way, to make things a bit harder to debug,I am using https://github.com/asdf-vm/asdf to install my stuff. But the workaround works
$ asdf list
argocd
1.5.4
docker-compose
1.26.0
gcloud
293.0.0
gohugo
extended_0.72.0
helm
3.2.0
kubectl
1.15.11
1.16.11
kubesec
0.9.2
kustomize
3.5.5
3.8.1
sops
v3.5.0
stern
1.11.0
terraform
0.12.16
0.12.18
So with
docker-compose
1.26.0
gcloud
293.0.0
export LD_LIBRARY_PATH=/usr/local/lib
worked for me too.
Debian 10 Buster
gcloud 308
export CLOUDSDK_PYTHON=/usr/bin/python
Worked for me too!
Installing Docker Compose with Pip fixed the issue for me. More details in https://github.com/GoogleCloudPlatform/docker-credential-gcr/issues/83#issuecomment-695072817.
Ran into this issue today using docker-compose to pull an image from GCR from my Ubuntu 20.10 WSL 2 distro.
I already had Python 3 installed (3.8.6 to be exact), but export CLOUDSDK_PYTHON=/usr/bin/python3
didn't work for me. The workaround export LD_LIBRARY_PATH=/usr/local/lib
worked for me.
I ran into this problem using docker-compose on Ubuntu 20.10 WSL 2 distro.
This fixed it for me:
export CLOUDSDK_PYTHON=/usr/bin/python2
(That's a 2 not a 3)
@michaelr, that only works if you install python2 or add a symlink from /usr/bin/python2
to /usr/bin/python3
(the latter not being recommended since it can lead to programs expecting Python 2 to crash because they're actually running under Python 3).
Ideally, this problem should be solved within the current state of modern Linux distros, which do not include Python 2.
Seeing this in Linux odin 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2 (2020-04-29) x86_64 GNU/Linux
Forced to revert back to python2.
I've started getting this after updating docker-compose on my machine. It happens if I run docker-compose pull
.
I was previously on a much older version of docker-compose, which worked. I can't remember which version, but it might have been installed as long ago as 2017.
EDIT: LD_LIBRARY_PATH=/usr/lib docker-compose pull my_service
works. I suppose I shall use that as a workaround for now. Why gcloud
expects to find libssl in /tmp/...
is... not something I understand.
Error:
ERROR: gcloud failed to load: /tmp/_MEIMOBjio/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /usr/lib/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so)
gcloud_main = _import_gcloud_main()
import googlecloudsdk.gcloud_main
from googlecloudsdk.api_lib.iamcredentials import util as iamcred_util
from googlecloudsdk.api_lib.util import apis_internal
from googlecloudsdk.core import properties
from googlecloudsdk.core import config
from oauth2client import client
from oauth2client import transport
import httplib2
from httplib2.python3.httplib2 import *
import ssl
import _ssl # if we can't import it, let the error propagate
This usually indicates corruption in your gcloud installation or problems with your Python interpreter.
Please verify that the following is the path to a working Python 2.7 or 3.5+ executable:
/usr/bin/python3
The above is truncated. Full error here.
Relevant versions:
/usr/bin/python3 --version
Python 3.6.9
I assume the Python executable being used is the one in /usr/bin/python3
, since that's what's in the error message. However, for completeness:
which python3
/home/thom/.pyenv/shims/python3
python3 --version
Python 3.6.9
gcloud version
Google Cloud SDK 319.0.0
alpha 2020.11.13
beta 2020.11.13
bq 2.0.62
core 2020.11.13
gsutil 4.55
kubectl 1.16.13
lsb_release -d
Description: Ubuntu 18.04.5 LTS
docker-compose version
docker-compose version 1.27.4, build 40524192
docker-py version: 4.3.1
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019
openssl version
OpenSSL 1.1.1 11 Sep 2018
which openssl
/usr/bin/openssl
locate libssl.so.1.1
/snap/core18/1885/usr/lib/x86_64-linux-gnu/libssl.so.1.1
/snap/core18/1932/usr/lib/x86_64-linux-gnu/libssl.so.1.1
/snap/gnome-3-28-1804/128/usr/lib/x86_64-linux-gnu/libssl.so.1.1
/snap/gnome-3-28-1804/145/usr/lib/x86_64-linux-gnu/libssl.so.1.1
/snap/gnome-3-34-1804/36/usr/lib/x86_64-linux-gnu/libssl.so.1.1
/snap/gnome-3-34-1804/60/usr/lib/x86_64-linux-gnu/libssl.so.1.1
/usr/lib/x86_64-linux-gnu/libssl.so.1.1
The magical incantation
export LD_LIBRARY_PATH=/usr/local/lib
as described above, worked for me.
Same problem here. Is there any known fix for the gihub action as described by the OP ?
export LD_LIBRARY_PATH=/usr/local/lib
worked for me too.
I had the exact same problem. The above mentioned export worked for me.
Ubuntu 20.04 Google Cloud SDK 322.0.0
Thanks
For docker-compose
on GitHub Actions, I'm also working around this by using:
- uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
with:
version: '297.0.1' # https://github.com/google-github-actions/setup-gcloud/issues/128
...
It would be great to not be stuck on 297.0.1 forever, the latest version at the time of writing is 324.0.0, any chances of an upstream fix?
The magical incantation
export LD_LIBRARY_PATH=/usr/local/lib
as described above, worked for me.
With newest version of gcloud (326.0.0) this no longer works :(
Same problem here.
GoogleCloudPlatform/github-actions/setup-gcloud
may have been deprecated but it actually works reliably.
Hi @vicpara
There should not be any difference between GoogleCloudPlatform/github-actions/setup-gcloud
and google-github-actions/setup-gcloud
w.r.t this issue as it is related to gcloud + docker-compose.
@jacek-jablonski I ran a quick test with 327 which seemed to work with the workaround https://github.com/bharathkkb/test-setup-gcloud-128/runs/1876995207 Could you share how it failed in your case?
@bharathkkb The issue I have with this new version is that it breaks gcloud auth configure-docker
.
Here is my use case below.
build-deploy:
needs: bump-version
name: "Build and Deploy"
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Setup GCP credentials
uses: google-github-actions/setup-gcloud@master
with:
project_id: ${{ secrets.GCLOUD_PROJECT_ID }}
service_account_key: ${{ secrets.GCP_SA_KEY_JSON }}
export_default_credentials: true
- name: Docker pull
run: |
gcloud auth configure-docker
docker-compose pull
The error I get is:
Run gcloud auth configure-docker
gcloud auth configure-docker
docker-compose pull
shell: /bin/bash -e ***0***
env:
GKE_PROJECT_ID: ***
GKE_EMAIL: ***
GITHUB_SHA: 819aa0a83ff3641e84c2ce6676260186c4344330
GITHUB_BRANCH: refs/heads/master
GKE_ZONE: ***
GKE_CLUSTER: ***
REGISTRY_HOSTNAME: eu.gcr.io
DEPLOYMENT_ENV: rc
DEPLOYMENT_COMMIT_SHA: 819aa0a83ff3641e84c2ce6676260186c4344330
SEM_VERSION: ***
VERSION_TIMESTAMP: 1614099296
BUILDHASH: 819aa0a83ff3641e8412de6676260186c4344330
BUILDTAG: 0.16.75.0
CLOUDSDK_METRICS_ENVIRONMENT: github-actions-setup-gcloud
GCLOUD_PROJECT: ***
GOOGLE_APPLICATION_CREDENTIALS: /home/runner/work/core/core/c7331fe3-c987-4291-9289-4b0ac681b435
Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
After update, the following will be written to your Docker config file
located at [/home/runner/.docker/config.json]:
***
"credHelpers": ***
"gcr.io": "gcloud",
"us.gcr.io": "gcloud",
"eu.gcr.io": "gcloud",
"asia.gcr.io": "gcloud",
"staging-k8s.gcr.io": "gcloud",
"marketplace.gcr.io": "gcloud"
***
***
Do you want to continue (Y/n)?
Docker configuration file updated.
Pulling xxxxxxxxxx ...
Pulling xxxxxxxxxx ...
Pulling xxxxxxxxxx ...
Pulling xxxxxxxxxx ...
Pulling xxxxxxxxxx-service ...
Pulling xxxxxxxxxx-service ...
ERROR: gcloud failed to load: /tmp/_MEINIeVty/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /usr/lib/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so)
gcloud_main = _import_gcloud_main()
import googlecloudsdk.gcloud_main
from googlecloudsdk.calliope import base
from googlecloudsdk.calliope import arg_parsers
from googlecloudsdk.core import log
ERROR: for artifacts-service Credentials store error: StoreError('Credentials store docker-credential-gcloud exited with "".')
Traceback (most recent call last):
File "docker/credentials/store.py", line 80, in _execute
File "subprocess.py", line 411, in check_output
File "subprocess.py", line 512, in run
subprocess.CalledProcessError: Command '['/opt/hostedtoolcache/gcloud/329.0.0/x64/bin/docker-credential-gcloud', 'get']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker/auth.py", line 264, in _resolve_authconfig_credstore
File "docker/credentials/store.py", line 35, in get
File "docker/credentials/store.py", line 93, in _execute
docker.credentials.errors.StoreError: Credentials store docker-credential-gcloud exited with "".
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker-compose", line 3, in <module>
File "compose/cli/main.py", line 80, in main
File "compose/cli/main.py", line 192, in perform_command
File "compose/metrics/decorator.py", line 18, in wrapper
File "compose/cli/main.py", line 828, in pull
File "compose/project.py", line 763, in pull
File "compose/project.py", line 813, in parallel_pull
File "compose/parallel.py", line 106, in parallel_execute
File "compose/parallel.py", line 204, in producer
File "compose/project.py", line 795, in pull_service
File "compose/service.py", line 1236, in _do_pull
File "docker/api/image.py", line 411, in pull
File "docker/auth.py", line 48, in get_config_header
File "docker/auth.py", line 324, in resolve_authconfig
File "docker/auth.py", line 235, in resolve_authconfig
File "docker/auth.py", line 281, in _resolve_authconfig_credstore
docker.errors.DockerException: Credentials store error: StoreError('Credentials store docker-credential-gcloud exited with "".')
Error: Process completed with exit code 255.
The magical incantation
export LD_LIBRARY_PATH=/usr/local/lib
as described above, worked for me.
With newest version of gcloud (326.0.0) this no longer works :(
Linux Mint 20.10 This is the cure in combination with
export CLOUDSDK_PYTHON=/usr/bin/python3
297.0.1 for arm64 doesn't seem to be available via apt-get anymore...
340.0.0-0 works with
export LD_LIBRARY_PATH=/usr/local/lib
None of the solutions mentioned above helped me.
Before finally finding my own workaround, I tried:
v297
v340
(it was 330
initially)export CLOUDSDK_PYTHON=/usr/bin/python3
and export LD_LIBRARY_PATH=/usr/local/lib
(both were already in my .zshrc
)pip3 install docker-compose --upgrade
3.7.10
which is the same as CPython
used by my docker-compose
(inspired by this answer)For anyone using WSL, and probably future me:
What eventually helped me was realizing that because my Ubuntu runs docker-compose
through WSL and the entire Docker runs on Windows, I might have some luck setting up the same container in Windows, and then just getting it back up from my usual WSL console and Ubuntu filesystem.
I cloned my repo to Windows filesystem, installed gcloud
for Windows, set up gcloud authentication, ran docker-compose up
from cmd, and sure enough Docker pulled the images and the container built just fine. Now that my Docker for Windows has all the dependency images, I don't have any trouble running the container from inside WSL.
Debian 10 here, with gcloud v346
and python 3.7.3 (from debian packages).
the export
workaround didn't worked: gcloud still tries to find libraries in /tmp/something
https://github.com/google-github-actions/setup-gcloud/issues/128#issuecomment-673864153
CLOUDSDK_PYTHON=/usr/bin/python
worked for me while export CLOUDSDK_PYTHON=$(which python3)
didn't work.
I ran into this issue when running docker-compose in a GCP VM instance running Debian 10. Downgrading to Debian 9 resolved the issue.
FYI Today I removed the v297.0.1 workaround mentioned here, and things are once again working on the latest version of ubuntu & glcoud; I also moved to the new google-github-actions/setup-gcloud@master
repository instead of uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
Just happened with github actions ubuntu:latest and command gcloud auth configure-docker -q
Everyone here put their lucky number with this sdk. i will start from the most like answer
TL;DR
Using
GoogleCloudPlatform/github-actions/setup-gcloud@master
withubuntu-latest
. Builds suddenly started failing overnight due todocker-credential-gcloud
throwing aversion OPENSSL_1_1_1' not found
error.Expected behavior
docker-credentials-gcloud should allow me to pull an image from my gcp repo
Observed behavior
docker-credentials-gcloud fails complaining about
version OPENSSL_1_1_1' not found
and build fails.https://github.com/austinpray/kaori/pull/134/checks?check_run_id=814820107#step:6:90
:point_right: Reverting to version
297.0.1
as in https://github.com/austinpray/kaori/pull/133 makes the build pass again.Reproduction
Here is me fixing it by pinning the gcloud version to an older one: https://github.com/austinpray/kaori/pull/133
Here is me reproducing the issue by unpinning the version: https://github.com/austinpray/kaori/pull/134 (https://github.com/austinpray/kaori/pull/134/checks?check_run_id=814820107#step:6:90)
Action YAML
Repository
https://github.com/austinpray/kaori/pull/133
If you want temporary access to this repo for debugging and such: I can give it to you. This is just a silly side project.