Images sometimes don't include ssh-agent

nunofgs commented 5 years ago

Something odd is going on. I'm using lambci/lambda:build-nodejs8.10 as a base image and got an error a few days ago:

sh: ssh-agent: command not found

(My image depends on ssh-agent being included in your base image).

A day later, everything was fine. Pulling lambci/lambda:build-nodejs8.10 has the correct ssh binaries. Now, today, I'm getting the same thing.

It seems this image is somehow compiling without ssh binaries but only sometimes. Maybe a problem in your build trigger?

debugwand commented 5 years ago

We also seeing this in CircleCI in the checkout step where it tries to get our code from github, which has been happening consistently since this morning.

Either git or ssh (required by git to clone through SSH) is not installed in the image. Falling back to CircleCI's native git client but the behavior may be different from official git. If this is an issue, please use an image that has official git and ssh installed.

SSHing into the build shows that git command works but ssh returns command not found

scottrmercer commented 5 years ago

Same here, also on circleCI, just started this morning as well. We are using Python 3.6, which I did notice was updated yesterday.

mhart commented 5 years ago

Hmmm, well the build image doesn't explicitly include ssh – but instead it installs the development group: https://github.com/lambci/docker-lambda/blob/master/base/build/Dockerfile#L20

Lambda has made some changes to the underlying OS recently (which is why I had to rebuild all the images) – I wonder if that's made a difference? Maybe ssh is no longer included in the development group or something?

I'll look into it

killthrush commented 5 years ago

Can confirm problem with python 3.6. This command uses github dependencies under the hood and worked fine yesterday: docker run --name lambdapy36 -it -v $(pwd):/src lambci/lambda:build-python3.6 /bin/sh -c "pip install -r /src/requirements.txt -t /src/vendored/". Error message is fatal: unable to fork and there is no ssh exe on the image.

mhart commented 5 years ago

Here's what's included in the group – I don't see any ssh, so either it's been removed, or it's a dependency of another package that may have changed:

$ bash-4.2# yum groupinfo development

Group: Development tools
 Group-Id: development
 Description: A basic development environment.
 Mandatory Packages:
   =autoconf
   =automake
    binutils
   =bison
   =flex
   =gcc
   =gcc-c++
    gdb
   =gettext
   =kexec-tools
   =latrace
   =libtool
    make
    patch
    pkgconfig
    rpm-build
   =strace
   =system-rpm-config
   =systemtap-runtime
 Default Packages:
   =byacc
   =crash
   =cscope
   =ctags
   =cvs
   =diffstat
   =doxygen
    elfutils
   =gcc-gfortran
   =git
   =indent
   =intltool
   =ltrace
   =patchutils
   =rcs
   =subversion
   =swig
   =systemtap
   =texinfo
   =valgrind
 Optional Packages:
   ant
   bzr-python27
   chrpath
   cmake
   dejagnu
   expect
   gcc-gnat
   glibc-utils
   imake
   jpackage-utils
   lslk
   mercurial-python27
   mod_dav_svn
   nasm
   perltidy
   python27-babel
   python27-docs
   rpmdevtools
   rpmlint
   systemtap-initscript

scottrmercer commented 5 years ago

For circleCI, we've worked around the issue by adding the following to our build script:

  - run: yum reinstall -y fipscheck-lib openssh-clients

(thanks to @nicholasyager for figuring this out)

killthrush commented 5 years ago

@scottrmercer - that worked for me! cheers!

mhart commented 5 years ago

I'll add this to the base image build too – no idea what happened, unfortunately I haven't had much communication from AWS over the OS changes. Sorry y'all.

killthrush commented 5 years ago

No worries @mhart - thanks for responding so quickly!

mhart commented 5 years ago

So AWS have updated glibc on Lambda, which has affected a whole bunch of things. Especially because it's using a different yum repo from what it was using before – and I think this has also affected openssh.

Unfortunately when I'm trying to fix the base build I keep getting:

Installed package openssh-clients-6.6.1p1-33.66.amzn1.x86_64 not available.

Something's up with the yum repos and my yum foo isn't sharp enough to figure it out yet.

mhart commented 5 years ago

Ok, as of https://github.com/lambci/docker-lambda/commit/4eaedc31dd2a6c3a14e68e8b989f19a86a121347 this should be fixed. Let me know if y'all run into any more issues!

killthrush commented 5 years ago

Confirmed fixed - thanks @mhart!

debugwand commented 5 years ago

thanks very much for such quick work 🥇

lambci / docker-lambda

Images sometimes don't include ssh-agent #162