SSH agent mounting is failing on Mac OS Sonoma

[frodri]

Python Version: 3.10.13 Package Version: 0.4.1 OS: Mac OS Sonoma 14.2.1

Reproducible: Always

Steps to reproduce:

• Create a pipeline step that needs to fetch data using SSH (e.g.: a private Git repo).
• Add keys using ssh-agent --apple-add-keychain.
• Use pipeline-runner to run that step.

Expected result: SSH agent is mounted in the container, ready for use.

Current result: An exception is thrown during the mount. Stack trace:

2024-04-22 13:31:16.400 pipeline_runner.runner: Error during pipeline execution
Traceback (most recent call last):
  File "/Users/francisco/Library/Caches/pypoetry/virtualenvs/project-12345678-py3.10/lib/python3.10/site-packages/docker/api/client.py", line 265, in _raise_for_status
    response.raise_for_status()
  File "/Users/francisco/Library/Caches/pypoetry/virtualenvs/project-12345678-py3.10/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.44/containers/db8deb7ffc0e9c102d7e85e449afd2a365329b9dcd89cd4d9de033949b84da01/start

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/francisco/Library/Caches/pypoetry/virtualenvs/project-12345678-py3.10/lib/python3.10/site-packages/pipeline_runner/runner.py", line 192, in run
    container_runner.start()
  File "/Users/francisco/Library/Caches/pypoetry/virtualenvs/project-12345678-py3.10/lib/python3.10/site-packages/pipeline_runner/container.py", line 58, in start
    self.start_container()
  File "/Users/francisco/Library/Caches/pypoetry/virtualenvs/project-12345678-py3.10/lib/python3.10/site-packages/pipeline_runner/container.py", line 155, in start_container
    container = self._client.containers.run(
  File "/Users/francisco/Library/Caches/pypoetry/virtualenvs/project-12345678-py3.10/lib/python3.10/site-packages/docker/models/containers.py", line 880, in run
    container.start()
  File "/Users/francisco/Library/Caches/pypoetry/virtualenvs/project-12345678-py3.10/lib/python3.10/site-packages/docker/models/containers.py", line 417, in start
    return self.client.api.start(self.id, **kwargs)
  File "/Users/francisco/Library/Caches/pypoetry/virtualenvs/project-12345678-py3.10/lib/python3.10/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/francisco/Library/Caches/pypoetry/virtualenvs/project-12345678-py3.10/lib/python3.10/site-packages/docker/api/container.py", line 1135, in start
    self._raise_for_status(res)
  File "/Users/francisco/Library/Caches/pypoetry/virtualenvs/project-12345678-py3.10/lib/python3.10/site-packages/docker/api/client.py", line 267, in _raise_for_status
    raise create_api_error_from_http_exception(e) from e
  File "/Users/francisco/Library/Caches/pypoetry/virtualenvs/project-12345678-py3.10/lib/python3.10/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation) from e
docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.44/containers/db8deb7ffc0e9c102d7e85e449afd2a365329b9dcd89cd4d9de033949b84da01/start: Internal Server Error ("error while creating mount source path '/host_mnt/private/tmp/com.apple.launchd.12345678/Listeners': mkdir /host_mnt/private/tmp/com.apple.launchd.12345678/Listeners: operation not supported")

[mathieu-lemay]

Hi @frodri, thanks for reporting this issue. I'll look into it as soon as I can, in the next few days.

I don't have a mac computer with Sonoma, only one with Monterrey (I think). Anyway, I should be able to reproduce. I'll keep you posted if I can't.

[mathieu-lemay]

I've tested on Monterey and I can't reproduce the issue. It looks like something changed in a later version. I'll figure out a way to run a Ventura or Sonoma VM so see if I can repro and fix.

[frodri]

@mathieu-lemay Understood. Just in case this is needed during the VM setup: the Docker Desktop version I'm using is 4.28.0.

[mathieu-lemay]

I've finally managed to reproduce the issue! After some investigation, it seems that it's actually a problem with docker for mac: https://github.com/docker/for-mac/issues/6375

I've tried a few of the solutions in that thread and none of them worked for me though. I'll try to see if I can find a solution for that. If you think of anything, I'm interested!

You should be able to reproduce the issue directly with docker if you are curious to try something:

docker run --rm -it -v $SSH_AUTH_SOCK:/ssh-sock:ro \
    -e SSH_AUTH_SOCK=/ssh-sock --entrypoint ssh-add linuxserver/openssh-server -l

[mathieu-lemay]

I found a solution but I'll need some more time to add proper tests and do some validation on different setups.

@frodri If you want to test it on your own, you can check this branch: https://github.com/mathieu-lemay/pipeline-runner/tree/fix-ssh-agent-forwarding-with-docker-desktop

[frodri]

@mathieu-lemay I've tested the branch over the last few hours. I can confirm that the test branch is able to get past the Docker Internal Server Error, but I've not been able to get the container to display the forwarded keys on ssh-add -L:

+ echo 'Host *' >> ~/.ssh/config

>>> Execution time: 0.000s

+ echo '   StrictHostKeyChecking no' >> ~/.ssh/config

>>> Execution time: 0.000s

+ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-woIwXnrmvX5c/agent.60; export SSH_AUTH_SOCK;
SSH_AGENT_PID=61; export SSH_AGENT_PID;
echo Agent pid 61;

>>> Execution time: 0.001s

+ ssh-add -L
error fetching identities: communication with agent failed

>>> Execution time: 0.002s

2024-04-30 13:41:39.381 pipeline_runner.runner: Step 'Initialize container': FAIL

[mathieu-lemay]

@frodri Just to make sure, did you add the --ssh parameter when you ran that pipeline? I get that same error message if I forget that param, otherwise it works just fine. If you did add the --ssh param, can you provide the whole log please?

You should see this message in the logs. If you don't, something is indeed wrong:

2024-04-30 14:33:17.706 pipeline_runner.container: Using docker desktop's host service ssh agent

To test, I'm running the following pipeline.

pipelines:
  default:
      - step:
          name: Clone
          image: alpine/git
          script:
            - echo 'Host *' >> ~/.ssh/config
            - echo '   StrictHostKeyChecking no' >> ~/.ssh/config
            - ssh-agent
            - ssh-add -L
            - git clone git@github.com:mathieu-lemay/pipeline-runner.git

Two small notes

1. You don't need to run ssh-agent in the pipeline, but I guess you only added it for testing
2. I'm cloning my own repo with ssh to ensure everything works. Replace with a repo of your own of course.

[frodri]

@mathieu-lemay I did. I'm running the following command thru a nodejs build script:

poetry run pipeline-runner run custom.build-to-multidev --env-file "./.env.local" --ssh

Strangely enough, I tested the git clone instead of the key testing commands this morning, and I was able to get the fetch to work - so I suppose the test commands might caused the ssh login problems in this case.

I'll keep testing the new branch over the next few days to see if I can find any more issues.

[mathieu-lemay]

I've merged the fix and closed the issue. I have done a few tests in different scenarios and I'm satisfied with the fix. I'll publish a new version as soon as the tests have passed.

Feel free to re-open if you encounter any issues.

[frodri]

Will do. Thank you for the assistance.

[mathieu-lemay]

It's my pleasure! Thank you for the feedback. I'm very happy to see my software being used by others!

[frodri]

👍

(Oh, and before I forget: I can confirm that the SSH agent worked as expected during my testing earlier in the week. So I think we're good there.)