Dockerhub builder can't access private repositories with forwarded ssh-agent (`--ssh default`)

[mjlbach]

I've set up Dockerhub builder to build a repository with the following dockerfile:

FROM alpine

RUN apk add --no-cache git openssh

# Needed for SSH authentication
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts

RUN --mount=type=ssh git clone git@github.com:mjlbach/test.git

In the root of this repo I added hooks/build (Note, I believe buildkit runs by default but I was directed to do these exact steps by support).

#!/bin/bash
eval $(ssh-agent)
ssh-add ~/.ssh/id_rsa
docker buildx install

docker buildx build \
    --ssh default \
    -f $DOCKERFILE_PATH \
    -t $IMAGE_NAME .

This yields the following error

2023-03-28T22:45:03Z > [3/3] RUN --mount=type=ssh git clone git@github.com:mjlbach/test.git:
2023-03-28T22:45:03Z #7 0.545 Cloning into 'neuman'...
2023-03-28T22:45:03Z #7 0.571 Host key verification failed.
2023-03-28T22:45:03Z #7 0.573 fatal: Could not read from remote repository.
2023-03-28T22:45:03Z #7 0.573
2023-03-28T22:45:03Z #7 0.573 Please make sure you have the correct access rights
2023-03-28T22:45:03Z #7 0.573 and the repository exists.
2023-03-28T22:45:03Z ------
2023-03-28T22:45:03Z ERROR: failed to solve: executor failed running [/bin/sh -c git clone git@github.com:mjlbach/test.git]: exit code: 128
2023-03-28T22:45:03Z build hook failed! (1)

Building the file locally with the same command works fine.