Doesn't work with submodules

[Pi-George]

Using regular CircleCI's -checkout call I can later fetch submodules without issue. However I'm unable to do that with this orbs shallow checkout.

Not sure what to give you in terms of code as my config file is rather large and I have no minimal example.

[guitarrapc]

I've tried submodule on ci, but success without failure.

https://app.circleci.com/pipelines/github/guitarrapc/git-shallow-clone-orb/211/workflows/5e0d2c70-0b9f-4335-aa96-a610f9ecae5d/jobs/627

Can you show me reproduce sample?

[Pi-George]

Sorry but it's been too long, I can no longer find the ticket where I was attempting this and any CircleCI runs from that long ago no longer show up. I could try to re-implement this again for the sake of this issue but it's no longer worth the time. Sorry for giving such a poor issue description initially.

[guitarrapc]

Sorry for the late response! If any issue found, please feel free to make an issue.

[Pi-George]

@guitarrapc so I found the branch, I made a new PR in my repo to kick off CircleCI see how it runs. Here are the submodule errors:

Updating submodules.
Submodule 'library/iputils' (git@bitbucket.org:****/iputils.git) registered for path 'library/iputils'
Cloning into '/srv/www/library/iputils'...
Load key "/root/.ssh/id_rsa": invalid format
git@bitbucket.org: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@bitbucket.org:****/iputils.git' into submodule path '/srv/www/library/iputils' failed
Failed to clone 'library/iputils'. Retry scheduled
Cloning into '/srv/www/library/iputils'...
Load key "/root/.ssh/id_rsa": invalid format
git@bitbucket.org: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@bitbucket.org:****/iputils.git' into submodule path '/srv/www/library/iputils' failed
Failed to clone 'library/iputils' a second time, aborting
Something went wrong during deployment, or you used ctrl-c, stopping process
Removing var/cache/config.
Removing var/cache/awscredsprovider.

Exited with code exit status 1

So here's the thing, CircleCI is building an image and running it as a container, inside that container there's then a set-up script which syncs and updates submodules. To ensure that the container has permission to update submodules this CircleCI step is used:

copy-circleci-configs:
    steps:
      - run:
          name: Provides needed .gitconfig and .ssh/ files to allow submodule access within docker build
          command: |
            mkdir -p tmp/
            cp -r /home/circleci/.ssh tmp/
            cp /home/circleci/.gitconfig tmp/

When the container is run those two files are mounted:

docker run -d\
             --mount type=bind,source=$(pwd),target=/srv/www/\
             --mount type=bind,source=/home/circleci/.gitconfig,target=/root/.gitconfig\
             --mount type=bind,source=/home/circleci/.ssh/,target=/root/.ssh/\
             --name="pi-app"\
             --env TERM=$TERM pi-app:testing

However it seems that using this orb has somehow broken this functionality since we're getting permission errors. I believe the problem must be something to do with CircleCI's default checkout creating the .ssh and .gitconfig files while this orb's checkout command doesn't?

Just using the simple version of this orbs checkout by the way:

- git-shallow-clone/checkout:
          depth: 10

[guitarrapc]

@Pi-George Thnak you for detail cralification, Interesting point is id_rsa detected as invalid format with this orb, but circleci checkout doesn't.

Load key "/root/.ssh/id_rsa": invalid format git@bitbucket.org: Permission denied (publickey).

Probably I need lab to reproduce issue and see detail. Is the step is following?

1. checkout (via orb or checkout) <- this is key point and circleci checkout may do something.
2. do copy-circleci-configs.
3. docker run and submodule checkout.
   • change permission for mounted .ssh and others
   • git submodule

Appreciate if minumum example is provided.

[Pi-George]

The permission for mounted .ssh isn't ever changed apart from in the initial checkout as far as I can see, other than that your 1, 2, 3 steps look correct.

The CircleCI Checkout has this code related to id_rsa:

rm -f "$SSH_CONFIG_DIR/id_rsa"
printf "%s" "$CHECKOUT_KEY" > "$SSH_CONFIG_DIR/id_rsa"
chmod 0600 "$SSH_CONFIG_DIR/id_rsa"
if (: "${CHECKOUT_KEY_PUBLIC?}") 2>/dev/null; then
  rm -f "$SSH_CONFIG_DIR/id_rsa.pub"
  printf "%s" "$CHECKOUT_KEY_PUBLIC" > "$SSH_CONFIG_DIR/id_rsa.pub"
fi

export GIT_SSH_COMMAND='ssh -i "$SSH_CONFIG_DIR/id_rsa" -o UserKnownHostsFile="$SSH_CONFIG_DIR/known_hosts"'

While this orb just has

(umask 077; touch ~/.ssh/id_rsa)
chmod 0600 ~/.ssh/id_rsa
(echo $CHECKOUT_KEY > ~/.ssh/id_rsa)

So it seems like printf "%s" "$CHECKOUT_KEY" > "$SSH_CONFIG_DIR/id_rsa" works but (echo $CHECKOUT_KEY > ~/.ssh/id_rsa) doesn't for some reason?

So I've just gone and ssh'd into the CircleCI process and it seems that id_rsa is completely empty when using this orb. Looking at the variables used for creating id_rsa I see that $SSH_CONFIG_DIR, $CHECKOUT_KEY are also blank for me while I'm ssh'd. I assume this is CircleCI messing with my ssh so that I can't see sensitive variables, but it does make it very hard to debug what's going on.

[Pi-George]

circleci@ip-172-28-43-16:~$ printf "%s" "foobar" > /tmp/testfile1
circleci@ip-172-28-43-16:~$ (echo "foobar" > /tmp/testfile2)
circleci@ip-172-28-43-16:~$ cmp /tmp/testfile1 /tmp/testfile2
cmp: EOF on /tmp/testfile1

Looks like the cause is to do with the echo causing an EOF missing issue while printf has no such problems. id_rsa could definitely get an invalid format error from a missing EOF. Not sure why echo doesn't create an EOF but this seems like it should work as a fix?

[Pi-George]

Want me to make a PR so this can get released?

[guitarrapc]

Sorry for late response, I couldn't researve time to take time for it. I'm very happy to see PR, thanks.

[guitarrapc]

I digged into issue. Perhaps circleci standard checkout do special with CHECKOUT_KEY and CHECKOUT_KEY_PUBLIC, and detail is hide from chekcout command. These Environment variables are missing on third-party command, git-shallow-clone/checkout is also missing env.

Currently I'm digging into what haappens. Any advise with $CHECKOUT_KEY is appreciated, somthining like path of these key existing.

[guitarrapc]

May be required to pass CHECKOUT_KEY / CHECKOUT_KEY_PUBLIC via Project's Environment variables?

[guitarrapc]

Custom Checkout command - Tips, Tricks and Hacks - CircleCI Discuss https://discuss.circleci.com/t/custom-checkout-command/31624

👎

[Pi-George]

@guitarrapc would you mind publishing a new version with the fix? Thanks

[Pi-George]

Oh strange, I didn't see that it was re-opened when I commented the above, so those environment variables are completely missing. How annoying

[Pi-George]

If they're completely missing how are submodules working outside of my niche docker conainer scenario?

I've tried submodule on ci, but success without failure.

https://app.circleci.com/pipelines/github/guitarrapc/git-shallow-clone-orb/211/workflows/5e0d2c70-0b9f-4335-aa96-a610f9ecae5d/jobs/627

Can you show me reproduce sample?

[Pi-George]

May be required to pass CHECKOUT_KEY / CHECKOUT_KEY_PUBLIC via Project's Environment variables?

I'll try this out

[Pi-George]

It seems like the lack of breaklines in project environment variables makes this a pain. I could probably throw something together where it converts spaces to breaklines and prepends/appends the whole -----END/START OPENSSH PRIVATE KEY----- stuff

[Pi-George]

Okay yeah I have a working solution from that:

      - git-shallow-clone/checkout:
          depth: 10
      - run:
          name: Sets the id_rsa for submodules later on, only needed with shallow clone
          command: |
            echo "" > ~/.ssh/id_rsa
            echo "-----BEGIN RSA PRIVATE KEY-----" >> ~/.ssh/id_rsa
            echo "$ID_RSA" | tr " " "\n" >> ~/.ssh/id_rsa
            echo "-----END RSA PRIVATE KEY-----" >> ~/.ssh/id_rsa

Where $ID_RSA is the private key contents

[guitarrapc]

those environment variables are completely missing. How annoying

Yes, you are correct. I think this is intended behaivour of CircleCI, but too match pain.

It seems like the lack of breaklines in project environment variables makes this a pain.

To avoid breakline of environment variables, I think encoding base64 is common way.

$ cat "./id_rsa" | base64 # register base64 to CHECKOUT_KEY

Then decode on checkout step.

echo $CHECKOUT_KEY | base64 --decode > ~/.ssh/id_rsa

[guitarrapc]

I thinks there are 2 options for users. B is what circleci offers, and I would like to prepare A for users. Is A fit to your usage @Pi-George ?

• A. Users who want restore $CHECKOUT_KEY to .ssh/id_rsa, you should set base64 id_rsa on environment variables, then git-shallow-clone-orb/checkout will decode it to .ssh/id_rsa". https://github.com/guitarrapc/git-shallow-clone-orb/issues/15#issuecomment-959108665 (I need prepare this)
• B. User who want restore additional restore key, use add_ssh_keys. refer: https://circleci.com/docs/2.0/add-ssh-key/

If people who want restore there key by theirself, then git-shallow-clone-orb will do nothing. Just set id_rsa by your self step if fine. https://github.com/guitarrapc/git-shallow-clone-orb/issues/15#issuecomment-958961319

NOTE: Users who don't restore $CHECKOUT_KEY to .ssh/id_rsa, never mind ignore what happens. Missing ./ssh/id_rsa will never affect your CI.

[Pi-George]

Yeah A works for me, this seems to work fine now

[Pi-George]

Thanks for your help

[Pi-George]

Hey I've since found a better solution than using janky environment variables to create an id_rsa key. If you shallow clone, then checkout the checkout will sort out the missing id_rsa keys without ruining the shallow clone.

      - git-shallow-clone/checkout:
          depth: 10
      - checkout

[guitarrapc]

Thanks, sounds excellent 👍