Issues deploying through gitlabci - (node:25) UnhandledPromiseRejectionWarning: Error: (HTTP code 500) server error denied: access forbidden

[ghost]

I am trying to integrate the caprover deploy --imageName command into my gitlab CI/CD workflow.

However, I am getting the following message:

Error: (HTTP code 500) server error - Get https://registry.gitlab.com/v2/group-name/project-name/my-build-image/manifests/5d22fa126f7c504e3aac53564f6eb8ca5dd0f696: denied: access forbidden

Firstly I thought there might be something wrong with my gitlab token so I made sure the token works fine. I logged into the box running caprover and ran docker login registry.gitlab.com and successfully logged in with the token as password. I was then able to pull the docker images from gitlab registry. For testing purposes I gave full access rights - api, read_user, read_api, read_repository, write_repository, read_registry, write_registry. So I can't imagine the token being the issue.

I then tailed the caprover container to see why its failing and got the following error message:

> POST /api/v2/login 200 219.602 ms - 307
GET /api/v2/user/apps/appDefinitions 200 3.679 ms - 5517
POST /api/v2/user/apps/appData/projectName-dev?detached=1 200 1.431 ms - 58
GET /api/v2/user/apps/appData/projectName-dev 200 1.529 ms - 750
(node:25) UnhandledPromiseRejectionWarning: Error: (HTTP code 500) server error - Get https://registry.gitlab.com/v2/group-name/project-name/my-build-image/manifests/5d22fa126f7c504e3aac53564f6eb8ca5dd0f696: denied: access forbidden
    at /usr/src/app/node_modules/docker-modem/lib/modem.js:301:17
    at IncomingMessage.<anonymous> (/usr/src/app/node_modules/docker-modem/lib/modem.js:328:9)
    at IncomingMessage.emit (events.js:327:22)
    at endReadableNT (_stream_readable.js:1327:12)
    at processTicksAndRejections (internal/process/task_queues.js:80:21)
(node:25) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 66)
GET /api/v2/user/apps/appData/projectName-dev 200 2.615 ms - 990`

I've also made sure that I've added the remote registry under cluster. Followed the instructions on https://caprover.com/docs/ci-cd-integration.html

It seems like its failing to pull docker images even though I'm able to pull it from the box running caprover. Am I missing something?

I am using caprover version caprover/caprover:1.8.2

Please assist.

Thanks, Junayed

[githubsaturn]

Can you please try this?

1- Add this as your docker registry:

URL: registry.gitlab.com
USERNAME: githubsaturn
IMAGE PRIFIX: githubsaturn
PASSWORD: K6hzgN8oNNAojPbeEoQ_

2- Create a new app

3- Go to deployment tab - Method 6: Deploy via ImageName and deploy this image:

registry.gitlab.com/githubsaturn/test-docker-build:my-image-name

Does this work?

[ghost]

That has worked. I can see from the logs

------------------------- Tue Feb 09 2021 00:29:04 GMT+0000 (Coordinated Universal Time) Build started for test-app An explicit image name was provided (registry.gitlab.com/githubsaturn/test-docker-build:my-image-name). Therefore, no build process is needed. Pulling this image: registry.gitlab.com/githubsaturn/test-docker-build:my-image-name This process might take a few minutes. Build has finished successfully!

I've just tried the same with my docker image and gitlab registry . I'm seeing the same error as above:

------------------------- Tue Feb 09 2021 00:47:26 GMT+0000 (Coordinated Universal Time)
Build started for junayed
An explicit image name was provided (registry.gitlab.com/group-name/app-name/my-build-image:5d22fa126f7c504e3aac53564f6eb8ca5dd0f696). Therefore, no build process is needed.
Pulling this image: registry.gitlab.com/group-name/app-name/my-build-image:5d22fa126f7c504e3aac53564f6eb8ca5dd0f696 This process might take a few minutes.
Build has failed!
----------------------
Deploy failed!
Error: (HTTP code 500) server error - Get https://registry.gitlab.com/v2/group-name/app-name/my-build-image/manifests/5d22fa126f7c504e3aac53564f6eb8ca5dd0f696: denied: access forbidden

The only different I see between is that when I'm adding the docker registry, I have dots in the username and prefix. e.g.

URL: registry.gitlab.com USERNAME: m.j.mizan IMAGE PRIFIX: m.j.mizan PASSWORD: someToken

Could it be the username and image prefix causing issues?

[githubsaturn]

I'd doubt that it is the case, but you can try it out and create a test account without any dots.

[ghost]

I've created a new gitlab account to try this out and it's worked. I'm not 100% sure what the issue is. The only difference between my old account and the new account

1. the old account has two factor authentication enabled and the new one does not. When I ran the above steps you mentioned, I created a new access token for each account and gave it all permissions. It worked on the new account but failed on the old account.

2. The username/prefix for the old account has a . and the new one does not.

I don't think its to do with .. I think the issue is to do with having 2 factor authentication enabled which is prob causing the caprover to fail using the auth token. The log message says denied: access forbidden which again makes me think that maybe having 2-factor authentication is causing the issue

[githubsaturn]

Interesting. Can you try enabling 2 factor auth for the second account to see if it makes any difference?

[ghost]

I've just tried on both account with 2-factor enabled and disabled. On the new account (it works fine with and without 2 factor enabled). On the old account it does not work with it being enabled or disabled.

This brings me back to what the differences are between the old and new account. The only difference being . in username/prefix.

I've also tried without the . in the prefix in the old account and no luck.

[ghost]

Just had a quick look at #85 , It looks similar to what I am seeing.

Looking at the images posted on issue #85 , it looks like its using - in its username/prefix.

It makes me think (again speculation) is it the special characters that's causing the issue.

The error message implies that its the token thats the issue but I'm able to use the token (login, pull and push)

[githubsaturn]

Okay... This is getting really weird... I just created an account with a dot and I was able to get the image deployed. Try this:

URL: registry.gitlab.com
USERNAME: test.username.dot
PREFIX: test.username.dot1
PASSWORD: NxuMy4LsV8rrzWypPr1W

Note that there is a 1 at the end of image prefix (but it's not on username).

And deploy this image:

registry.gitlab.com/test.username.dot1/test-project/my-build-image:459f2539e52c3eb0a8a09f97c1f4743a1fd94a5a

[ghost]

This is really weird. That has worked

------------------------- Tue Feb 09 2021 12:37:12 GMT+0000 (Coordinated Universal Time) Build started for test-app An explicit image name was provided (registry.gitlab.com/test.username.dot1/test-project/my-build-image:459f2539e52c3eb0a8a09f97c1f4743a1fd94a5a). Therefore, no build process is needed. Pulling this image: registry.gitlab.com/test.username.dot1/test-project/my-build-image:459f2539e52c3eb0a8a09f97c1f4743a1fd94a5a This process might take a few minutes. Build has finished successfully!

I can't see what is causing the old account to fail. I've removed 2 factor auth. I tried multiple auth tokens

[ghost]

Instead of using auth tokens, I though I'd trying using the main account password (since 2 factor is disabled). Same issue in the old account.

Here is the output

------------------------- Tue Feb 09 2021 12:46:40 GMT+0000 (Coordinated Universal Time)
Build started for test-app
An explicit image name was provided (registry.gitlab.com/naked-shef/web-client/my-build-image:5d22fa126f7c504e3aac53564f6eb8ca5dd0f696). Therefore, no build process is needed.
Pulling this image: registry.gitlab.com/naked-shef/web-client/my-build-image:5d22fa126f7c504e3aac53564f6eb8ca5dd0f696 This process might take a few minutes.
Build has failed!
----------------------
Deploy failed!
Error: (HTTP code 500) server error - Get https://registry.gitlab.com/v2/naked-shef/web-client/my-build-image/manifests/5d22fa126f7c504e3aac53564f6eb8ca5dd0f696: denied: access forbidden

Here is what the registry looks like:

[ghost]

Can you try with the following details:

URL: registry.gitlab.com USERNAME: m.j.mizan PREFIX: m.j.mizan PASSWORD:kvq9ApszKgNJLXXqtsrJ

image: registry.gitlab.com/naked-shef/test:latest

You should be able to see replicate the error

[githubsaturn]

This is incorrect. Your image prefix is is naked-shef in this example.

[ghost]

It's the same issue. You can also try registry.gitlab.com/naked-shef/web-client. It's the same issue.

I've just been trying afew different things to see what's going on.

If you try this image it works registry.gitlab.com/m.j.mizan/test:latest

[ghost]

I think I see where Im going wrong. It's the prefix that't wrong. I though it was supposed to be the username but it's not. It depends on the hierarchy of the project.

Thank you for the help and looking into it.