search

epics-containers / edge-containers-cli

command line shortcuts for epics containers developers
Apache License 2.0
3 stars 1 forks source link

Helm charts access denied on GHCR #125

Open gilesknap opened 2 months ago

gilesknap commented 2 months ago

This is a bit worrying but maybe I just broke something that I can't immediately see.

On Fri I started getting the following error when trying to do ec deploy-local. In the past I have seen the same error intemittantly but now its happening every time and is still doing it on Mon.

@marcelldls please can you see if this is happening for you and take a look into it if it is? Thanks.

[hgv27681@pc0116 bl47p]$ ec -v deploy-local services/bl47p-ea-test-01/
kubectl get namespace p47-iocs -o name
Deploy bl47p-ea-test-01 TEMPORARY version 2024.4.66ba-b from /scratch/hgv27681/work/bl47p/services/bl47p-ea-test-01 to domain p47-iocs
Are you sure ? [y/N]: y
helm dependency update /scratch/hgv27681/work/bl47p/services/bl47p-ea-test-01/../../helm/shared

Command Failed:
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "gitlab" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 1 charts
Downloading ioc-instance from repo oci://ghcr.io/epics-containers
Save error occurred:  could not download oci://ghcr.io/epics-containers/ioc-instance: failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Aepics-containers%2Fioc-instance%3Apull&service=ghcr.io: 403 Forbidden

Error: could not download oci://ghcr.io/epics-containers/ioc-instance: failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Aepics-containers%2Fioc-instance%3Apull&service=ghcr.io: 403 Forbidden
gilesknap commented 2 months ago

also note that a direct helm pull is seeing 403

[hgv27681@pc0116 bl47p]$ helm pull oci://ghcr.io/epics-containers/ioc-instance:3.4.0
Error: GET "https://ghcr.io/v2/epics-containers/ioc-instance/tags/list": GET "https://ghcr.io/token?scope=repository%3Aepics-containers%2Fioc-instance%3Apull&service=ghcr.io": unexpected status code 403: denied: denied

That error seems to imply that it cant get the listing of versions.

If the package ioc-instance was private I would expect to see this error. But it is not. It sort of looks like the index listing is now private.

One thing to check if you can reproduce this is what happens if you log in to github first with a PAT.

marcelldls commented 2 months ago 
[esq51579@pc0146 bl01t]$ helm pull oci://ghcr.io/epics-containers/ioc-instance:3.4.0
Error: invalid_reference: invalid tag
[esq51579@pc0146 bl01t]$ helm pull oci://ghcr.io/epics-containers/ioc-instance --version 3.4.0
Pulled: ghcr.io/epics-containers/ioc-instance:3.4.0
Digest: sha256:89637f92ebb55363d739f330602d731a58d283d2df199e2dca68b086e26c7bbe
marcelldls commented 2 months ago

No issue with local deploy

[esq51579@pc0146 bl01t]$ ec -v deploy-local services/bl01t-ea-test-01/
kubectl get namespace esq51579 -o name
Deploy bl01t-ea-test-01 TEMPORARY version 2024.4.85b2-b from /home/esq51579/WIP/copy/bl01t/services/bl01t-ea-test-01 to domain esq51579
Are you sure ? [y/N]: y
helm dependency update /home/esq51579/WIP/copy/bl01t/services/bl01t-ea-test-01/../../helm/shared
helm dependency update /home/esq51579/WIP/copy/bl01t/services/bl01t-ea-test-01; helm package /home/esq51579/WIP/copy/bl01t/services/bl01t-ea-test-01 --app-version 2024.4.85b2-b
bash -c "helm upgrade --install bl01t-ea-test-01 /home/esq51579/WIP/copy/bl01t/services/bl01t-ea-test-01/ioc-instance-3.4.0.tgz --namespace esq51579  2> >(grep -v 'found symbolic link' >&2) "
Release "bl01t-ea-test-01" does not exist. Installing it now.
NAME: bl01t-ea-test-01
LAST DEPLOYED: Mon Apr 15 09:30:32 2024
NAMESPACE: esq51579
STATUS: deployed
REVISION: 1
TEST SUITE: None
gilesknap commented 2 months ago

Thats good. Please can you try a bl47p IOC? any will do

marcelldls commented 2 months ago 
[esq51579@pc0146 bl47p]$ ec ps
| name              | version       | running | restarts | deployed            |
|-------------------|---------------|---------|----------|---------------------|
| epics-opis        | 2024.2.3      | true    | 0        | 2024-02-18 14:15:15 |
| bl47p-ea-dcam-01  | 2024.4.1      | true    | 0        | 2024-03-26 12:47:15 |
| bl47p-ea-dcam-02  | 2024.4.1      | true    | 0        | 2024-03-26 12:52:48 |
| bl47p-ea-panda-01 | 2024.4.1      | true    | 0        | 2024-03-26 13:29:26 |
| bl47p-ea-test-01  | 2024.3.d8d3-b | true    | 0        | 2024-03-25 15:25:17 |
| bl47p-mo-ioc-01   | 2024.4.1      | true    | 0        | 2024-03-26 12:56:32 |
[esq51579@pc0146 bl47p]$ ec deploy bl47p-ea-test-01 2024.4.1 
Release "bl47p-ea-test-01" has been upgraded. Happy Helming!
NAME: bl47p-ea-test-01
LAST DEPLOYED: Mon Apr 15 14:36:58 2024
NAMESPACE: p47-iocs
STATUS: deployed
REVISION: 2
TEST SUITE: None
[esq51579@pc0146 bl47p]$ ec ps
| name              | version  | running | restarts | deployed            |
|-------------------|----------|---------|----------|---------------------|
| epics-opis        | 2024.2.3 | true    | 0        | 2024-02-18 14:15:15 |
| bl47p-ea-dcam-01  | 2024.4.1 | true    | 0        | 2024-03-26 12:47:15 |
| bl47p-ea-dcam-02  | 2024.4.1 | true    | 0        | 2024-03-26 12:52:48 |
| bl47p-ea-panda-01 | 2024.4.1 | true    | 0        | 2024-03-26 13:29:26 |
| bl47p-ea-test-01  | 2024.4.1 | true    | 0        | 2024-04-15 14:36:58 |
| bl47p-mo-ioc-01   | 2024.4.1 | true    | 0        | 2024-03-26 12:56:32 |
[esq51579@pc0146 bl47p]$ ec deploy-local services/bl47p-ea-test-01/
Deploy bl47p-ea-test-01 TEMPORARY version 2024.4.ce5b-b from /home/esq51579/WIP/copy/bl47p/services/bl47p-ea-test-01 to domain p47-iocs
Are you sure ? [y/N]: y
Release "bl47p-ea-test-01" has been upgraded. Happy Helming!
NAME: bl47p-ea-test-01
LAST DEPLOYED: Mon Apr 15 14:40:32 2024
NAMESPACE: p47-iocs
STATUS: deployed
REVISION: 3
TEST SUITE: None
[esq51579@pc0146 bl47p]$ ec ps
| name              | version       | running | restarts | deployed            |
|-------------------|---------------|---------|----------|---------------------|
| epics-opis        | 2024.2.3      | true    | 0        | 2024-02-18 14:15:15 |
| bl47p-ea-dcam-01  | 2024.4.1      | true    | 0        | 2024-03-26 12:47:15 |
| bl47p-ea-dcam-02  | 2024.4.1      | true    | 0        | 2024-03-26 12:52:48 |
| bl47p-ea-panda-01 | 2024.4.1      | true    | 0        | 2024-03-26 13:29:26 |
| bl47p-ea-test-01  | 2024.4.ce5b-b | true    | 0        | 2024-04-15 14:40:32 |
| bl47p-mo-ioc-01   | 2024.4.1      | true    | 0        | 2024-03-26 12:56:32 |
gilesknap commented 2 months ago

Thanks Marcell - this is reassuring. Now I just need to work out what is going on for me!