HELM proxy The proxy does not use the original absolute address

loverto commented 5 years ago

Thanks for creating an issue! Please fill out this form so we can be sure to have all the information we need, and to minimize back and forth.

What are you trying to do? I try to use one helm repo proxy ( to https://charts.gitlab.io ) but I have : helm repo add nexusrepo http://nexus/repository/helm-stable-proxy/ helm fetch nexusrepo/mongodb Error: Failed to fetch http://10.100.100.11/repository/helm-gitlab-charts/auto-deploy-app-0.2.4.tgz : 404 Not Found

Server-side log open DEBUG see the following log

2018-09-17 14:34:39,554+0800 DEBUG [qtp872755663-49]  *UNKNOWN org.sonatype.repository.helm.internal.proxy.HelmProxyFacetImpl - Fetching: GET https://charts.gitlab.io/auto-deploy-app-0.2.4.tgz HTTP/1.1
2018-09-17 14:34:39,555+0800 DEBUG [qtp872755663-49]  *UNKNOWN org.sonatype.nexus.httpclient.outbound - https://charts.gitlab.io/auto-deploy-app-0.2.4.tgz > GET /auto-deploy-app-0.2.4.tgz HTTP/1.1
2018-09-17 14:34:39,555+0800 DEBUG [qtp872755663-49]  *UNKNOWN org.sonatype.nexus.internal.httpclient.SharedHttpClientConnectionManager - Connection request: [route: {s}->https://charts.gitlab.io:443][total kep
t alive: 1; route allocated: 0 of 20; total allocated: 1 of 200]

What feature or behavior is this required for?
How could we solve this issue? (Not knowing is okay!)
Anything else?

There is a problem here, adding that the chart is not the official address, that is to say, the downloaded address is not the same domain name as the chart, there will be problems, for example, https://charts.gitlab.io His index.yaml format is like this


  - created: 2018-06-15T22:48:32.992821306Z
    description: GitLab Runner
    digest: 034f08a9e6bb733e7e56b3bf2542498da75872d1e373dce70334485829f68d0e
    icon: https://gitlab.com/uploads/-/system/project/avatar/250833/runner_logo.png
    keywords:
    - git
    - ci
    - deploy
    maintainers:
    - email: support@gitlab.com
      name: GitLab Inc.
    - email: dj@gitlab.com
      name: DJ Mountney
    name: gitlab-runner
    sources:
    - https://hub.docker.com/r/gitlab/gitlab-runner/
    - https://docs.gitlab.com/runner/
    urls:
    - https://gitlab-charts.s3.amazonaws.com/gitlab-runner-0.1.23.tgz
    version: 0.1.23
  - created: 2018-06-15T22:35:02.572502983Z
    description: GitLab Runner
    digest: c82d3d98a810322bf87edf47934814e6d3f24f26f614acb1d322f6586a96d593
    icon: https://gitlab.com/uploads/-/system/project/avatar/250833/runner_logo.png
    keywords:
    - git
    - ci
    - deploy
    maintainers:
    - email: support@gitlab.com
      name: GitLab Inc.
    - email: dj@gitlab.com
      name: DJ Mountney
    name: gitlab-runner
    sources:
    - https://hub.docker.com/r/gitlab/gitlab-runner/
    - https://docs.gitlab.com/runner/
    urls:
    - https://gitlab-charts.s3.amazonaws.com/gitlab-runner-0.1.22.tgz
    version: 0.1.22
  - created: 2018-06-14T16:30:13.81292576Z
    description: GitLab Runner
    digest: be75401889b81945faa52305bf734117648513f0e7814286ed1a52e2652758a7
    icon: https://gitlab.com/uploads/-/system/project/avatar/250833/runner_logo.png
    keywords:
    - git
    - ci
    - deploy
    maintainers:
    - email: support@gitlab.com
      name: GitLab Inc.
    - email: dj@gitlab.com
      name: DJ Mountney
    name: gitlab-runner
    sources:
    - https://hub.docker.com/r/gitlab/gitlab-runner/
    - https://docs.gitlab.com/runner/
    urls:
    - https://gitlab-charts.s3.amazonaws.com/gitlab-runner-0.1.21.tgz
    version: 0.1.21
  - created: 2018-06-07T15:22:28.486461965Z
    description: GitLab Runner
    digest: d8258443107d13792875f6ae3fdb90cc4fcaabc5f83f3bbf3317143c44493b1f
    icon: https://gitlab.com/uploads/-/system/project/avatar/250833/runner_logo.png
    keywords:
    - git
    - ci
    - deploy
    maintainers:
    - email: support@gitlab.com
      name: GitLab Inc.
    - email: dj@gitlab.com
      name: DJ Mountney
    name: gitlab-runner
    sources:
    - https://hub.docker.com/r/gitlab/gitlab-runner/
    - https://docs.gitlab.com/runner/
    urls:
    - https://gitlab-charts.s3.amazonaws.com/gitlab-runner-0.1.20.tgz
    version: 0.1.20
  - created: 2018-06-06T14:26:15.253296358Z
    description: GitLab Runner
    digest: 3bf24c26fc9cc64bb6d860fccba99672e90445dab5ce1d745115042615759b6f
    icon: https://gitlab.com/uploads/-/system/project/avatar/250833/runner_logo.png
    keywords:
    - git
    - ci
    - deploy
    maintainers:
    - email: support@gitlab.com
      name: GitLab Inc.
    - email: dj@gitlab.com
      name: DJ Mountney
    name: gitlab-runner
    sources:
    - https://hub.docker.com/r/gitlab/gitlab-runner/
    - https://docs.gitlab.com/runner/
    urls:
    - https://gitlab-charts.s3.amazonaws.com/gitlab-runner-0.1.19.tgz
    version: 0.1.19
  - created: 2018-05-30T10:35:21.5443457Z
    description: GitLab Runner
    digest: 540bf4756d198ae75824e733a3d1740cf37464d9ec33e80f62e0097ad2453e26
    icon: https://gitlab.com/uploads/-/system/project/avatar/250833/runner_logo.png
    keywords:
    - git
    - ci
    - deploy
    maintainers:
    - email: support@gitlab.com
      name: GitLab Inc.
    - email: dj@gitlab.com
      name: DJ Mountney
    name: gitlab-runner
    sources:
    - https://hub.docker.com/r/gitlab/gitlab-runner/
    - https://docs.gitlab.com/runner/
    urls:
    - https://gitlab-charts.s3.amazonaws.com/gitlab-runner-0.1.18.tgz
    version: 0.1.18
  - created: 2018-05-25T17:47:52.665684883Z
    description: GitLab Runner
    digest: 88886a71b1ec6065117bda3341f3c4f0bbe2360fad8a22c6d8a8dbcd3c7db880
    icon: https://gitlab.com/uploads/-/system/project/avatar/250833/runner_logo.png
    keywords:
    - git
    - ci
    - deploy
    maintainers:
    - email: support@gitlab.com
      name: GitLab Inc.
    - email: dj@gitlab.com
      name: DJ Mountney
    name: gitlab-runner
    sources:
    - https://hub.docker.com/r/gitlab/gitlab-runner/
    - https://docs.gitlab.com/runner/
    urls:
    - https://gitlab-charts.s3.amazonaws.com/gitlab-runner-0.1.17.tgz
    version: 0.1.17
  - created: 2018-05-21T14:12:40.310853577Z
    description: GitLab Runner
    digest: 06fd24339ed112b961e79d40f4509efd350c2bb5d3ed66b214ce28f067b07aea
    icon: https://gitlab.com/uploads/-/system/project/avatar/250833/runner_logo.png
    keywords:
    - git
    - ci
    - deploy
    maintainers:
    - email: support@gitlab.com
      name: GitLab Inc.
    - email: dj@gitlab.com
      name: DJ Mountney
    name: gitlab-runner
    sources:
    - https://hub.docker.com/r/gitlab/gitlab-runner/
    - https://docs.gitlab.com/runner/
    urls:
    - https://gitlab-charts.s3.amazonaws.com/gitlab-runner-0.1.16.tgz
    version: 0.1.16
  - created: 2018-05-18T13:13:54.657084419Z
    description: GitLab Runner
    digest: 6ac756566fbbfebd7b49f18e61357747c28424eec9ac1a2aa295f475bd0f4e94
    icon: https://gitlab.com/uploads/-/system/project/avatar/250833/runner_logo.png
    keywords:
    - git
    - ci
    - deploy
    maintainers:
    - email: support@gitlab.com
      name: GitLab Inc.
    - email: dj@gitlab.com
      name: DJ Mountney
    name: gitlab-runner
    sources:
    - https://hub.docker.com/r/gitlab/gitlab-runner/
    - https://docs.gitlab.com/runner/
    urls:
    - https://gitlab-charts.s3.amazonaws.com/gitlab-runner-0.1.15.tgz
    version: 0.1.15
  - created: 2018-05-18T12:12:35.192390219Z
    description: GitLab Runner
    digest: 3eb963ee322434e56bb0b9878476a1a734e9b1dce26901a96c1dff8ee16f4e1b
    icon: https://gitlab.com/uploads/-/system/project/avatar/250833/runner_logo.png
    keywords:
    - git
    - ci
    - deploy
    maintainers:
    - email: support@gitlab.com
      name: GitLab Inc.
    - email: dj@gitlab.com
      name: DJ Mountney
    name: gitlab-runner
    sources:
    - https://hub.docker.com/r/gitlab/gitlab-runner/
    - https://docs.gitlab.com/runner/
    urls:
    - https://gitlab-charts.s3.amazonaws.com/gitlab-runner-0.1.14.tgz
    version: 0.1.14
  - created: 2018-05-15T12:09:04.553132449Z
    description: GitLab Runner
    digest: 82192920f22fd497b5fe08556a2af6ad41b024950c46f69376003adc9dbd1102
    icon: https://gitlab.com/uploads/-/system/project/avatar/250833/runner_logo.png
    keywords:
    - git
    - ci
    - deploy
    maintainers:
    - email: support@gitlab.com
      name: GitLab Inc.
    - email: dj@gitlab.com
      name: DJ Mountney
    name: gitlab-runner
    sources:
    - https://hub.docker.com/r/gitlab/gitlab-runner/
    - https://docs.gitlab.com/runner/
    urls:
    - https://gitlab-charts.s3.amazonaws.com/gitlab-runner-0.1.13.tgz
    version: 0.1.13
  kubernetes-gitlab-demo:
  - apiVersion: v1
    created: 2018-05-10T08:51:48.195111007Z
    description: GitLab running on Kubernetes suitable for demos
    digest: e41d0aef3616687c4327f9d447d5132bc7325cb1e5f380591a40496462441ab8
    home: https://about.gitlab.com
    icon: https://gitlab.com/gitlab-com/gitlab-artwork/raw/master/logo/logo-square.png
    keywords:
    - git
    - ci
    - cd
    - deploy
    - issue tracker
    - code review
    - wiki
    maintainers:
    - email: support@gitlab.com
      name: GitLab Inc.
    - name: Mark Pundsack
    - name: Jason Plum
    - name: DJ Mountney
    name: kubernetes-gitlab-demo
    sources:
    - https://hub.docker.com/r/gitlab/gitlab-ce/
    - https://docs.gitlab.com/omnibus/
    urls:
    - https://gitlab-charts.s3.amazonaws.com/kubernetes-gitlab-demo-0.1.29.tgz
    version: 0.1.29
generated: 2018-08-28T22:30:11.426662552Z

dtshepherd commented 5 years ago

See PR #4. I'm guessing its the same issue.

DarthHater commented 5 years ago

Hmmm, this is gonna be tricky actually. Being they use a different URL for the charts than the actual repo, we'd need to make outbound requests to whatever the URL indicates. I'll see if I can noodle on this in a bit, but other PRs, etc... are welcome!

The url itself needs to be set to relative versus absolute so that requests are made through Nexus Repo (otherwise what's the point of using it, hah), but this is kinda interesting in that they are maintaining two different URLs (one for the repo, one for their charts).

loverto commented 5 years ago

There are two different addresses for a helm repo, there is a way to handle it,

Assuming that the storage addresses in the url are the same, then you can directly proxy the storage address.
Adding url addresses are different, then you need to store a different url in index.yaml in the helm component.

Fortunately, the situation is now the first one in chart.gitlab.io, then the solution is to set https://gitlab-charts.s3.amazonaws.com to the remote proxy address.

like this

DarthHater commented 5 years ago

@loverto you are correct, you can totally do that as long as the index.yaml and the charts are accessible via that URL. It might be worth reaching out to the maintainers of that repo and asking them why they maintain two separate URLs, maybe they can merge them. I can totally understand wanting to host your charts on S3 or something akin, but if the index is also there it just begs the question of why you'd have two different URLs in general.

notanaverageman commented 4 years ago

I have also encountered this issue with a few helm repos. (e.g. https://helm.elastic.co/ or https://oteemo.github.io/charts)

In elastic's case they serve the .tgz files on the same domain, but under different subpaths. For example the .tgz file for apm-server is at: https://helm.elastic.co/helm/apm-server/apm-server-7.6.1.tgz. But, from what I see, URLs are never taken into account whether they are absolute or relative and the index.yaml on Nexus contains just apm-server-7.6.1.tgz. The below code should be the place where the URLs are constructed.

https://github.com/sonatype-nexus-community/nexus-repository-helm/blob/b7afdc5117c63a6ac35a295c4f844bed6bac3e83/nexus-repository-helm/src/main/java/org/sonatype/repository/helm/internal/createindex/CreateIndexServiceImpl.java#L113-L119

If more and more repos deprecate the stable repo and take the same route to host their files, proxying will possibly be useless.

LeoQuote commented 4 years ago

The issue in https://issues.sonatype.org/browse/NEXUS-20929 was marked as "wont fix", don't know why, hope this can be fixed in the comming releases.

mlukaretkyi commented 3 years ago

sonatype-nexus-community / nexus-repository-helm

HELM proxy The proxy does not use the original absolute address #8