Galactory no longer support proxying to galaxy.ansible.com out of the box

[mshonichev]

Hello, Brian!

First of all thank you for the great software package you've delivered!

I try to use Galactory in scenario 'proxy/caching the upstream Ansible Galaxy' as to lower the outgoing bandwidth for the CI.

Recently, Ansible Galaxy NG has gone out of the beta and they bumped their main site to the 'v3' API version.

If we browse directly to https://galaxy.ansible.com/api/ we would see that no 'v2' API is supported now.

Which leads me to the problem, Galactory can no longer proxy requests to the upstream Galaxy, because ansible-galaxy client seems to ultimately using 'v2' version when negotiating with Galactory, no matter which --api-version option value I use.

Scenario:

Galactory galactory:0.11.1 image is running with following options:

--listen-addr 0.0.0.0 --listen-port 80 --server-name galaxy.local --artifactory-path https://artifactory.local:443/artifactory/ansible_collections/ --artifactory-access-token $(ART_TOKEN) --proxy-upstream https://galaxy.ansible.com --galaxy-auth-type access_token --use-galaxy-auth --prefer-configured-auth --publish-skip-configured-auth

client version & test command:

$ ansible --version
ansible [core 2.15.4]
  config file = /tmp/aaa/ansible.cfg
  configured module search path = ['/Users/m.shonichev/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.11/site-packages/ansible
  ansible collection location = /Users/m.shonichev/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.11.4 (main, Jun 20 2023, 16:52:35) [Clang 13.0.0 (clang-1300.0.29.30)] (/usr/local/opt/python@3.11/bin/python3.11)
  jinja version = 3.1.2
  libyaml = True

$ ansible-galaxy collection install --server http://galaxy.local  cisco.nxos

Starting galaxy collection install process
Process install dependency map
ERROR! Failed to resolve the requested dependencies map. Could not satisfy the following requirements:
* cisco.nxos:* (direct request)

Log:

Tue, Oct 10 2023 5:52:46 pm INFO:galactory:Rewriting 'http://galaxy.local/api/v2/collections/cisco/nxos/' to 'https://galaxy.ansible.com/api/v2/collections/cisco/nxos/'
Tue, Oct 10 2023 5:52:46 pm INFO:galactory:https://galaxy.ansible.com/api/v2/collections/cisco/nxos/?page_size=100
Tue, Oct 10 2023 5:52:46 pm INFO:galactory:None
Tue, Oct 10 2023 5:52:46 pm INFO:galactory:{'X-Request-Id': '5cdd2b7897d98e98c3d41f7aa9e73146', 'X-Real-Ip': '10.0.0.1', 'X-Forwarded-For': '10.0.0.1', 'X-Forwarded-Host': 'galaxy.local', 'X-Forwarded-Port': '443', 'X-Forwarded-Proto': 'http', 'X-Forwarded-Scheme': 'http', 'X-Scheme': 'http', 'Accept-Encoding': 'identity', 'User-Agent': 'ansible-galaxy/2.15.4 (Darwin; python:3.11.4)', 'Accept': 'application/json, */*'}
Tue, Oct 10 2023 5:52:46 pm INFO:werkzeug:10.0.0.2- - [10/Oct/2023 14:52:46] "GET /api/v2/collections/cisco/nxos/ HTTP/1.1" 404 -

However, if use galaxy.ansible.com directly, the collection is installed successfully:

$ ansible-galaxy collection install cisco.nxos --server https://galaxy.ansible.com

Starting galaxy collection install process
Process install dependency map
Starting collection install process
Downloading https://galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/artifacts/cisco-nxos-5.2.1.tar.gz to /Users/m.shonichev/.ansible/tmp/ansible-local-69069ixz4ij5e/tmp0vgzpo5b/cisco-nxos-5.2.1-rpenpawh
Installing 'cisco.nxos:5.2.1' to '/Users/m.shonichev/.ansible/collections/ansible_collections/cisco/nxos'
Downloading https://galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/artifacts/ansible-netcommon-5.2.0.tar.gz to /Users/m.shonichev/.ansible/tmp/ansible-local-69069ixz4ij5e/tmp0vgzpo5b/ansible-netcommon-5.2.0-pisa6k3g
cisco.nxos:5.2.1 was installed successfully

Is there any workaround or quick fix for this situation?

[briantist]

Hi @mshonichev , welcome! Thanks for the including so much info in this report, that's very helpful.

This line is especially interesting:

Tue, Oct 10 2023 5:52:46 pm INFO:galactory:Rewriting 'http://galaxy.local/api/v2/collections/cisco/nxos/' to 'https://galaxy.ansible.com/api/v2/collections/cisco/nxos/'

It means that the request from the ansible-galaxy client came in as v2.

Typically that means that the client detected no v3 from the server. The only reason that should have been the case is that galactory was not serving it.

If you hit http://galaxy.local/api/ what do you see?

Also, because you're using the INFO log level, the output from when galactory first starts should show which Galaxy API versions it's initializing.

Finally, it might be helpful to add -vvvv to the ansible-galaxy run.

---

I have one theory so far: ansible-galaxy caches upstream API responses, so perhaps it's using a cached v2 response? Try adding --no-cache to the galaxy call as well.

[mshonichev]

Yep, you're absolutely right!

So, production setting to proxy for https://galaxy.ansible.com is to use --api-version v3.

AND

Don't forget to clear response cache at client side, e.g.:

$ ansible-galaxy collection install --server http://galaxy.local --clear-response-cache cisco.nxos

Starting galaxy collection install process
Process install dependency map
Starting collection install process
Downloading http://galaxy.local/download/cisco-nxos-5.2.1.tar.gz?galactory_upstream_url=https://galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/artifacts/cisco-nxos-5.2.1.tar.gz to /Users/m.shonichev/.ansible/tmp/ansible-local-70469xx4sc8et/tmptjzewn3r/cisco-nxos-5.2.1-4crdg6b1
Installing 'cisco.nxos:5.2.1' to '/Users/m.shonichev/.ansible/collections/ansible_collections/cisco/nxos'
cisco.nxos:5.2.1 was installed successfully

I see you don't plan to automatically forward v2 request for v3 upstream for now, so this issue can be closed.

Thank you so much for your support!

[briantist]

Glad to hear that worked!

Indeed I don't plan to explicitly support v2 <-> v3 proxying; if it comes it would be as a side effect of re-doing how proxying is done using an actual client that's aware of both versions, but I doubt it will be any time soon.

Thanks very much for your interest in the software, I'd love to hear more about how you're using it and what your experience has been.

[briantist]

So, production setting to proxy for https://galaxy.ansible.com is to use --api-version v3.

One other thing to clarify (for anyone reading) is that you don't have to use --api-version v3 unless you want to never serve v2. By default it will serve both (but proxying would not work cross-versions).

In this case clearing the cache (or using --no-cache) is what would have fixed the issue.