400 Bad Request errors when pulling Docker Hub images behind a proxy

[diamonwiggins]

Hello,

On October 27th at around 3AM EST, we noticed that image pulls to Dockerhub via our proxy were failing with "400 Bad Request" errors. We tracked down the source of the bad request to the X-Forwarded-For header and noticed that not sending it in the request allowed image pulls to succeed. The issue seems to be when multiple addresses are provided in the X-Forwarded-For header. The following reproduces the issue:

USER=<valid user>
PASSWORD=<valid password>
TOKEN=`curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/nginx:pull" --user $USER:$PASSWORD | jq -r .token`

curl https://registry-1.docker.io/v2/library/nginx/manifests/latest  -D - -H "Authorization: Bearer ${TOKEN}" \
 -H "X-Forwarded-For: 34.102.1.93,10.2.130.127"

The same request with just one address in the header gets a normal response. Any assistance with this is appreciated as we believe this was working prior to the time I specified above. Thanks!

[mikeparker]

Looking into this now. From a quick glance it looks like your header is non-standard as per the spec https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For as you should separate IP addresses with a comma and a space, not just a comma. If you can change that it may start working, otherwise hold tight and I'll get back to you.

[mikeparker]

We've found the bug. Thanks for the report, this should be rolled out within the next day or two.

[diamonwiggins]

Thanks for the quick turnaround here. Much appreciated!

[divolgin]

@mikeparker unfortunately we don't control the header. The proxy actually uses a semicolon as a separator. The comma in the example above is us trying to troubleshoot this. The actual header looks like this: "X-Forwarded-For: 34.102.1.93;10.2.130.127"

[divolgin]

@mikeparker also, do you know if this is something that was changed on the dockerhub's side? It's possible the change is in our 3rd party proxy, so we are just trying to track it down.

[mikeparker]

@divolgin yes we're rolling out changes to support IPv6 so this is mostly likely our side, we're discussing what exactly to do here, as well as whether we want to support non-standard delimiters - the semicolon as a delimiter is not a standard mechanism for passing information.. but regardless that probably shouldn't make the request fail. If anything, it should just drop the header and carry on.

From our side we should be 'starting again' with your proxy IP rather than trusting whatever IP(s) you send us so this header shouldn't really be used by us anyway.

[github-actions[bot]]

We are clearing up our old issues and your ticket has been open for 6 months with no activity. Remove stale label or comment or this will be closed in 15 days.