search

docker / hub-feedback

Feedback and bug reports for the Docker Hub
https://hub.docker.com
233 stars 48 forks source link

Remove IP from blacklist #2201

Closed sharon-amir closed 2 years ago

sharon-amir commented 2 years ago

Hi, We have 2 NAT instances we use for our parallel automation flows, It seems like we got blocked by dockerhub with the following error message: too many failed login attempts for username or IP address Could you please check and remove both IPs from blacklist? 3.14.215.238 3.13.244.50

ingshtrom commented 2 years ago

When are you getting this error? When you are pulling images from Docker Hub?

This usually means that you have authentication set up incorrectly and are continually trying to log in with invalid credentials. Have you verified that the Personal Access Token you are using is valid? Or if you are using username and password, then double-check those work on a different machine (local, maybe)?

After these failed login requests stop being sent, there is a short cool-down period. I would wait ~10 minutes and then login should work again.

sharon-amir commented 2 years ago

We have multiple vms which runs in parallel, all of them go through 2 NAT instances. Some of them pulls images, some have them just integrate with DH. username and password are being used and doubled checked.

ingshtrom commented 2 years ago

Ok, this specific error seems to only be used with authentication when pulling images.

It looks like there is a script using curl trying to hit Docker Hub Registry very frequently and continuing to receive HTTP status code 401 (unauthorized).

image

^^ These are by far and away the top two requests which have received 401 responses during January 31, 2022 (UTC).

sharon-amir commented 2 years ago

Thanks, but I'm not sure what is the root cause of the error. We have a scenario which we pull public image with anonymous user, then we pull images and integrate with dockerhub, both with authenticated user. is there any chance we reach to limit with anonymous user which affects our actions against DH with the authenticated user?

ingshtrom commented 2 years ago

We have a scenario which we pull public image with anonymous user, then we pull images and integrate with dockerhub, both with authenticated user.

What do you mean by "integrate with dockerhub"? If you mean push an image, then that is not possible with an anonymous user.

The endpoints I posted in the screenshot are coming from a client using the user agent curl, so if you are using curl directly against the Docker Hub API, that would be a good place to check for errors. These endpoints also require authentication, so curl https://registry-1.docker.io/v2/ -v or curl https://registry-1.docker.io/v2/_catalog?n=10 both return 401 without an Authorization header. I know that most Docker clients will automatically do a GET /v2/ and check the www-authenticate header to find out what server to authenticate with. In this case, it is the curl tool most likely, so it has been explicitly told to query this endpoint, which is likely unnecessary.

is there any chance we reach to limit with anonymous user which affects our actions against DH with the authenticated user?

I have verified this is not possible with our engineering team. So the error you are receiving is only possible when passing a username and bad password to the service.

Can you share what script you are using to do all of this? Or maybe it is some third-party tool which happens to set its user agent to curl?

sharon-amir commented 2 years ago

Our product integrates (login) with docker hub to allow the end user communicating with it and scan the requested image, the integration is done with authenticated user. Not sure how its implemented but the thing here is that its not consistent behavior as one application succeed to and the other fail with this error (both process are identical)

ingshtrom commented 2 years ago

Are you using customer credentials to access Docker Hub? Because that would make all of this make sense. e.g. if user1 entered correct credentials, their flows continue to work. If user2 enters incorrect credentials, then they continue to see failures.

Please correct me if I'm completely off-base.

sharon-amir commented 2 years ago

We have our own automation Docker Hub account that being used in parallel from multiple VMs (all of them have internet access from the NAT I mentioned above) and each VM run the same test flow which uses the same credentials.

ingshtrom commented 2 years ago

Hmm. Well that is odd indeed.

I have some more information that might help on your side.

Registry Auth Rate Limiting

This is a list of the hostnames, user agent, http path, and count of request received where one of the IPs you gave us received a 429 HTTP status code, but was not related to our anti-abuse and Docker Hub Registry pull rate limiting. These numbers are for all of February thus far.

image

Registry Pull Rate Limiting

Now if we look at the same query, but only include Docker Hub Registry pull rate limiting, we get this. It includes a long tail of 142 total combinations of hostname + user agent + http_path

image

Docker Hub Anti-Abuse Limiting

Lastly, if we look at the same query, but only for our anti-abuse rate limiting, we get these results:

image

So it seems that you are actually being hit by other Docker Hub rate limits a LOT more than the error too many failed login attempts for username or IP address (which also results in a 429 status code).

Out of curiosity, do you work for Aqua Security? I noticed a couple of forks of yours on GitHub related to Aqua Security.

tikolsky commented 2 years ago

Hi ingshtrom,

I am working with Sharon on the above framework that keeps failing with this "too many failed login attempts" failure. Do you have records of actual failed login attempts from the IPs 3.14.215.238 and 3.13.244.50? If I issue x failed login attempts from IP 3.14.215.238 for example, but then issue a correct login attempt right afterwards, may I be blocked with this error? because it comes from the same IP? for how long exactly?

Thank you so much, Lihi

ingshtrom commented 2 years ago

The first screenshot with the curl user agent was of the 401s as well. Below is the full list of the path, user agent, and number of responses with the path + user agent combination during the time frame of 02/08/2022 from 00:00-24:00 UTC.

401-for-02082022_hub-feedback-2201.csv

After a bit more digging, that error message actually happens against the username, not the IP. So that explains why it doesn't matter which NAT you are going through in your infra--they are both being rate limited since you are authenticating your requests. In addition, once you are rate limited, you cannot successfully log in until the cooldown period expires. This cool down period is 10 minutes.

To more precisely answer your example...

If I issue x failed login attempts from IP 3.14.215.238 for example, but then issue a correct login attempt right afterwards, may I be blocked with this error? because it comes from the same IP? for how long exactly?

The username (not IP) will be locked out for 10 minutes.

binman-docker commented 2 years ago

Closing this due to lack of activity, seems like it may be resolved.

Reminder for anyone finding this in the future - we pretty much never manually add IPs to our blocklist. If you get a 429 it's because of abuse rate limits, pull rate limits, or login rate limits. The system will automatically stop limiting you some period of time after you stop issuing requests above the rate limit. Details: https://docs.docker.com/docker-hub/download-rate-limit/