search

verdaccio / monorepo

🏰 Core dependencies and plugins for verdaccio 5.x branch ⚠️ DEPRECATED
https://verdaccio.org
MIT License
81 stars 62 forks source link

S3 throttling due to package.json overwrites #404

Open webengineer opened 3 years ago

webengineer commented 3 years ago

Hi

I am receiving following error on Verdaccio 4.8.1 with aws-s3-storage plugin

error--- unexpected error: Please reduce your request rate.
SlowDown: Please reduce your request rate.
at Request.extractError (/opt/verdaccio/node_modules/aws-sdk/lib/services/s3.js:837:35)
at Request.callListeners (/opt/verdaccio/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/opt/verdaccio/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/opt/verdaccio/node_modules/aws-sdk/lib/request.js:688:14)
at Request.transition (/opt/verdaccio/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/opt/verdaccio/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /opt/verdaccio/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/opt/verdaccio/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/opt/verdaccio/node_modules/aws-sdk/lib/request.js:690:12)
at Request.callListeners (/opt/verdaccio/node_modules/aws-sdk/lib/sequential_executor.js:116:18)

most probably caused by S3 throttling.

I'd like to have configurable exponential backoff to be able to work-around errors like this.

-- Max

favoyang commented 3 years ago

Which command/endpoint raises this error?

And how many packages in the repository?

webengineer commented 3 years ago

It's a result of GET request

http <-- 500, user: null(1.2.3.4 via 127.0.0.1), req: 'GET /postcss-value-parser', error: internal server error

produced by npm install command.

Currently there are 5942 packages in the repository (21,016 s3 objects). Only 7 of them were published locally. Remaining are proxied.

favoyang commented 3 years ago

If you're using AWS s3 (not another vendor), it has a high throttling per prefix in an S3 bucket. It is usually unlikely to happen when hitting the individual endpoints, except for the /-/all or search endpoints which will go through all package metafiles.

You may want to check if it happens only once? or every time when installing the same package? or you share the s3 bucket with other busy services.

My point is that you can not just let your server-side endpoint wait a very long time to re-try a third-party service, your client-side (npm) will timeout first.

webengineer commented 3 years ago

Maybe half of errors are produced by requests for the same package. And remainder are from requests to other packages. The bucket isn't accessed by other services.

I can see that errors were happened less frequently during the last week (around 50 errors), then the previous week (around 2600 errors). I am using keyPrefix: verdaccio. According to https://stackoverflow.com/a/62145533, initially it might be considered as a the sole prefix for throttling all files, and with time s3 makes repartitions.

I suppose that throttling response is returned by s3 right after a request reaches it. In that case the retry request might be sent in 100 ms and would not cause npm client timeout.

webengineer commented 3 years ago

By querying s3 access logs I can see that only PUT requests are throttled (during last hours at least, when logging was enabled). I believe it's because every package metadata request is making a PUT operation. E.g. a package @types/node was requested from Verdaccio 15 times per minute, from which one request resulted in status 500 with the SlowDown message. In s3 access logs during that time there were 13 PUT operations with status 200, and 16 PUT operations with status 503 having SlowDown error code. Probably the exponential backoff or another algorithm already retries on the 503 SlowDown responses from s3. And after 10 retries Verdaccio responds with the error status 500.

I am wondering why the default 2 minutes of the uplinks maxage option does not prevent Verdaccio from writing metadata on every request. Is it expected?

webengineer commented 3 years ago

Maybe package.json is overwritten on each request because of malfunctioning ETag comparison? Because I am using KMS bucket encryption and thus ETag is not an MD5 digest of an object data.