search

payloadcms / payload

The best way to build a modern backend + admin UI. No black magic, all TypeScript, and fully open-source, Payload is both an app framework and a headless CMS.
https://payloadcms.com
MIT License
21.28k stars 1.28k forks source link

Cloud storage plugin socket usage at capacity #6382

Open TimLanzi opened 1 month ago

TimLanzi commented 1 month ago

Link to reproduction

No response

Describe the Bug

I am using the cloud storage plugin with an S3 adapter pointing to DigitalOcean Spaces. I have users making a lot of edits relating to images lately. After a significant amount of uploads and image swapping, I get reports of performance issues on the admin dashboard and uploads failing. I looked at the logs and I found these error messages:

@smithy/node-http-handler:WARN socket usage at capacity=50 and 102 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 114 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 120 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 121 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 125 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 135 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 145 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 156 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 158 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 161 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 163 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 166 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 169 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 170 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.

These socket capacity warnings are sometimes preceded by error messages like this:

ERROR (payload): UnknownError
    err: {
      "type": "NoSuchKey",
      "message": "UnknownError",
      "stack":
          NoSuchKey: UnknownError
              at de_NoSuchKeyRes (/home/node/app/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4809:21)
              at de_CommandError (/home/node/app/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4747:19)
              at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
              at async /home/node/app/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
              at async /home/node/app/node_modules/@aws-sdk/middleware-signing/dist-cjs/index.js:225:18
              at async /home/node/app/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
              at async /home/node/app/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:173:18
              at async /home/node/app/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:97:20
              at async /home/node/app/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:120:14
              at async /home/node/app/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
      "name": "NoSuchKey",
      "$fault": "client",
      "$metadata": {
        "httpStatusCode": 404,
        "requestId": "[request id]",
        "attempts": 1,
        "totalRetryDelay": 0
      },
      "Code": "NoSuchKey",
      "BucketName": "bucket-name",
      "RequestId": "[request id]",
      "HostId": "[host id]"
    }

I have my Payload app hosted on a basic DigitalOcean Droplet (1 vCPU, 1GB RAM, 25GB storage, added 1GB swap space) if that helps at all.

To Reproduce

It seems like it happens when a lot of uploading is done. I haven't seen a clear-cut way to reproduce, but the best I can do is:

  1. Set up a payload app with the cloud storage plugin
  2. Point to a DigitalOcean Spaces bucket
  3. Create an upload collection
  4. Upload images in quick succession. Upload new images into existing records. Whatever can be done to trigger the errors.

Payload Version

2.14.1

Adapters and Plugins

"@payloadcms/bundler-webpack": "^1.0.6", "@payloadcms/db-mongodb": "^1.5.1", "@payloadcms/plugin-cloud-storage": "^1.1.2", "@payloadcms/plugin-nested-docs": "^1.0.12", "@payloadcms/plugin-seo": "^2.3.1", "@payloadcms/richtext-lexical": "^0.9.1", "@aws-sdk/client-s3": "^3.556.0", "@aws-sdk/lib-storage": "^3.556.0", "aws-crt": "^1.21.2"

Stroi commented 1 month ago

Also having this problem, after a while(not more than hour) we stop getting images and the same socket warning start showing up... Increasing the socket number didn't help. We are hosting on render, images in S3 and DB is Mongo

denolfe commented 3 weeks ago

@TimLanzi @Stroi Can either of you give me an idea of your throughput for uploads? Seems like this might be tricky to reproduce.

TimLanzi commented 3 weeks ago

It's been a while so I can't really give exact numbers. If I had to guess, I'd say 15 upload events in 10 minutes. Image sizes were around 1MB each. I also have imageSizes set up as follows:

    imageSizes: [
      {
        name: 'thumbnail',
        width: 400,
        height: 300,
        position: 'centre',
      },
      {
        name: 'card',
        width: 768,
        height: 1024,
        position: 'centre',
      },
      {
        name: 'tablet',
        width: 1024,
        height: null,
        position: 'centre',
      },
    ],
blobmold commented 1 week ago

I have the same issue with almost the same payloadcms settings as @TimLanzi, same payloadcms and s3 errors when uploading images.

Edit: looks like the plugin is creating s3 socket connections that its not closing @denolfe

creative-andrew commented 23 hours ago

I'm having exactly the same issue.