Crashing when processing multiple images

[dottodot]

I trying to get this up and running on kubernetes but having issues when it comes to processing multiple images. i.e I have a page that loads 100+ thumbnails but only half of those load.

Adding more memory and cpu seem to help a bit, so I've currently got it on 2Gb memory and 2cpu, but there are still not all loading.

I could obviously keep throwing more resources at it but not sure what the recommended resources are to make it work or if there is some other issue causing it to not process all the images.

One thing I was wondering is that I'm using digitialocean spaces and I know they have rate limits but not sure how I log the errors from image-steam-s3

this is my current setup

const options = {
  processor: {
    concurrency: 2
  },
  storage: {
    defaults: {
      driverPath: 'image-steam-s3',
      endpoint: process.env.SPACES_ENDPOINT,
      accessKey: process.env.SPACES_KEY,
      secretKey: process.env.SPACES_SECRET
    },
    app: {
      bx: {
        bucket: 'bx-images'
      },
    },
    cache: {
      bucket: 'image-cache'
    }
  }
};

const server = http.createServer(
  new imgSteam.http.Connect(options).getHandler()
);

server.listen(PORT);

[asilvas]

Howdy.

Every module inherits from EventEmitter so you should be able to:

const isteamConnect = new imgSteam.http.Connect(options);
isteamConnect.on('warn', msg => console.warn('isteam:', msg.stack || msg);
isteamConnect.on('error', msg => console.error('isteam:', msg.stack || msg);
const server = http.createServer(isteamConnect.getHandler());

If you find the need to tune throttling, you can control the concurrency options: https://github.com/asilvas/node-image-steam#throttle-options

If you're using a single machine to host the service then I would expect those 100+ images to load very slowly, at least until the cache is primed. Original images are often multi-MB each, meaning if the cache is not primed, 100+ images can easily translate into hundred of MB's just in network transfers. Once the cache is primed, isteam generates Optimized Originals for all subsequent images (thumbnails or otherwise), and those OO images are a tiny fraction of the original size.

[dottodot]

Thanks

Actually just tried it using a gcloud function instead with just 512MB memory and the following throttle settings and it loads the images super fast even when they've not been cached, so think I might go with that option

  throttle: {
    ccProcessors: 8,
    ccPrefetchers: 100,
    ccRequests: 100
  }

[asilvas]

Closing. Can re-open if issue.