What is missing so that the build process does not exceed 1 minute with "responsive-loader" in my app Nuxtjs?

[johan-Rm]

With my nuxt js project, in my "assets" folder I have 150 images for a weight of 122M, I use the "responsive-loader", but when I run the "build" command the process is too long. How can I debug? why does it take so long to load 150 images? I will need to load 1500 images in the future ...

Moreover the images that I have already optimized with "jpegoptim -quality 20" still exceed the recommended size, how do you do?

WARNING in asset size limit: The following asset(s) exceed the recommended size limit (244 KiB).
This can impact web performance.
Assets: 
  img/activities-img-golf.daadaee.jpeg (856 KiB)

Can you tell me the best practice?

[dazuaz]

@johan-Rm Hello, One of the drawbacks of using this loader or any method that transform images at build time its gonna have that speed limitation, 150 images should be fine, and there are some steps you can take to improve your workflow,

But for 1500+ images I would look into using something like this https://aws.amazon.com/solutions/implementations/serverless-image-handler/

Anyways make sure you are using it with the Sharp adapter, and you can use the disable flag in development to skip the transformation step, I do have a Nuxt project with 200+ images and It takes about a minute to load, but only a few of the images I need them to be fullscreen.

[pkpowell]

We have 300+ images (expecting that to tripple) and were hoping this could be sped up by not resizing all the pics that haven't changed. Is there no way to ignore the images already present during build? All the pics are cached in node_modules/.cache/nuxt/dist/client so it seems like it'd be a matter of comparing dates and only add if older or not present.

[johan-Rm]

thank you for your feedback and your valuable information, @pkpowell I will also explore this track there, if I find a solution I will inform you

[pkpowell]

Great, thanks!

On 2 Nov 2020, at 19:41, REMY Johan notifications@github.com wrote:

thank you for your feedback and your valuable information, @pkpowell https://github.com/pkpowell I will also explore this track there, if I find a solution I will inform you

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dazuaz/responsive-loader/issues/124#issuecomment-720653987, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGUMMFXMJFHZEQWMPTFWHDSN34ONANCNFSM4S7STAPA.

[pkpowell]

I did some sleuthing and discovered that all images are being converted to all sizes referenced in the site. So for instance the front page requires images of size 840w, an overview page requires 180w, a details page 530w etc. The image below is should only be 530w but for some reason all possible sizes are being emitted. Additionally the original file is emitted twice. One time with its width and also with its original file name.

Any ideas on how to optimise this? Is this a webpack issue or maybe an issue with aceforth/nuxt-optimized-images which calls responsive-loader?

Cheers Phil

My webpack config is

{
            name: "images/[name]-[width]-[contenthash:7].[ext]",
            placeholder: false,
            quality: 60,
            adapter: require("./js/responsive.js")
}

The images are loaded via

require(`~/assets/uploads/${w.image.filename()}?size=180&format=webp`)

This image is only required in one template at 530w so should only emit 2 files (1 jpg 1 webp).

image-7322-fc6e867.jpg
image-1440-4054217.webp
image-1440-0c7a678.jpg
image-840-010f365.webp
image-840-6c786f1.jpg
image-530-d8e4066.jpg
image-530-7b006a7.webp
image-270-f67c19e.jpg
image-270-317650e.webp
image-180-71374b5.jpg
image-180-20e567e.webp

[ldrummond]

@pkpowell When you do a dynamic image request through webpack: require('~/assets/uploads/${w.image.filename()}?size=180&format=webp')

Webpack is unable to resolve the image at build time, so it creates a context which includes every image within the referenced directory (in this case "~/assets/uploads/"). So if you are using a dynamic require for three sizes (840, 1440, 530...), all with the same folder path, then webpack generates a version of every file in the referenced directory for each one of those sizes.

Here is a more detailed description on stackoverflow. ...

Also, I am wondering the same thing regarding caching generated images, specifically to speed up build time on Netlify. Did you find any solutions there?

[DotIN13]

After adding avif to the build tasks, the build time has exceeded 2 minutes. Would really love a cache feature to avoid constant rebuilds.

[DotIN13]

Managed to use the cache option to lower rebuild time. As in my comment in #52

[dazuaz]

Hey guys,

You can use responsive-loader@next to try out experimental cache option cacheDirectory: true

This will cache the result object but not the image files, I have been debating on how to implement this, if I save the generated images on the cache, and copy them in, there is an expensive copy operation for every build, and not ideal when you have to many images.

That being said, having the results in cache, you can copy the generated images to static/img (outside build-land)

options: {
  adapter: require('responsive-loader/sharp'),
  outputPath: 'img', // _nuxt/img
  publicPath: '/img', // static/img
  cacheDirectory: true,
},

default cache location is node_modules/.cache/responsive-loader

Any feedback is appreciated as we figure out a good way to handle this the best way possible.

[Trinovantes]

I'm getting 404s for my images with cacheDirectory: true in development mode. I haven't tried it in production. The public paths are the same but dev server is unable to serve the images.

I'm assuming this has something to do with the images not copied over to webpack dev server's memory file system if responsive-loader detects the output json exists in cache?

[dazuaz]

@Trinovantes Yes, the images are only produced once, when they are not found in the cache, or when the options change (cache key).

What this saves in the cache is the generated results object:

        srcSet: ...,
        images: Array,
        src: ...,
        toString: () => String,
        placeholder: ...,
        width: ...,
        height: ...,

I am still debating if adding an option to also store the generated images on the cache and copying them over for every build., this does unnecessary disk read/writes but it is simpler for the end-user.

For now, you can create a script that copies files generated by webpack build to your static folder. And running only when adding new images.

[Trinovantes]

I tried setting

  adapter: require('responsive-loader/sharp'),
  outputPath: path.resolve(myRootDir, 'static', 'public'),
  publicPath: '/public',
  cacheDirectory: true,

But don't see any generated images in my static directory Nevermind I reread your comment. Yea it's kind of clunky to have to run nuxt build, copy files, then nuxt dev

[ldrummond]

@dazuaz Thanks for your work on this. For my current use case, caching the generated images would be really helpful. I use responsive loader with a static site generator and a file-based CMS, so when CMS editors add a new image, the site rebuilds and generates responsive sizes for that image.

Right now I am using a system that combines 'responsive-loader' with a PR of the recently deprecated 'cache-loader' that allows for caching of generated files. Here is the PR that adds the caching of files. This system isn't perfect though, and I'm with you in that there is probably a more efficient approach, but even using this caching method helps speed up site generating on content changes.

[Trinovantes]

I've been using v3 for awhile now and it has greatly reduced my build times. One pain point I've encountered is that If I add new images while using webpack-dev-server, responsive-loader generates a cached file in node_modules/.cache even though dev server writes files to memory-fs instead of /dist (and thus not persisted between builds). As a result, I have to purge .cache and run a production build every time I want to add new images.

Can you add a new option that allows me to disable emitting new cache files but still try to check node_modules/.cache?

[Porges]

This will cache the result object but not the image files, I have been debating on how to implement this, if I save the generated images on the cache, and copy them in, there is an expensive copy operation for every build, and not ideal when you have to many images.

You could maybe hardlink the images to the ones in the cache? This would save a copy, at least.