img-dim is slow and not cached

theor commented 3 years ago

Hello,

I realized that the img processing is executed multiple times per image - typically, if I add an image to a blog post, it will get processed 3 times (the blog post page itself, the /posts list page, the / list page) + once per tag . I have a site icon processed 22 times per build, for a total of 1721ms according to the 11ty benchmark log.

Caching would be great ; I see a solution in 2 steps :

keep an in-memory, transient cache indexed by path: should be easy to do. fill the cache in img-dim, invalidate it when the image changes. pay the cost once per image when starting watch mode, still way better
then get to a persistent cache. we could write an image-hashes.json file with the previous hash and generated file path, check it once during watch startup and compare the stored hashes against the actual hashes to invalidate or not the cache entries.

Thoughts ?

Also: it looks like gif2mp4 (which is even slower) is not running if an mp4 file already exists in the source folder (see https://github.com/google/eleventy-high-performance-blog/blob/676789e95149e0c92aee4bb4b89cabc6680db1c0/_11ty/video-gif.js#L32 ). is that intended to act as a basic cache ? in that case it should be if (await exists(join("_site", dest))) { instead of if (await exists(dest)) {

cramforce commented 3 years ago

Did you see which part of img-dim is slow?

I'm definitely supportive of caching it. I just have never seen it being slow. You definitely found a bug for the GIF special case, but otherwise the image processing is the slow part and obviously that is being done only once per build.

theor commented 3 years ago

await sizeOf("_site/" + src); is slow and done every time (that's where the time is spent the next 21 times in my example) srcset doing all the actual resizing is slow too, but done only once

cramforce commented 3 years ago

OK. sizeOf is an O(1) operation but I can see it getting a factor for N images :)

I'm definitely supportive of caching it.

cozarkd commented 2 years ago

I'm having serious problems with the image processing scripts. My project had an average of 1-2min building time, now it's +20min. So I can't use that in netlify or cloudflare pages. Probably need to do it locally in a separate script and store the files.

indcoder commented 2 years ago

Agree with @cozarkd . It has dramatically increased the build time and it is taking up so much memory that we cannot now use Netlify for building [so CI/CD is busted].

cramforce commented 2 years ago

Did you try running .persistimages.sh after a local build? It should eliminate all slowness.

cozarkd commented 2 years ago

Did you try running .persistimages.sh after a local build? It should eliminate all slowness.

oh, I thought that file was for remote images, makes sense now. In my use case I had to implement some features but not the full template. Problem solved, thanks

google / eleventy-high-performance-blog

img-dim is slow and not cached #107