Speed up compile time

[fregante]

I noticed that Jekyll was taking increasingly longer to compile even though I only have 7 pages.
7 seconds for a live preview is not quite live.

The culprit is the opening and MD5ing of the images even if they are already generated. Every {% image %} is taking upwards of 100ms. Disabling the script brings the total compile time to 2 seconds.

When generating the image's url/src, we have the name and preset, so what about looking up the generated images with those two? Maybe by generating them with this filename: name-preset-generatedSize-digest and look up any files that start with name-preset before doing anything else.

Also, any specific reason to include the digest? Could we skip that too?

Can you think of any other way to remember the images we already generated?

[robwierzbowski]

Nice research. We need to use the MD5 because image content can change, and then need re-generation. I know the compile time goes up a lot with many images, but hadn't considered the time it takes to do the md5 hashing. Maybe we can limit the hash to run only on 10k of image or so, so we're not processing a 15 meg image.

[fregante]

Should we perhaps default on the file's modified date instead?

While I like automation, I would rather have to manually force a rebuild, in the rare cases where it's needed (changed image and same modified date), rather than waiting for an increasingly long time on every jekyll build. Perhaps with a flag on jekyll or simply by deleting the old generated files.

---

In the meanwhile at least I managed to get grunt to only livereload the CSS or JS files instead of rebuilding the whole thing.

[robwierzbowski]

I definitely want to regenerate each file based on whether its contents have changed. When a person edits a source file and then re-compiles it should re-generate that image.

Modified date can change without the contents changing, and vice versa, but only on rare occasions. That might be a good compromise in speed vs. certainty.

I didn't see anything earlier in this issue with Grunt, and this plugin doesn't do anything to CSS or JS, so not sure what that's about.

Are you seeing all of your files being re-generated on build? The plugin already has a caching mechanism (see https://github.com/robwierzbowski/jekyll-image-tag#managing-generated-images). If you're using Grunt you'll need to do some extra configuration. The images are cached in the output directory. If the output directory is being erased by Grunt every time, you're going to regenerate all of your images every time.

[robwierzbowski]

Haven't forgotten about this, just on more pressing repos with limited time now.

[fregante]

Same here, I was using this for a personal project and now I have to work on something else. I'll definitely come back to it eventually.

[robwierzbowski]

I'm confused as to why this is happening though. Excluding the generated file dir from the .jekyll and build directory clean tasks should work.

[fregante]

I don't think the problem ever was the regeneration of images taking too long, it's simply the MD5ing. I suggest to default to created/modified date check + size check, instead of MD5 (or to offer it as an option). I mean, what are the chances of getting both of them exactly the same when replacing an image? I would think that they are much faster than MD5ing dozens of 2-4MB images

What I said about Grunt: the watch task was rebuilding the whole site (and thus running through the images) even when I edited CSS and JS files (obviously unnecessary), so I optimized that and got around this issue for edits outside Jekyll's HTML files. Nothing to do with your plugin, actually :smile:

[robwierzbowski]

Cool cool. I will work on the md5 time issue — we'll see if we can get a good compromise between speed and meaningful unique identifiers.

On Sunday, May 25, 2014, Federico Brigante notifications@github.com wrote:

I don't think the problem ever was the regeneration of images taking too long, it's simply the MD5 taking that long. I suggested to default to created/modified date check + size check, instead of MD5 (or to offer it as an option). I mean, what are the chances of getting both of them exactly the same when replacing an image? I would think that they are much faster checks than MD5ing dozens of 2-4MB images

What I said about Grunt: the watch task was rebuilding the whole site (and thus running through the images) even when I edited CSS and JS files (obviously unnecessary), so I optimized that and got around this issue for edits outside Jekyll's HTML files. Nothing to do with your plugin, actually [image: :smile:]

— Reply to this email directly or view it on GitHubhttps://github.com/robwierzbowski/jekyll-image-tag/issues/16#issuecomment-44141007 .

Rob Wierzbowski @robwierzbowski http://twitter.com/#!/robwierzbowski http://github.com/robwierzbowski http://robwierzbowski.com

[lieutenantstan]

Hi there, is there any progress on this issue? I started to use this plugin for my photoblog, I'd dump my big (~5-10MB) 'source' images into a directory and have jekyll-image-tag generate smaller more web-friendly images.

At first it was great, but as I added more photos, my jekyll build time increased dramatically. At least now I know the root cause. It'd be great if we could get an update on the issue. Thanks!

update:

I ended up writing a lazy-check which determine if a generated image exists for the source image with the specified preset. It's a lot faster then creating a MiniMagick obj from the source image and generating an md5 of it. Obviously it won't detect if a source image file changes, but for the purposes of a 'live'-updating jekyll dev process it's fine for me and brought my build time w/ 150 images down from ~35s to 4s.

[funkenstrahlen]

@Daniel0524 I have the same problem with a lot of images right now. Could you post your tweak, so I can use it too?

[funkenstrahlen]

I have about 100 images in my blog posts. Every images takes about 1s to process. This takes very long to build!

Especially as it processes every image on each run again.

[lieutenantstan]

Hey @i42n here's my modified version of the script, not exactly the most beautiful piece of code - and there is some other stuff in there too that I added to integrate with S3 - but it works for me.

https://github.com/Daniel0524/nordness.net/blob/master/jekyll_blog/_plugins/image_tag.rb

[fregante]

This doesn't seem to be a problem with jekyll 3's watch mode :)