d4l-data4life / kirby3-static-site-generator

Static site generator plugin for Kirby 3+. With this plugin you can create a directory with assets, media and static html files generated from your pages. The result is an even faster site with less potential vulnerabilities.
MIT License
139 stars 7 forks source link

Timeout on page with large srcsets #10

Closed mrflix closed 4 years ago

mrflix commented 5 years ago

Thank you Jonathan for all the work you put into this plugin and thanks for making it public :)

I have the following problem when using it: When running it on a page with lots of large images and image versions the generator is very slow, shows no progress and eventually dies.

Expected Behavior

The static files should get generated. A progress indicator would show the progress of the media generation, like imagekit does: image

Current Behavior

The generator stops unexpectedly after some time: CleanShot 2019-07-11 at 10 46 01 One weird thing I noticed: the size of the media directory goes up and down: static site generator media folder size going down I expect it to go up because missing thumbs are being generated. But since it's going down I suspect that it's deleting and re-creating already created thumbs. This would explain why the generator doesn't eventually process the pages faster and come to an end without a timeout.

Possible Solution

Steps to Reproduce

This is my srcsets config:

'srcsets' => [
  'default' => [640, 1280, 2560, 3840, 5120],
  'rondel-product' => [320, 640, 1280],
  'gallery' => [640, 1024, 1440, 2560],
  'slideshow' => [1280, 2560, 3840, 5120],
  'slideshow-portrait' => [640, 1024, 1280, 2560],
  'product-menu' => [128, 256, 512, 1024]
]

When I remove all use of srcsets and just use one resized version per image the generator works as expected and doesn't timeout.

Context

Additional plugins: builder, color, focus, matomo. Of all of them focus might be the only one that interferes.

jonathan-reisdorf commented 5 years ago

Thank you, I'll see what I can do here. The way it's structured is to render all these pages in one row. Restructuring this is some work but not impossible. Not quite sure though why the MB in your folder are decreased from time to time - no files are deleted by the plugin except in the beginning, so I'm guessing it's either Kirby or a bug in your file manager.

scsskid commented 5 years ago

increase php max execution time should solve the problem, the default in most envs is 30s

cvholten commented 5 years ago

Hello,

I have got 1.6 GB of media files. I am interrested in any solution as well. After some minutes of generating static pages I get the info "The JSON response from the API could not be parsed. Please check your API connection."

mrflix commented 5 years ago

The latest Kirby 3.2.3 changelog says:

Media files are now correctly generated again in multi-site setups

Maybe that fixes it.

jonathan-reisdorf commented 5 years ago

Hi @rs-vonholten, that is a lot of media files. Can you tell me what the request responds with when you get this error? To do so, go into your browser's developer tools and open the network tab, then once you see the error, click on the corresponding request and open the response tab.

cvholten commented 5 years ago

Hi @rs-vonholten, that is a lot of media files. Can you tell me what the request responds with when you get this error? To do so, go into your browser's developer tools and open the network tab, then once you see the error, click on the corresponding request and open the response tab.

I get an error page by my host as response:

Sehr geehrter Besucher

500 - Interner Serverfehler

_________________________________________________________ 

Sehr geehrter Besucher,

leider ist ein Problem aufgetreten. Die angeforderte Seite hat einen Script-Fehler verursacht.

Haben Sie sich vielleicht vertippt oder eine alte URL aufgerufen? Wenn nicht, informieren Sie bitte den Webmaster dieser Homepage per Email. Um zu der vorherigen Seite zurückzukehren, verwenden Sie bitte einfach die "Zurück" - Taste Ihres Browsers.

 

_________________________________________________________

jonathan-reisdorf commented 5 years ago

Then it's probably either a script timeout or some other resource-related problem. Hard to tell if the actual PHP error is not displayed. (Can you enable the displaying of PHP errors?) Resolving this issue will take a while because it is a complex problem (because there are three ways to use the plugin, and any change in one of these should not break the others). Also, so far I've only used the plugin on sites with a negligible amount of media assets.

cvholten commented 5 years ago

Then it's probably either a script timeout or some other resource-related problem. Hard to tell if the actual PHP error is not displayed. (Can you enable the displaying of PHP errors?) Resolving this issue will take a while because it is a complex problem (because there are three ways to use the plugin, and any change in one of these should not break the others). Also, so far I've only used the plugin on sites with a negligible amount of media assets.

Is it possible to exclude the entire media folder from being copied, so I have to copy it manually?

jonathan-reisdorf commented 5 years ago

Not yet, but adding that would be an option (also particularly handy if you have your own implementation, e.g. where you upload new images to a CDN like cloudinary and the images in kirby are not relevant). I'll create a PR for that later today.

jonathan-reisdorf commented 5 years ago

The option will be added with https://github.com/d4l-data4life/kirby3-static-site-generator/pull/19

To skip copying the files, add 'skip_media' => true to your config. In this case however, any media folder present in the output directory will always be erased. To prevent that, combine it with preserve:

'preserve' => ['media'],
'skip_media' => true

Also, when making any change to media files, you'll now need to manually take care of regenerating and copying them (maybe a kirby file hook would work?)

cvholten commented 5 years ago

The option will be added with #19

To skip copying the files, add 'skip_media' => true to your config. In this case however, any media folder present in the output directory will always be erased. To prevent that, combine it with preserve:

'preserve' => ['media'],
'skip_media' => true

Also, when making any change to media files, you'll now need to manually take care of regenerating and copying them (maybe a kirby file hook would work?)

Thank you. This worked.

When I do not skip media, the media folder content will be deleted everytime I use your plugin. Is it necessary to delete all files over and over again?

jonathan-reisdorf commented 5 years ago

@rs-vonholten yes, because we do not know / keep track if images have been changed. See also last comment of https://github.com/d4l-data4life/kirby3-static-site-generator/issues/3 (when the mentioned issue is solved in Kirby, changing this might be easier)

jonathan-reisdorf commented 4 years ago

Hi everyone :wave: Sorry for the belated response, but I've now released a version https://github.com/d4l-data4life/kirby3-static-site-generator/releases/tag/1.0.10 which would drastically improve rebuild times for sites with lots of media, because the media cache is now finally taken into account. Please let me know if that works and helps to reduce this issue. The initial build might still be slow if the media cache is not populated, yet, but follow-up builds should be way faster.

jonathan-reisdorf commented 4 years ago

Closing this because with the latest update the issue seems fixed on subsequent runs. Feel free to reopen if there's an issue.