Allow whole directories to include (Include additional URLs setting)

[thomasfrobieter]

I have a problem with a theme which loads some Javascript files dynamically. Also they.. re-generate those JS files on cache clear, the name of those files is changing then. So i have no chance to add the single files.

I tried to include the whole folder "/wp-content/themes/blocksy/static/bundle/", but this doesn't work.

Any chance you can & will extend the include additional URLs setting to do this?

(Version 6.6.20)

[leonstafford]

Hi @thomasfrobieter please try the 6.6.21 version from releases page, just in case it fixes that, but unlikely.

So, as long as those files exist while crawling and are referenced in either HTML or CSS files, they should be picked up.

Could you please add the steps to reproduce for me here, if that's a publicly available theme?

It's nearing end of night for me and brain a bit foggy, but let's try to dissect:

• site exports index.html and style.css for example
• within these files, there is a reference to script1234.js

on each update to the site in backend, the script name changes ie, script5678.js, but that reference to the script should also be updated within the HTML/CSS files that WP serves, else it would be broken on your dev site, too.

So, where it the broken reference happening?

ie, if you open up browser's console > Network tab and reload your static homepage, there is a 404 for a JS file? What is the path for the missing file it's expecting and who's responsible for telling it to load (ie, some HTML, CSS or possibly even other JS file?)

If you can't easily track it down, you can copy the 404 script's filename, export site as zip and then recursively search for reference to that filename to see if it's being told to load from an HTML, CSS or JS file.

We shouldn't need to manually include such files, but if need be, we do have a filter in V7 plugin that I'll check if we have in V6 already, that can modify the initial crawl list and we can force it to grab all files from a directory on disk.

[leonstafford]

tracking similar issue reported with same theme here: https://wordpress.org/support/topic/blocky-wp2static-uncaught-in-promise-chunkloaderror-loading-chunk-3-failed/

[thomasfrobieter]

Thank you you so much again! BTW: I don't see any donation options on the wp2static website? Am i able to donate somewhere?

Sadly, no change with v6.6.21.

The Wordpress instance is just a local docker container, but the export is public: https://wp2static.spielwiese.webks.de/ Here is the latest export as ZIP: https://send.firefox.com/download/fa41690fc37ce161/#AysiQ9mMLUT99AwT6iPNgQ

The theme itself is public: https://wordpress.org/support/theme/blocksy/

within these files, there is a reference to script1234.js

I think the problem is, that the missing JS files are loaded inside another Javascript file (main.js):

              var a, c = document.createElement("script");
              c.charset = "utf-8", c.timeout = 120, o.nc && c.setAttribute("nonce", o.nc), c.src = function(e) {
                  return o.p + "" + e + "." + {
                      0: "0af8b497d0914c00e11a",
                      1: "94066414e4689cee2aef",
                      2: "3546d1cf97fbbe3572ac",
                      3: "780707ab4d1e679dc221",
                      5: "c2f180cdbe732e535477",
                      6: "94e78eb143dff0433fff",
                      7: "334c81ab7095934e63f7",
                      8: "beca7dcda5797486012e",
                      9: "c0f54c9ca945f614789e",
                      10: "c8a2fa6badcca351ab79"
                  } [e] + ".js"
              }(e);

The JS errors (just on smaller devices): GET https://wp2static.spielwiese.webks.de/wp-content/themes/blocksy/static/bundle/2.3546d1cf97fbbe3572ac.js [HTTP/2 404 Not Found 173ms]

GET https://wp2static.spielwiese.webks.de/wp-content/themes/blocksy/static/bundle/3.780707ab4d1e679dc221.js [HTTP/2 404 Not Found 172ms]

Uncaught (in promise) ChunkLoadError: Loading chunk 2 failed. (error: https://wp2static.spielwiese.webks.de/wp-content/themes/blocksy/static/bundle/2.3546d1cf97fbbe3572ac.js)

tracking similar issue reported with same theme here: https://wordpress.org/support/topic/blocky-wp2static-uncaught-in-promise-chunkloaderror-loading-chunk-3-failed/

Yes, this is also created by me, i pointed them to this issue here.

[andreiglingeanu]

Hey guys,

Blocksy developer here.

Indeed the problem is that the exported bundle folder doesn't have the files it tries to load (see the 404s): https://tppr.me/R4Llp. This looks like something from WP2Static needs to be configured to get all the files included in the final build.

Also, when I've tried to export a site using WP2Static & Blocksy -- the bundle folder contained everything and no JS errors were present, even on small screens.

Here's how the real bundle folder should look: https://tppr.me/2bFSE. Note the dynamic files generated by Webpack there. And yes, I know, it looks scary at first but I want to assure you it all gets loaded conditionally, depending on the various features you enable in Blocksy, so in real frontends it all is very lean :)

Hope this helps to track down where's the issues and please let me know if there's anything I can do to help you track this down.

[thomasfrobieter]

Hey @andreiglingeanu,

thank you so much for your feedback, can you tell me what version of WP2Static / Static HTML Output you've used? Have you changed anything in the plugin configuration what may be relevant here?

I've played alot with the settings.. but without any success.

Here's how the real bundle folder should look: https://tppr.me/2bFSE. Note the dynamic files generated by Webpack there. And yes, I know, it looks scary at first but I want to assure you it all gets loaded conditionally, depending on the various features you enable in Blocksy, so in real frontends it all is very lean :)

Im pretty sure this is state of the art shit :D Just trying to figure out whats the problem here.. no offense.

[andreiglingeanu]

@thomasfrobieter used just the last version of the plugin from WP.org (6.6.7), without any changes in the settings (I've activated permalinks and changed Target Directory though). Here's my output from the plugin, for me it was able to detect the bundle/ folder and other static files too :)

https://send.firefox.com/download/f848ab597dd7635f/#n2LVq6RTQrYRbXulxPPtSg

Btw, this button was useful for me to see which files are included in the final build: https://tppr.me/2xxtC And all the files that get a 404 for you (and you get the error) are present there for me. Can you provide here the list of files that you get from this repport?

---

And yea, that conditional loading is done with the dynamic import() for loading modules, all handled by webpack.

[thomasfrobieter]

@andreiglingeanu Yeah, this is the 'old' plugin it was renamed recently (https://github.com/WP2Static/wp2static vs. https://github.com/WP2Static/static-html-output-plugin).

I've switched to the new module (https://github.com/WP2Static/static-html-output-plugin/releases/tag/6.6.21) because of: https://github.com/WP2Static/static-html-output-plugin/issues/111#issuecomment-648077675

My crawl list is very short, but those entries are definitely not all detected files as you can see in the demo (https://wp2static.spielwiese.webks.de/):

/ /ads.txt /beispiel-seite/ /blog/ /blog/2020/ /blog/2020/06/ /blog/2020/06/17/ /blog/2020/06/17/hallo-welt/ /blog/2020/06/18/ /blog/2020/06/18/blocksy-child/ /blog/2020/06/18/blocksy/ /blog/category/allgemein/ /favicon.ico /robots.txt /sitemap.xml

[andreiglingeanu]

@thomasfrobieter funny, we had the same issue with variables but with the page speed module (because of their outdated CSS parser written in CPP): https://github.com/apache/incubator-pagespeed-mod/issues/1971

I'll try to test the 6.6.21 revision and get back to you -- I've missed the recent rename of the plugin

[andreiglingeanu]

Sorry for the late reply -- finally got my hands on testing the latest version of the plugin. I've tested the master version and it indeed has the problem mentioned by @thomasfrobieter and I also get the shorter crawl list.

I guess it'll be very hard to crawl the dynamically loaded JS files that main.js is referencing, only by statically analysing the main.js file, see how they are referenced.

The easiest solution for this is going to be a filter for adding a couple of files to the final output (which I'll be happy to implement from the Blocksy side) or by allowing a wildcard in the Include additional URLs.

Please note that you don't have to statically replace the URLs in the main.js file or in the dynamic JS files, we are loading the dynamic files relative to this variable in JS, which you already replace correctly :)

Hope this helps to come up with a solution for this, please let me know if there's any other way I can help!

[thomasfrobieter]

@leonstafford I just tried to copy the JS files manually and it worked.

So if you allow to add entire folders, it would fix this thing (:

[leonstafford]

Glad to hear, @thomasfrobieter. Indeed. I like the new approach of "infinite crawling"/not trying to infer from WP what files need to included, but realizing that we need a bit of a combination of both techniques to ensure all sites can be crawled.

[andreiglingeanu]

Some things just can't be detected only by using a crawler... Allowing a glob of files to be included in the final build will indeed fix this :)

[leonstafford]

Definitely. Also, sorry for not replying sooner to you, @andreiglingeanu