assets/scully-routes.json is NOT generated while using baseFilter and routeFilter

Amith278 commented 2 years ago

🐞 Bug report

Description

While running 'npx scully --routeFilter', scully-routes.json is NOT generated, which leads to "TimeoutError: Navigation timeout of 30000 ms exceeded we will retry rendering this page up to 3 times".

I have noticed, if scully-routes.json is missing, this issue arises. If the scully-routes.json is present and i run the above command it works perfectly.

I even tried 'npx scully --scanRoutes --routeFilter'.

The above step is missing which running with routeFilter/baseFilter

🔬 Minimal Reproduction

💻Your Environment

Angular Version:




Angular CLI: 13.0.4
Node: 14.16.0
Package Manager: npm 6.14.11
OS: win32 x64

Angular: 13.0.3
... animations, common, compiler, compiler-cli, core, forms
... platform-browser, platform-browser-dynamic, router

Package                         Version
---------------------------------------------------------
@angular-devkit/architect       0.1300.4
@angular-devkit/build-angular   13.0.4
@angular-devkit/core            13.0.4
@angular-devkit/schematics      13.0.4
@angular/cdk                    12.2.13
@angular/cli                    13.0.4
@schematics/angular             13.0.4
rxjs                            6.6.7
typescript                      4.4.4

Scully Version:




  "@scullyio/init": "^1.1.4",
  "@scullyio/ng-lib": "^1.0.0",
  "@scullyio/scully": "^1.1.1",

🔥 Exception or Error








Puppeteer error while rendering "/newsroom/canalys-global-smartphone-market-Q4-2021" TimeoutError: Navigation timeout of 30000 ms exceeded  we will retry rendering this page up to 3 times.
--
265 | Puppeteer error while rendering "/newsroom/global-pc-market-Q4-2021" TimeoutError: Navigation timeout of 30000 ms exceeded  we will retry rendering this page up to 3 times.

SanderElias commented 2 years ago

This is by design. If you use the filtering options, Scully works on a subset of the pages. That means that the route-list it uses is only a fraction of the complete list. If that gets written to disk, it overwrites a probably more complete route-list. Also, there is an issue, if you use a filter for /blog/:sluf, but /docs/:slug, when you write the complete list n that case, there will be routes in there, that are not generated.

If you want it to write the list, it has to work off the complete list, otherwise, there are too many risks of the routes.json being out of sync with what is generated. If you have a specific use case, you can always create your own custom plugin that updates the route list. But that kind of plugin will have to have some knowledge about your solution, that is not something that can be solved in a generic way.

Amith278 commented 2 years ago

@SanderElias

Use case is : Generate static page for only new dynamic path. I don't want to generate static pages for default paths every build. So only for blog/:id

If a new blog/:id is comes in and it's not present in route-list, the build timesout.

Any work around?

SanderElias commented 2 years ago

@Amith278 Can you make a minimal repro of this? I just tested, and if I just add a file into my blog folder, and run Scully, it seems to work?

robinComa commented 2 years ago

@SanderElias , you should remove all the scully-routes.json files before running scully to reproduce the issue :

find . -type f -name scully-routes.json -delete
npx scully --noPrompt --scanRoutes --routeFilter /blog/new-or-updated-blog-slug

When the scully-routes.json doesn't exit, I have this error :

Json Route plugin loading data for "/blog/:slug" (node:23004) UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, stat '/Users/robincomadelperier/Documents/Workspace/placeme/placeme-website/dist/static/assets/scully-routes.json'
    at Object.statSync (fs.js:1127:3)
    at Object.statSync (/Users/robincomadelperier/Documents/Workspace/placeme/placeme-website/node_modules/graceful-fs/polyfills.js:319:16)
    at loadHandledRoutes (/Users/robincomadelperier/Documents/Workspace/placeme/placeme-website/node_modules/@scullyio/scully/src/lib/utils/serverstuff/handleUnknownRoute.js:103:39)
    at handleUnknownRoute (/Users/robincomadelperier/Documents/Workspace/placeme/placeme-website/node_modules/@scullyio/scully/src/lib/utils/serverstuff/handleUnknownRoute.js:26:41)
    at Layer.handle [as handle_request] (/Users/robincomadelperier/Documents/Workspace/placeme/placeme-website/node_modules/express/lib/router/layer.js:95:5)
    at next (/Users/robincomadelperier/Documents/Workspace/placeme/placeme-website/node_modules/express/lib/router/route.js:144:13)
    at Route.dispatch (/Users/robincomadelperier/Documents/Workspace/placeme/placeme-website/node_modules/express/lib/router/route.js:114:3)
    at Layer.handle [as handle_request] (/Users/robincomadelperier/Documents/Workspace/placeme/placeme-website/node_modules/express/lib/router/layer.js:95:5)
    at /Users/robincomadelperier/Documents/Workspace/placeme/placeme-website/node_modules/express/lib/router/index.js:286:22
    at param (/Users/robincomadelperier/Documents/Workspace/placeme/placeme-website/node_modules/express/lib/router/index.js:367:14)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:23004) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:23004) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

When I use --scanRoutes with --routeFilter , I would like to regenerate the scully-routes.json and build only the matching routes.

SanderElias commented 2 years ago

I'm afraid the options --scanRoutes and --routeFilter are incompatible with each other. The filter options prevent writing the route files because they would end up in incomplete ones.

There should be a waring instead of an error, so I will eave this issue open so we can take care of that.

WhyINeedToFillUsername commented 2 years ago

Hi, I would like to add my reasoning to this question.

My use-case and expectations are: have a list of events on the main page (route "/") and their details on separate pages (routes /detail/:id). Run scully build to generate first version of sites, ship it to CDN. Then, when some new event occurs, generate only page for that. I expect that should be done by using npx scully --baseFilter="/detail/newEventId".

This works (= only the new page is generated) but scully doesn't add this new route to the list of handled routes in the scully-routes.json (I would expect the new route will be added there, or at least to have some option to add it there with a switch). Since the baseFilter switch doesn't do this, I tried it with the scanRoutes npx scully --scanRoutes --baseFilter="/detail/*", but it also didn't add it to the list.

Am I missing something? Is there another way to do this? Or should we handle this use case some other way?

scullyio / scully