Closed pi0 closed 3 years ago
Do you have plans for this to support generate
as well? As discussed on discord some time ago I have been thinking about the future of nuxt-generate-cluster for quite some time and I think the best course would be to work towards deprecating that package, more precisely to step away from 'just' supporting Cluster as it would be nice to have remote support as well. Or did you step away from full IPC support currently (as it seems the above now only mentions Cluster Worker)?
@pimlie Sure. I'm not just started it. BTW would you please attach the link to your related works and hints for generating to this topic? That would be really helpful.
it would be nice to have remote support as well.
Indeed planned too. Not just Inter Process communication workers.
My thoughts with regards to the future of nuxt-generate-cluster (ngc). These thoughts should be read as in an ideal world because looking at the download count of ngc I dont think the need for this is large enough and therefore not (yet) worth the full effort. But we could probably pick certain features.
ngc uses the cluster package to support multi-threaded generating, mostly useful when you have a large number of dynamic routes. When running ngc it starts a master which is responsible for:
Both master as workers use the same nuxt buildDir
. This works because the master only builds at startup before any worker is started (or even any route is retrieved).
Messaging between master and workers is done through the IPC channel which the Cluster package provides.
N
routes should complete by increasing the number of hosts we run the generate workers onIn general this is to run ngc daemonised and support infinite scaling. Fully support: $time_to_finish = $routes.length / $workers.count
with the (almost) only limitation that $time_to_finish = $time_to_generate_single_heaviest_route + $some_little_overhead
.
Find a suitable messaging protocol, preferably one that understand a master, proxies and services. In ngc I implemented a (maybe poor-man's) message-broker as a proof of concept for this.
Here both the master as a proxy could be understood as a single message-broker instance, with the exception that you can only have one master and multiple proxies.
Eg if you have a master daemon running on a remote host for distributed generation then its only job would be to run as a communication end-point for the real master to start generate workers on-demand. The started generate workers would be proxies themselves as well which connect with the master themselves (at least atm). This is implemented by having each message-broker instance register with an alias, so eg a watchdog worker registers with the master under the alias watchdog
and then a generate worker on some host only has to send a message to alias watchdog
and the proxies in between make sure that messages ends up in the message-broker instance with the alias watchdog
. Additionally it understands services so e.g. a watchdog message-broker instance can register a different method for when a worker reports it succesfully generated a route vs when it reports an error.
When I said poor-man's implementation that is because atm:
There is probably a lot more to say about this but I am dried-up for the moment. Will continue in another comment if I think of other things :)
Can we close this one @pi0 ?
Clarification: After huge efforts on this solution we decided to discontinue because of architucture complexities it brings.
Alternative approach is rewriting nuxt3 server engine in a way that can be embeded as a serverless function or worker (for nuxt dev
). A demo is public here
Meta
Working branch: pi0/feat/workersObjectives
nuxt dev
run and reload whennuxt.config.js
is being changed. So we don't have any memory leaks.@nuxt/cli
dependency fromnuxt-start
for making it fast as šDescription
The requirement is that for forking a new process (Cluster Worker) or running in a new context, we need an entry point, which should be fast, standalone and most important accepting arguments like cli commands. The simplest solution is to execute our cli script on each worker but it not only adds lot's of overhead but also makes problems for Cluster.
The proposed idea is creating embeddable standalone code that will be executed by normal CLI commands or directly by cluster runners. For passing arguments to them, we can pass a JSON argument as the only argument like this:
This makes them small because we don't need to depend on any argv parser also powerful as we can pass any kind of args without extra configs.
Remarks
nuxt.config
and mostly create a newNuxt
instance.Workers
Workers are emebeded recepies to do certain tasks. They can hopefuly will replace all CLI logics as well as custom programatic usage for use cases like Lambda/Serverless.
Server
Builder
Combinitions [TODO]
We combine workers in different use cases.
Start
Dev