ProjectEvergreen / greenwood

Greenwood is your workbench for the web, embracing web standards from the ground up to empower your stack from front to back.
https://www.greenwoodjs.io
MIT License
96 stars 9 forks source link

performance benchmarking / optimizing Greenwood #970

Open thescientist13 opened 2 years ago

thescientist13 commented 2 years ago

Type of Change

Summary

Can't say I've focused to heavily on build time performances but since Zach Leatherman of 11ty put together a bench mark blog post and repo of some of the top SSGs, and so I figured I would test Greenwood just at least get a sense of where it sat amongst the rest.

Screen Shot 2022-08-05 at 7 06 09 PM

I made a fork and decided to add Greenwood. 😬 🤞

I tried it on two machines and both had varying struggles:

    % sh ./bench-greenwood.sh 100
    [bench] Clean complete
    [bench] Setup complete 100 pages
    -------------------------------------------------------
    Welcome to Greenwood (v0.26.0) ♻️
    -------------------------------------------------------
    Initializing project config
    Initializing project workspace contexts
    Generating graph of workspace files...
    building from local sources...
    Running Greenwood with the build command.
    pages to generate 
     /
     /posts/ad-deserunt-cillum-consectetur-occaecat copy/
     /posts/ad-deserunt-cillum-consectetur-occaecat/
     /posts/ad-in-id-ex-sunt copy/
     /posts/ad-in-id-ex-sunt/
     /posts/ad-in-laborum-anim-ex copy/
     /posts/ad-in-laborum-anim-ex/
     /posts/ad-nulla-tempor-ipsum-nostrud copy/
     /posts/ad-nulla-tempor-ipsum-nostrud/
     /posts/adipisicing-dolore-ea-do-magna copy/
     /posts/adipisicing-dolore-ea-do-magna/
     /posts/adipisicing-dolore-ullamco-in-amet copy/
     /posts/adipisicing-dolore-ullamco-in-amet/
     /posts/adipisicing-exercitation-do-nisi-consectetur copy/
     /posts/adipisicing-exercitation-do-nisi-consectetur/
     /posts/aliqua-consequat-proident-reprehenderit-id copy/
     /posts/aliqua-consequat-proident-reprehenderit-id/
     /posts/aliqua-dolor-ipsum-fugiat-ullamco copy/
     /posts/aliqua-dolor-ipsum-fugiat-ullamco/
     /posts/aliqua-eiusmod-cupidatat-proident-ex copy/
     /posts/aliqua-eiusmod-cupidatat-proident-ex/
     /posts/aliqua-esse-irure-labore-ex copy/
     /posts/aliqua-esse-irure-labore-ex/
     /posts/aliqua-sint-ut-nulla-sunt copy/
     /posts/aliqua-sint-ut-nulla-sunt/
     /posts/aliquip-aliquip-duis-ad-labore copy/
     /posts/aliquip-aliquip-duis-ad-labore/
     /posts/aliquip-consequat-culpa-irure-amet copy/
     /posts/aliquip-consequat-culpa-irure-amet/
     /posts/aliquip-ea-deserunt-consectetur-eu copy/
     /posts/aliquip-ea-deserunt-consectetur-eu/
     /posts/aliquip-nisi-in-et-exercitation copy/
     /posts/aliquip-nisi-in-et-exercitation/
     /posts/aliquip-sint-veniam-nulla-mollit copy/
     /posts/aliquip-sint-veniam-nulla-mollit/
     /posts/amet-ad-culpa-laborum-magna copy/
     /posts/amet-ad-culpa-laborum-magna/
     /posts/amet-ad-ut-eiusmod-magna copy/
     /posts/amet-ad-ut-eiusmod-magna/
     /posts/amet-aliquip-voluptate-sint-ut copy/
     /posts/amet-aliquip-voluptate-sint-ut/
     /posts/amet-nisi-minim-duis-et copy/
     /posts/amet-nisi-minim-duis-et/
     /posts/amet-ut-consectetur-laborum-dolore copy/
     /posts/amet-ut-consectetur-laborum-dolore/
     /posts/anim-labore-labore-reprehenderit-in copy/
     /posts/anim-labore-labore-reprehenderit-in/
     /posts/anim-minim-dolore-eiusmod-fugiat copy/
     /posts/anim-minim-dolore-eiusmod-fugiat/
     /posts/aute-dolore-elit-ex-esse copy/
     /posts/aute-dolore-elit-ex-esse/
     /posts/aute-eiusmod-veniam-fugiat-commodo copy/
     /posts/aute-eiusmod-veniam-fugiat-commodo/
     /posts/cillum-officia-sint-culpa-aute copy/
     /posts/cillum-officia-sint-culpa-aute/
     /posts/cillum-sit-ad-qui-dolor copy/
     /posts/cillum-sit-ad-qui-dolor/
     /posts/commodo-cupidatat-adipisicing-dolor-occaecat copy/
     /posts/commodo-cupidatat-adipisicing-dolor-occaecat/
     /posts/commodo-duis-magna-do-adipisicing copy/
     /posts/commodo-duis-magna-do-adipisicing/
     /posts/consectetur-consectetur-ex-pariatur-ex copy/
     /posts/consectetur-consectetur-ex-pariatur-ex/
     /posts/consectetur-labore-eu-et-dolor copy/
     /posts/consectetur-labore-eu-et-dolor/
     /posts/consectetur-minim-aliqua-quis-exercitation copy/
     /posts/consectetur-minim-aliqua-quis-exercitation/
     /posts/consectetur-non-cupidatat-laboris-eu copy/
     /posts/consectetur-non-cupidatat-laboris-eu/
     /posts/consequat-eu-duis-cupidatat-eiusmod copy/
     /posts/consequat-eu-duis-cupidatat-eiusmod/
     /posts/consequat-proident-dolor-mollit-commodo copy/
     /posts/consequat-proident-dolor-mollit-commodo/
     /posts/consequat-quis-cillum-id-proident copy/
     /posts/consequat-quis-cillum-id-proident/
     /posts/culpa-anim-veniam-eiusmod-veniam copy/
     /posts/culpa-anim-veniam-eiusmod-veniam/
     /posts/culpa-est-qui-eiusmod-esse copy/
     /posts/culpa-est-qui-eiusmod-esse/
     /posts/culpa-id-enim-consectetur-amet copy/
     /posts/culpa-id-enim-consectetur-amet/
     /posts/culpa-magna-minim-in-minim copy/
     /posts/culpa-magna-minim-in-minim/
     /posts/culpa-nostrud-excepteur-minim-dolor copy/
     /posts/culpa-nostrud-excepteur-minim-dolor/
     /posts/culpa-sit-incididunt-ea-minim copy/
     /posts/culpa-sit-incididunt-ea-minim/
     /posts/cupidatat-aute-occaecat-id-veniam copy/
     /posts/cupidatat-aute-occaecat-id-veniam/
     /posts/cupidatat-eu-quis-aliquip-exercitation copy/
     /posts/cupidatat-eu-quis-aliquip-exercitation/
     /posts/cupidatat-velit-et-sunt-laboris copy/
     /posts/cupidatat-velit-et-sunt-laboris/
     /posts/cupidatat-veniam-sunt-ad-nostrud copy/
     /posts/cupidatat-veniam-sunt-ad-nostrud/
     /posts/deserunt-adipisicing-ea-officia-id copy/
     /posts/deserunt-adipisicing-ea-officia-id/
     /posts/deserunt-magna-eu-elit-commodo copy/
     /posts/deserunt-magna-eu-elit-commodo/
     /posts/do-adipisicing-culpa-deserunt-in copy/
     /posts/do-adipisicing-culpa-deserunt-in/
     /404/
    generated page... /404/
    generated page... /posts/ad-deserunt-cillum-consectetur-occaecat/
    generated page... /posts/ad-in-id-ex-sunt copy/
    generated page... /posts/ad-in-laborum-anim-ex copy/
    generated page... /posts/cillum-officia-sint-culpa-aute copy/
    generated page... /posts/aliqua-eiusmod-cupidatat-proident-ex/
    generated page... /posts/adipisicing-dolore-ea-do-magna copy/
    generated page... /posts/adipisicing-dolore-ullamco-in-amet copy/
    generated page... /posts/ad-deserunt-cillum-consectetur-occaecat copy/
    generated page... /posts/aliqua-dolor-ipsum-fugiat-ullamco/
    generated page... /
    generated page... /posts/adipisicing-dolore-ullamco-in-amet/
    generated page... /posts/adipisicing-exercitation-do-nisi-consectetur/

    #
    # Fatal error in , line 0
    # Check failed: result.second.
    #
    #
    #
    #FailureMessage Object: 0x50651b1e8
     1: 0x1041a5874 node::NodePlatform::GetStackTracePrinter()::$_3::__invoke() [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
     2: 0x104f46d10 V8_Fatal(char const*, ...) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
     3: 0x10452b50c v8::internal::GlobalBackingStoreRegistry::Register(std::__1::shared_ptr) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
     4: 0x1042a7c44 v8::ArrayBuffer::GetBackingStore() [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
     5: 0x1041241f0 node::Buffer::Data(v8::Local) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
     6: 0x10414ed20 node::fs::Read(v8::FunctionCallbackInfo const&) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
     7: 0x104301d30 v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
     8: 0x104301828 v8::internal::MaybeHandle v8::internal::(anonymous namespace)::HandleApiCallHelper(v8::internal::Isolate*, v8::internal::Handle, v8::internal::Handle, v8::internal::Handle, v8::internal::Handle, v8::internal::BuiltinArguments) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
     9: 0x1043010b4 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    10: 0x104a41b8c Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    11: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    12: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    13: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    14: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    15: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    16: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    17: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    18: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    19: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    20: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    21: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    22: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    23: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    24: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    25: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    26: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    27: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    28: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    29: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    30: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    31: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    32: 0x1049d34cc Builtins_JSEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    33: 0x1049d3164 Builtins_JSEntry [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    34: 0x1043a4d1c v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    35: 0x1043a43b0 v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle, v8::internal::Handle, int, v8::internal::Handle*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    36: 0x1042b1840 v8::Function::Call(v8::Local, v8::Local, int, v8::Local*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    37: 0x104110908 node::loader::ModuleWrap::SyntheticModuleEvaluationStepsCallback(v8::Local, v8::Local) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    38: 0x10466afb8 v8::internal::SyntheticModule::Evaluate(v8::internal::Isolate*, v8::internal::Handle) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    39: 0x10465595c v8::internal::SourceTextModule::InnerModuleEvaluation(v8::internal::Isolate*, v8::internal::Handle, v8::internal::ZoneForwardList >*, unsigned int*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    40: 0x10465599c v8::internal::SourceTextModule::InnerModuleEvaluation(v8::internal::Isolate*, v8::internal::Handle, v8::internal::ZoneForwardList >*, unsigned int*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    41: 0x10465564c v8::internal::SourceTextModule::Evaluate(v8::internal::Isolate*, v8::internal::Handle) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    42: 0x10465551c v8::internal::SourceTextModule::EvaluateMaybeAsync(v8::internal::Isolate*, v8::internal::Handle) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    43: 0x10429e9a8 v8::Module::Evaluate(v8::Local) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    44: 0x104111edc node::loader::ModuleWrap::Evaluate(v8::FunctionCallbackInfo const&) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    45: 0x104301d30 v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    46: 0x104301828 v8::internal::MaybeHandle v8::internal::(anonymous namespace)::HandleApiCallHelper(v8::internal::Isolate*, v8::internal::Handle, v8::internal::Handle, v8::internal::Handle, v8::internal::Handle, v8::internal::BuiltinArguments) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    47: 0x1043010b4 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    48: 0x104a41b8c Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    49: 0x1049d56d8 Builtins_InterpreterEntryTrampoline [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    50: 0x104a04cd4 Builtins_AsyncFunctionAwaitResolveClosure [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    51: 0x104a89b78 Builtins_PromiseFulfillReactionJob [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    52: 0x1049f70b4 Builtins_RunMicrotasks [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    53: 0x1049d33a4 Builtins_JSRunMicrotasksEntry [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    54: 0x1043a4ce4 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    55: 0x1043a5118 v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    56: 0x1043a5204 v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*, v8::internal::MaybeHandle*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    57: 0x1043c7e40 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    58: 0x1043c86d4 v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    59: 0x104092db4 node::InternalCallbackScope::Close() [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    60: 0x1040927c4 node::InternalCallbackScope::~InternalCallbackScope() [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    61: 0x1041491d0 node::fs::FileHandle::CloseReq::Resolve() [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    62: 0x10415bf00 node::fs::FileHandle::ClosePromise()::$_0::__invoke(uv_fs_s*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    63: 0x1049b3a5c uv__work_done [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    64: 0x1049b71f8 uv__async_io [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    65: 0x1049c8f70 uv__io_poll [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    66: 0x1049b7688 uv_run [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    67: 0x104093ccc node::SpinEventLoop(node::Environment*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    68: 0x1041d8598 node::worker::Worker::Run() [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    69: 0x1041db7c0 node::worker::Worker::StartThread(v8::FunctionCallbackInfo const&)::$_3::__invoke(void*) [/Users/owenbuckley/.nvm/versions/node/v16.14.0/bin/node]
    70: 0x1ba24226c _pthread_start [/usr/lib/system/libsystem_pthread.dylib]
    71: 0x1ba23d08c thread_start [/usr/lib/system/libsystem_pthread.dylib]
    ./bench-greenwood.sh: line 17: 97189 Trace/BPT trap: 5       npx @greenwood/cli build

  

or

    **RangeError [Error]: WebAssembly.instantiate(): Out of memory: wasm memory
    at node:internal/deps/cjs-module-lexer/dist/lexer:1:33593
    at async initCJSParse (node:internal/modules/esm/translators:64:5)
    at async ESMLoader.commonjsStrategy (node:internal/modules/esm/translators:154:18)
    FATAL ERROR: v8::FromJust Maybe value is Nothing.
    1: 0x100ef2260 node::Abort() [/Users/owenbuckley/.nvm/**
 

Here are specs, using the same approach Zach took. He didn't mention Node version, so for mine I used 16.14.0. The metric to track is real.

Each run was repeated 3 times and the lowest/fastest time was selected. This result set was generated on a MacBook Air (M1, 2020), macOS Monterey 12.5 12.3, 16 8 GB memory.

Results

MBP ⚠️

250
real    0m14.058s
user    0m37.615s
sys 0m6.054s
500
real    0m34.519s
user    1m20.436s
sys 0m14.706s
1000
real    1m6.309s
user    2m36.338s
sys 0m29.113s
2000
real    3m4.827s
user    5m56.250s
sys 1m16.211s
4000

🚫

MBA 🚨

25
real    0m1.303s
user    0m3.432s
sys 0m0.437s
50
real    0m1.762s
user    0m5.776s
sys 0m0.796s
100
real    0m2.980s
user    0m10.943s
sys 0m1.712s
250
real    0m5.540s
user    0m24.713s
sys 0m3.614s
500

🚫

1000

🚫

2000

🚫

4000

🚫

Details

So yeah, definitely some work we could / should do here, at least to chip away it for 1.0. Aside from the issues with the M1 chip one area of opportunity for sure seems to be the bundling phase as each benchmark run always seems to hang a little bit on this part of the process

success, done generating all pages...

Anyway, not sure what is realistic but would certainly like to be somewhat "competitive", as for v1.

Thoughts / Next Steps

  1. [x] Confirm I am not running Node via Rosetta (for M1) - https://stackoverflow.com/a/65449002/417806
  2. [x] Could be due to Worker threads, try with prerender: false - https://github.com/ProjectEvergreen/greenwood/pull/983
  3. [x] Bisect lifecycles (e.g. no bundle step) and see if that makes a difference. I know we do a lot of duplicate work in the bundle lifecycle which would be good to refactor out anyway. - #971
  4. [ ] Stamp out unique page templates only once, re-use between page renders
  5. [x] Throttle max pages to build at once? Or just have it be a Promise.all free for all? - https://github.com/ProjectEvergreen/greenwood/pull/983
  6. [x] Out of even more curiosity, I wonder would it would look like to benchmark the 0.25.0 version, to see how the results would have come out if using Puppeteer. 😅
  7. [x] Between prerender and bundle lifecycles, could we reduce "walks" over compilation.graph when iterating over pages. Maybe a Map / Set would be better here? - #971
  8. [ ] Benchmarking in CI (new issue)
  9. [ ] Actually run a profiler
  10. [ ] Maybe this would help (or hurt)? - #823
  11. [ ] Does multiplying / increasing number of workers in the poll hurt or help?
  12. [x] So... got cache busting to work for ESM, wonder if maybe we don't need to use Worker Threads anymore? (At least for production builds - https://github.com/ProjectEvergreen/greenwood/pull/1110)
  13. [ ] After removing Worker threads for https://github.com/ProjectEvergreen/greenwood/pull/1110, wonder if we can do this as well for thread pooling? Or would we still want Workers in some capacity?
thescientist13 commented 1 year ago

So after applying the changes to from #983 , I was able to confirm now that we have no problems at least building 4000 pages without crashing now! 🙌

To help with comparisons, I will be focusing on the MBA benchmark for 4000 pages (user seems like the real world metric)

# 8 threads (os.cpus().length)
real    3m18.260s
user    3m24.175s
sys 0m27.565s
[bench] Build complete 4000 pages
[bench] Framework version

# 80 thread (os.cpus().length * 10)
real    3m21.573s
user    3m29.311s
sys 0m31.081s
[bench] Build complete 4000 pages
[bench] Framework version

# 160 thread (os.cpus().length * 20)
# it crashed hah 😅 

Would probably good to actually do some performance profiling using the NodeJS inspector. Could be good as a follow up PR.