feat: switching to write stream and sequential building

YazeedLoonat commented 1 week ago

This PR addresses https://github.com/metrotranscom/doorway/issues/753 and also addresses: https://github.com/bloom-housing/bloom/issues/4337

blocked by https://github.com/bloom-housing/bloom/pull/4322

[x] Addresses only certain aspects of the issue

Description

This switches the way we write the spreadsheet to leveraging a write stream, and also changes from a promise.All() call to handle the promises to instead handle that sequentially

How Can This Be Tested/Reviewed?

go in and export a large lottery, you should notice the time required to build the export has decreased, and if you are tracking the mem usage that should drop significantly from what it was previously

Author Checklist:

[ ] Added QA notes to the issue with applicable URLs
[x] Reviewed in a desktop view
[ ] Reviewed in a mobile view
[ ] Reviewed considering accessibility
[ ] Added tests covering the changes
[ ] Made corresponding changes to the documentation
[ ] Ran yarn generate:client and/or created a migration when required

Review Process:

Read and understand the issue
Ensure the author has added QA notes
Review the code itself from a style point of view
Pull the changes down locally and test that the acceptance criteria is met
Either (1) explicitly ask a clarifying question, (2) request changes, or (3) approve the PR, even if there are very small remaining changes, if you don't need to re-review after the updates

emilyjablonski commented 1 week ago

From the testing instructions go in and export a large lottery - how have you been generating a large lottery locally / tracking the memory usage?

YazeedLoonat commented 1 week ago

Hey @emilyjablonski good questions:

I used a copy of production and used one of the very large listings from there to generate lottery results and then export the lottery

For looking at mem management: you'll notice in this pr I removed a series of console logs that log out a string and then process.memoryUsage() those values are how much memory the application is using

there is a spreadsheet with aa breakdown of mem management across runs and also across different things I tried I can DM it to you if you want to look at that instead of running exporting lotteries a bunch of times

netlify[bot] commented 4 days ago

Deploy Preview for partners-bloom-dev ready!

Name	Link
Latest commit	bc791d19d05f615bc2f0d0c9bfdc8f2e0207e999
Latest deploy log	https://app.netlify.com/sites/partners-bloom-dev/deploys/66ec9b372d0b8400087c32e1
Deploy Preview	https://deploy-preview-4329--partners-bloom-dev.netlify.app
Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

netlify[bot] commented 4 days ago

Deploy Preview for bloom-exygy-dev ready!

Name	Link
Latest commit	bc791d19d05f615bc2f0d0c9bfdc8f2e0207e999
Latest deploy log	https://app.netlify.com/sites/bloom-exygy-dev/deploys/66ec9b37f1493c0008b6f505
Deploy Preview	https://deploy-preview-4329--bloom-exygy-dev.netlify.app
Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

bloom-housing / bloom