rewrite-bigdata-in-rust / RBIR

A collection of RBIR projects and posts for anyone interested in joining this journey.
194 stars 7 forks source link

Build files concurrently #33

Closed sanders41 closed 1 month ago

sanders41 commented 2 months ago

I did some quick and dirty benchmarking and by building the files concurrently the time it took to run the program went from ~19 seconds to ~2 seconds (roughly a 89% decrease).

The GitHub docs aren't the easiest to follow on API rate limits, but as far as I can tell this will be well under the allowed limits.

This does add code complexity to gain the speed so if you want to prioritize simplicity over speed I understand that, but I wanted to offer it since it made a fairly large difference.