Optimize memoization - Githubissues

Summary

Memoization of large numbers of steps can take a long time in a single request. This is at its worst when steps reach higher limits and are all synchronous with no parallelism.

This particular loop causes a nasty performance hitch, resulting in functions with large synchronous step counts to take an incredibly long time to memoize before execution starts. With this change, we see a 2x improvement in performance for low step counts, a 4x improvement for high step counts, and a 48x improvement for exceedingly high step counts of 10k.

Our preservation of Promise.race() functionality is what causes the remainder of the latency; we allow the JS event loop to find many steps and process them as a batches to ensure that we can order their resolution appropriately, thereby preserving the output of Promise.race() calls.

Ceasing to support Promise.race() would mean removal of this technique, resulting in a huge performance increase as we no longer need to batch found steps. This can be discussed outside of this PR, but should be noted as the remaining delay after these improvements.

Benchmarks

These are performed with the scrappy benchmarking added in #559.

Before

``` benchmark time (avg) (min … max) p75 p99 p999 ------------------------------------------------------------ ----------------------------- • execution ------------------------------------------------------------ ----------------------------- memoized-steps/0 8'743 ns/iter (6'075 ns … 2'320 µs) 7'683 ns 23'873 ns 145 µs memoized-steps/1 165 µs/iter (134 µs … 1'930 µs) 157 µs 343 µs 1'600 µs memoized-steps/10 937 µs/iter (767 µs … 2'935 µs) 927 µs 2'356 µs 2'935 µs memoized-steps/100 10'330 µs/iter (9'120 µs … 11'984 µs) 11'063 µs 11'984 µs 11'984 µs memoized-steps/500 144 ms/iter (138 ms … 152 ms) 146 ms 152 ms 152 ms memoized-steps/1000 797 ms/iter (781 ms … 813 ms) 811 ms 813 ms 813 ms memoized-steps/2000 5'337 ms/iter (5'260 ms … 5'465 ms) 5'452 ms 5'465 ms 5'465 ms memoized-steps/5000 78'726 ms/iter (75'928 ms … 84'101 ms) 81'857 ms 84'101 ms 84'101 ms memoized-steps/10000 724 s/iter (705 s … 746 s) 738 s 746 s 746 s ```

After

``` benchmark time (avg) (min … max) p75 p99 p999 ------------------------------------------------------------ ----------------------------- • execution ------------------------------------------------------------ ----------------------------- memoized-steps/0 9'925 ns/iter (6'533 ns … 2'539 µs) 8'695 ns 27'181 ns 123 µs memoized-steps/1 178 µs/iter (139 µs … 2'092 µs) 172 µs 364 µs 2'018 µs memoized-steps/10 930 µs/iter (780 µs … 2'993 µs) 906 µs 2'613 µs 2'993 µs memoized-steps/100 9'595 µs/iter (8'443 µs … 11'699 µs) 10'347 µs 11'699 µs 11'699 µs memoized-steps/500 71'044 µs/iter (67'939 µs … 72'297 µs) 71'996 µs 72'297 µs 72'297 µs memoized-steps/1000 198 ms/iter (191 ms … 206 ms) 205 ms 206 ms 206 ms memoized-steps/2000 605 ms/iter (582 ms … 668 ms) 610 ms 668 ms 668 ms memoized-steps/5000 3'452 ms/iter (3'300 ms … 3'621 ms) 3'589 ms 3'621 ms 3'621 ms memoized-steps/10000 15'414 ms/iter (14'917 ms … 15'666 ms) 15'575 ms 15'666 ms 15'666 ms ```

Checklist

[ ] ~Added a docs PR that references this PR~ N/A Perf fix
[ ] ~Added unit/integration tests~ N/A Adding benchmarks in #559
[x] Added changesets if applicable

Benchmarking in #559 used

inngest / inngest-js

Optimize memoization #561

Summary

Benchmarks

Checklist

Related

🦋 Changeset detected