DestinyItemManager / DIM

Destiny Item Manager
https://destinyitemmanager.com
MIT License
2.07k stars 643 forks source link

Loadout Optimizer integration tests and benchmarks #9993

Open robojumper opened 1 year ago

robojumper commented 1 year ago

Some utility functions for LO have unit tests (mod assignment, auto mods) but the main process loop doesn't. It'd be good to have tests for that to assert qualities about the output - right now we could break the worker by not adding any sets in the loop and no tests would fail.

It'd also be good to have benchmarks. Right now I'm optimizing LO by judging the reported process time, which does include message passing to/from the worker and it's not particularly reliable/conclusive. Is there a JS testing framework that does proper warmup runs, statistical run analysis, and ideally also allows the benchmarks to be compiled to a single JS file to be profiled with more V8 flags, e.g. to then analyze with the deopt explorer?

robojumper commented 1 year ago

I've played around a bit with compiling to a JS file that can be ran offline in node with V8 flags that gets as close as possible (in terms of babel/terser options) to what we run in the browser and come to the conclusion that it's painful to configure (I couldn't figure out how to turn off whitespace compression) and it's not particularly useful to get JIT deopt information - JITs are just too sensitive to changes in the environment and even just calling the process in a loop ends up with object shapes and "code dependency changes" we don't see in beta/prod.