search

anthonycr / Mockingbird

A minimalist faking framework exclusively for verifying interactions
MIT License
3 stars 2 forks source link

Benchmarks code gen vs reflection #2

Open WhosNickDoglio opened 5 months ago

WhosNickDoglio commented 5 months ago

I'd be curious about what this affect build times compared to using a reflection based mocking library like mockk vs having the code generated fakes like this

anthonycr commented 5 months ago

I would like to figure out a good way to actually benchmark this, as in this repo it's basically

:sample:kspTestKotlin + :sample:compileTestKotlin + :sample:test (unscientific average of 3 runs of this is ~2469ms)

vs

:sample:compileTestKotlin + :sample:test (unscientific average 3 runs of this is ~2930ms)

anthonycr commented 5 months ago

In theory the savings could be better if :sample:kspTestKotlin and :sample:compileTestKotlin are both cached but the tests still need to run, as the :sample:test run is at least 3x as long (on larger tests i have seen it be more like 5x)

WhosNickDoglio commented 5 months ago

Gradle profiler would be good for this. We could profile the test task for projects that use mockingbird vs mockk.

It might also be helpful to write a script to generate a large project (100+ modules) to do a more thorough benchmark. The Tangle docs have that which is nice.

anthonycr commented 5 months ago

I ran some preliminary basic benchmarks of clean :sample:test while disabling the tests that expect exceptions and adding in dagger code gen to normalize the difference in compile speed of not having to run ksp if mockingbird isn't enabled. The performance is basically the same. It would be better to do profiling on a larger project like you mentioned though.

mockingbird-benchmark.csv mockk-benchmark.csv