search

seedwing-io / seedwing-policy

A functional type system for policy inspection, audit and enforcement.
https://www.seedwing.io
Apache License 2.0
12 stars 15 forks source link

Add package level metadata #161

Closed ctron closed 1 year ago

ctron commented 1 year ago

did you notice any performance change?

I am having a hard time getting stable results out of cargo bench. My expectation would be that world building is slower, as more work is being done there. But then, serving package level metadata should be much faster, because the work is only to serialize the current state.

Then again, switching from this branch to main, first gives me a decrease of performance (PR -> main):

smoke build             time:   [100.13 µs 101.48 µs 103.31 µs]
                        change: [+25.204% +32.290% +39.906%] (p = 0.00 < 0.05)
                        Performance has regressed.

smoke eval              time:   [1.3620 µs 1.3637 µs 1.3658 µs]
                        change: [+1.2047% +1.3135% +1.4268%] (p = 0.00 < 0.05)
                        Performance has regressed.

Running it again (main -> main):

smoke build             time:   [97.984 µs 99.569 µs 101.71 µs]
                        change: [-14.604% -8.7069% -2.6507%] (p = 0.01 < 0.05)
                        Performance has improved.

smoke eval              time:   [1.4268 µs 1.4278 µs 1.4288 µs]
                        change: [+4.5482% +4.6723% +4.8016%] (p = 0.00 < 0.05)
                        Performance has regressed.

Honestly, I have no idea what that tells me. Maybe I am doing it wrong.

lulf commented 1 year ago

I think that qualifies as 'not significantly changed' :) The cargo bench benchmarks have some variance, I'm not sure yet the reason for it (I tried to dig into that a bit with #20, but left that for another time)

lulf commented 1 year ago

I'll give it a spin on the data in https://github.com/seedwing-io/seedwing-benchmarks