Open sbarzowski opened 6 years ago
Perhaps a lot of time is stored resolving library functions. Every time a function value is called, the same values are computed, which do not depend on arguments. If there is std.something
in a tight loop, every time std
needs to get resolved (which by itself may mean a lot of map lookups), then something
needs to be looked up in the object and then a function value needs to be created. Creating a local outside of the function for std.something can be used as a workaround.
It could be helped by keeping things like std.something
as potentialValues inside function values, instead of always evaluating them from ast.
We would need to be careful not to store too much data, if we kept everything that doesn't depend on arguments, the memory usage could get out of hand pretty easily.
If we had resolved function values, we could go further and cache some call processing - the part which doesn't depend on actual argument values - checking if the number of arguments is correct, mapping positional/named arguments, etc. It could be a massive improvement for builtins.
if there is std.something in a tight loop, every time std needs to get resolved
is it same for $
objects?
Yep, $.something
behaves similarly to std.something
.
Note that by "resolved" I mean "looked up in the environment", just like any other variable.
It is a standard optimization to hoist invariant expressions out of functions, loops, etc.
I tried desugaring LiteralString to a special ast that embedded a pre-constructed stringValue object, to avoid having to create duplicate copies on the heap each time the string literal was executed. However this did not improve performance at all, in fact the profile indicated that makeStringValue was never being called, even though the code was full of string literals. I am not sure why.
This one generate unique numbers for variables statically and use that instead of names
is going to give us the biggest bang for buck next I think.
I have a hunch that Go's maps are not as fast as the C++ std::map, which is responsible for some of the relative performance gap. We have to use maps for objects (in general) but there's no reason for environments.
An interesting observation is the effect of disabling the GC, compiling kube-prometheus. Best of 3 runs:
$ time -J vendor -m manifests kube-prometheus.jsonnet
real 0m6.503s
user 0m8.852s
sys 0m0.352s
$ time GOGC=off jsonnet -J vendor -m manifests kube-prometheus.jsonnet
real 0m5.440s
user 0m4.488s
sys 0m0.952s
It's possible to play with SetGCPercent to tune the GC behaviour, jsonnet is a short lived program and very often doesn't need to garbage collect at all.
I've been toying with this sonnet:
local k = import "k.libsonnet";
local deployment = k.apps.v1beta2.deployment;
local container = deployment.mixin.spec.template.spec.containersType;
local containerPort = container.portsType;
local myAppContainer = container.new("hello", "world");
local myAppDeployment = deployment.new("hello", 2, [myAppContainer], podLabels={"app": "myapp"});
k.core.v1.list.new(myAppDeployment)
It references these two files:
These are the results I get with 3 different implementations:
$ time jsonnet-cpp my.jsonnet
{...}
real 0m0.545s
user 0m0.477s
sys 0m0.047s
$ time jsonnet-go my.jsonnet
{...}
real 0m1.703s
user 0m2.082s
sys 0m0.193s
$ time ./sjsonnet.jar my.jsonnet
{...}
real 0m1.833s
user 0m0.376s
sys 0m0.045s
and a second run of sjsonnet.jar
, as it's calling into a background server (to avoid the JVM boot time):
$ time ./sjsonnet.jar my.jsonnet
{...}
real 0m0.232s
user 0m0.327s
sys 0m0.029s
Seems the Scala version managed to get some interesting boost here.
My stack is in Go, so I'd love so much to have a speedy version here :)
As a side note, I'm interested in go-jsonnet
as a real-time JSON stream reshaper.. disabling the garbage collecting in this situation isn't really practical.
It looks like in this case it's parsing performance that really matters in this case. I don't think garbage collection is a problem at all here.
EDIT:
I ran a quick experiment and just processing k8s.libsonnet
takes pretty much the same amount of time as the provided example (the difference is well within random fluctuations).
I'd like to contribute to this effort. I had a couple questions though to help me get oriented.
That's awesome!
1) Actually I think it's best to keep the benchmarks completely implementation-agnostic, we shouldn't care if it's Go, Jsonnet or C++ - we want to compare the implementations. You can just create a Jsonnet file on which the performance improvement is visible here https://github.com/google/jsonnet/tree/master/benchmarks. Ideally we would have more benchmarks, including some realistic ones and a script which runs them all and prepares a nice looking report, allowing for easy comparison of performance across versions. But we don't have that yet, and frankly it's not critical, because there is still a lot of low hanging fruit performance-wise and the improvements tend to be obvious. So, don't worry too much about it, adding an example for which the improvement is clearly visible is enough for stuff like adding a builtin implementation of a stdlib function.
2) All the builtins in the builtin table will override the ones defined in std.jsonnet
. It's enough to define a new function implementing a builtin and add it to the table there. For reference the code which builds the std object and performs the overriding is here: https://github.com/google/go-jsonnet/blob/v0.14.0/interpreter.go#L1098.
If you have any questions or need help with anything, please let me know!
sjsonnet
has some very good benchmarks because it caches values once they are evaluated as outlined here: https://github.com/databricks/sjsonnet#performance
Additionally, they moved all the native functions into Scala, which I think would also be very worthwhile. I started doing that to some of the more complex functions already.
jrsonnet brought an evaluation down from 145s in go-jsonnet to a mere 0.35s (https://github.com/CertainLach/jrsonnet/issues/156). The bottleneck from a pprof (svg) was probably flattenArrays and garbage collection. The author rewrote std.prune
to drop from the initial 12s runtime, so that's likely a bottleneck here too. It is such a stark difference in runtimes for the same output.
@ben-manes ya, it's clear that functions written in jsonnet are going to going to be the cause of many performance bottlenecks.
First we need to have some benchmarking in place, and some reasonable "corpus" of jsonnet to test on. We can start by using current test suite, but dedicated benchmarks make more sense.
Once we have that, we can try to find bottlenecks and potentially optimize. Instead of having them scattered throughout the code, I'll dump some of these ideas here:
Before we implement any of these we should have a benchmark that proves that it actually helps.