poor performance with Jarl execution- compared to response consumed over OPA as REST api

[rai-prashanna]

RED line is used for JARL. GREEN line is used for OPA REST.

Tested on OPA Version: 0.53.0

It seems that Jarl was designed only to fulfill the correctness of plan.json but not with performance. In my experiment, I have one class that interacts with OPA as a REST API and another class that uses Jarl to make decisions. With the executions, it generates metrics into Prometheus Server. Grafana is used to plot graphs.

My policy can be found in this link.
https://play.openpolicyagent.org/p/W7dh9TIqMi https://play.openpolicyagent.org/p/LYgbgZj0Qg

My link to the repo of testing https://github.com/rai-prashanna/performanceTesting

[anderseknert]

Hi @rai-prashanna 👋 And thanks for the examples!

You're right that Jarl is "optimized" for correctness only at this point. Looking at your policy code however, I don't see anything apparent that would be slower in Jarl except for perhaps the regex.match call which uses the https://github.com/google/re2j library. Another case where native OPA "always" will be faster is when rule indexing kicks in.

We'd need to take a closer look at your example. It should be noted though — Jarl is primarily meant to be embedded in JVM applications. If you compare both running as a server, the overhead of the server and whatever implementation is used for that will need to be taken into account as well.

[rai-prashanna]

Thanks for your response. In my experiment, I embedded Jarl into Java applications and created a Plan object instance during the initialization of the Java application. So metrics are generated only during input evaluation by Jarl and OPA rest.

[rai-prashanna]

Also, when experimenting with wasm binary, performance of JARL library is very poor. Both plan.json and wasm files are located locally Despite OPA is run as REST API server in separate container with overhead of sending and processing request/response.

The green line is used for OPA rest, The violent line is used for WASM, The purple line is used fro JARL.

[anderseknert]

Something doesn't seem right if Wasm evaluation is slower than a HTTP call 🤨 We'll take a look at this when time permits, but remember that this is a side project of ours.

[rai-prashanna]

@anderseknert Does Jarl have support features like partial evaluation or rule indexing (Trie data structure)?

[anderseknert]

No. I don't think the IR supports partial evaluation, but could be I'm wrong on that. Indexing could be supported, but isn't currently (same as Wasm). Any policy that make heavy use of indexing is likely going to be faster using OPA.