search

SAP / cloud-security-services-integration-library

Integration libraries and samples for authenticating users and clients bound to XSUAA authentication and authorization service or Identity authentication service.
Apache License 2.0
151 stars 135 forks source link

Performance: degradation in JWT decoding throughput #339

Closed patrickmhaller closed 4 years ago

patrickmhaller commented 4 years ago

Hi team,

I've been measuring how fast a single CPU core can decode JWTs and compared different libraries: proprietary XS2 Sec Library with some native code embedded, and the current one which leverages com.auth0's java-jwt and nimbus-jose-jwt.

I measured the following results

We're losing about 0,12ms CPU-bound time per request (assuming JWT is only decoded once). So we're up on CPU load a little (within measurable range) and have a tiny hit on request response times.

This is a minor issue, though probably not entirely negligible. You know best if there are any optimization opportunities.

Thanks, Patrick

nenaraab commented 4 years ago

@mwdb can you please have a look at that?

nenaraab commented 4 years ago

@patrickmhaller thanks for the performance measurement!!! That are great news in regard to the self-written jwt token validation.

nenaraab commented 4 years ago

@patrickmhaller Have you already checked the performance of https://github.com/SAP/cloud-security-xsuaa-integration/tree/master/java-security

And could you provide us insights into the measurement of spring-xsuaa?

hassler-d commented 4 years ago

Hi Patrick,

I executed the benchmark myself on my machine using your bp-networking-service-client-java repository. For this I just ran the benchmarkJWTDecodePerSecond method from intellij two times. The system info my machine produced in the test is:

Running on SAP SE's OpenJDK 64-Bit Server VM 11.0.7+10-LTS-sapmachine, home is /Library/Java/JavaVirtualMachines/sapmachine-11.jdk/Contents/Home
   HotSpot 64-Bit Tiered Compilers, mixed mode

To have comparable numbers I also need to run the xs2/sec benchmark on my machine. For this I looked at the git history and tried to find the last working version using xs2/sec. I am not sure I found the correct version, though. All I found was the implementation before our library was integrated into the project. But this uses Spring's JwtHelper and not something that looks like xs2/sec to decode the jwt token. Maybe I had to go further back into the history or maybe it isn't even visible in the git history. Anyhow I have run the benchmark with this pre spring-xsuaa implementation to have a number baseline.

After that I used the current master of the project to benchmark spring-xsuaa. Finally I forked the repo, ported the benchmark to java-security and run the benchmark with this fork. Here are my results:

Implementation Git repo Result 1 Result 2
pre spring-xsuaa (xs2/sec?) bp-networking-service-client-java @ revision 45c2dd306547e95b27dc115f1e0f1eb392388b69 1000000 JWT decodes in 26552962,60000 microseconds: 37660,58105 decodes/second 1000000 JWT decodes in 26339758,46600 microseconds: 37965,42027 decodes/second
spring-xsuaa bp-networking-service-client-java @ revision ccb6f1c6924f8f16931178d0c501de5db10fa155 (current master) 1000000 JWT decodes in 93147100,44600 microseconds: 10735,70723 decodes/second 1000000 JWT decodes in 95257242,77500 microseconds: 10497,88941 decodes/second
java-security My fork based on current master 1000000 JWT decodes in 21274147,42700 microseconds: 47005,40896 decodes/second 1000000 JWT decodes in 21501046,20000 microseconds: 46509,36474 decodes/second

So our native java implementation is pretty fast. Even faster than the old implementation! It is weird to me that spring-xsuaa is so utterly slow. That is why I looked into it more closely. In spring-xsuaa using nimbus the decoding and the validation of the token is not really separated. From the outside it is hard to see but the token is actually validated after decoding. The validation, especially the signature validation, is very expensive. That explains the order of magnitude difference in throughput. From what I have seen in the old implementation, the token was not validated in the benchmark. In my fork using java-security the token is also not validated. So those implementations cannot be compared to spring-xsuaa like that.

Did I have used the same implementation you have used to generate your xs2/sec test results? Or do you have used something different?

Best Regards Daniel

patrickmhaller commented 4 years ago

Hi @hassler-d,

great insights!

We have minor differences in the setup (different hardware, different JVM). But the ratio of runtimes between XS2 and xsuaa-spring is comparable. But I think your analysis is good, and the differences do not matter at all.

"nimbus the decoding and the validation of the token is not really separated" explains a lot. With this insight, it's just interesting to see that the token is actually only decided once per request. I suspect there might have been repetitive decoding in elder versions of CAP and S4SDK. If I find time, I'll do some tracing in this direction. But if xsuaa-spring does not do any of the sort, we can close the issue.

Thanks! Patrick