drop authorizable cache

Netcentric / accesscontroltool

Rights and roles management for AEM made easy

Eclipse Public License 1.0

147 stars 92 forks source link

drop authorizable cache #647

Closed kwin closed 1 year ago

kwin commented 1 year ago

takes too much memory looking up authorizable by ID is really fast also cache empty memberships

kwin commented 1 year ago

This needs further investigation as it seems at least sometimes lookup of authorizables by IDs is relying on Query internally. @otarsko Please provide further insights here.

otarsko commented 1 year ago

Did local test on dummy data and got next results: With ~20k groups in AEM and ~3k groups affected by configuration got next results:

Authorizables installation:
- with cache - Finished installation of authorizables without errors in 24.9sec
- without cache - Finished installation of authorizables without errors in 6.7min
Complete installation:
- with cache - Successfully applied AC Tool configuration in 10.6min
- without cache - Successfully applied AC Tool configuration in 16.3min
biz.netcentric.cq.tools.actool.impl.AcInstallationServiceImpl.installAuthorizables() run time:
- with cache - 3.9% of execution time
- without cache - 41.3% of execution time

Top image - with cache, bottom - without (changes from this PR + #649

Looks like Query is used to extract the Authorizable, which leads to the slowness:

otarsko commented 1 year ago

2 ideas (not very original though) I have:

Provide possibility to enable/disable cache, notifying user that second is decreasing consumption of the memory, but will lead to the increased execution time.
Use cache based on SoftReferences, to let GC collect them, when this is really needed.

@kwin wdyt?

kwin commented 1 year ago

The query being used for authorizable ids is the following: https://github.com/apache/jackrabbit-oak/blob/5a1a902e7a89fc44cb9e2f59b0c6939efa9c16e4/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/identifier/IdentifierManager.java#L342-L366. Usually that should be fast, but it is obviously not neglectable. The most important metric is how big the memory impact of removing the authorizable cache really is. @otarsko Do you have metrics for that as well?

If that is considerable we could just cache the path per authorizable id, as that lookup does not require a search.

otarsko commented 1 year ago

@kwin, in the cloud we had nearly 4 GB in the cache with 3.0.4:

However locally, on the bigger amount of authorizables to process (17314 authorizables), I didn't manage to get even close to that amount with cache in place: