search

JanusGraph / janusgraph

JanusGraph: an open-source, distributed graph database
https://janusgraph.org
Other
5.32k stars 1.17k forks source link

Centralized Redis cache #3018

Open washerath opened 2 years ago

washerath commented 2 years ago

Describe the feature: In a multi node Janusgraph cluster, data modification done from one instance does not sync with others until it reaches the given expiry time (cache.db-cache-time)

As per the documentation[1] it does not recommends to enable database level cache in a distributed setup as cached data does not share amoung instances.

Once we enable the db.cache, what ever the modification done for particular vertex only visible for that JG instance untill the cache expires. So if we have multiple JG instances, the modifications done from one instance does not reflect on other immediately.

If we can have centralized cache which syncs up with all JG instances this can be avoided.

Describe a specific use case for the feature:

We were able to overcome sync issue on multiple JG instance after doing modifications on void invalidate(StaticBuffer key, List entries) as suggested on here [2]

I have replaced existing guava cache implementation to connect with remote Redis db. So that Redis DB will be act as centralized cache which can connects with all other JG instances.

I would like to contribute my implementation to community. Please guide.

NOTE - Although cache sync between JG instances happens on real time, it can expect slight delays due to network latencies and redis performance. So this implemented Redis cache approach will compromise the DB consistancy.

[1] https://docs.janusgraph.org/operations/cache/#cache-expiration-time [2] https://lists.lfaidata.foundation/g/janusgraph-users/topic/88877425

li-boxuan commented 2 years ago

Hi @washerath You may want to check out the contributing guide. Looking forward to seeing your PR!

li-boxuan commented 2 years ago

Hi, @washerath Just checking if you need any help/guidance to contribute.

n5nk commented 11 months ago

@li-boxuan / @washerath are we planning to merge this Redis PR and make it available for all? Is anything stopping here for review ?

I recently started realizing we also need a distributed cache, since we run atlas in distributed mode. And querying Cassandra turns out to be costly for us since we ours is a read-heavy system.