search

apache / druid

Apache Druid: a high performance real-time analytics database.
https://druid.apache.org/
Apache License 2.0
13.46k stars 3.7k forks source link

applyAll and unapplyAll #2693

Closed erikdubbelboer closed 6 years ago

erikdubbelboer commented 8 years ago

2077 introduced the applyAll and unapplyAll lookup functions. Is there any plan to actually start using these functions?

I think if lookups could be bulked it would allow for more flexible lookup modules to be developed.

For example we have a lookup table with over 6 million entries and growing. Keeping this table cached in memory uses a lot of resources. We would like to develop a lookup module that does an SQL query for each lookup. Doing this in bulk would be much faster than doing a separate query for each row.

drcrallen commented 8 years ago

Yes, QTL is undergoing a lot of improvements in the near future, and proper handling of applyAll and unapplyAll is among them.

In the mean time, have you tried offHeap caching for the existing lookups? that should help reduce heap memory pressure.

erikdubbelboer commented 8 years ago

Any time frame for these changes?

offHeap does seem to reduce heap memory pressure but still maps everything to memory so in theory it still uses a lot of memory.

b-slim commented 8 years ago

@erikdubbelboer i am working on this https://github.com/druid-io/druid/issues/3030 keep tuned feel free to interact.

erikdubbelboer commented 6 years ago

Since #3030 is closed I'm closing this as well for now.