Closed whitfin closed 2 months ago
I tested some approaches, and I think I will do a lot of changes here for v4.x.
Cachex.export/2
, simply delegate to Cachex.stream/3
Cachex.dump/3
with a batched implementation using Cachex.stream/3
Cachex.load/3
with a streaming implementation (consumed by Cachex.import/3
)I think Cachex.import/3
can stay as is, although it should accept a stream of records as well. There's potential to simply remove Cachex.export/2
but keeping it as a grace function is probably fine.
The result of all of this is that we're reusing common foundations in the implementations, and we're much lower on memory. Performance is a little slower, but we gain memory safety and we also get the ability to export based on a query, which is pretty cool.
The current approach requires loading a full cache and exporting an entire cache via Erlang terms. We should see if we can do something to improve this, such as streamed input.
If this isn't possible, it's not the end of the world, just want to take a look and see if we can do better.