IO concurrency can be configured via IOStrategy for each block & file. This IOStrategy is used when opening, decompressing and caching a file or blocks within the file.
Concurrent - Multiple threads are allowed. One thread does not care about the state of the other so this gives high concurrency but could lead to multiple open channels to the same file (eventually only one will remain open).
Asynchronous - If a file is being opened, decompressed or cached by a thread, other threads will asynchronously wait for that thread to finish before continuing from that same checkpoint.
Synchronous - Similar to Asynchronous but this blocks.
Issue
Asynchronous requires the need for Future in core which requires the need to Bag[_] which requires memory allocations that can be removed if IOStrategy.AsyncIO is removed.
If IOStrategy.AsyncIO can be removed from core then the need for Bag[_] would only be at API level which is optional anyway.
Task
Benchmark the cost of IOStrategy.AsyncIO and at what point it becomes useful. Check if IOStrategy.AsyncIO can be removed completely.
Overview
IO concurrency can be configured via IOStrategy for each block & file. This
IOStrategyis used when opening, decompressing and caching a file or blocks within the file.Concurrent- Multiple threads are allowed. One thread does not care about the state of the other so this gives high concurrency but could lead to multiple open channels to the same file (eventually only one will remain open).Asynchronous- If a file is being opened, decompressed or cached by a thread, other threads will asynchronously wait for that thread to finish before continuing from that same checkpoint.Synchronous- Similar toAsynchronousbut this blocks.Issue
Asynchronousrequires the need forFuturein core which requires the need toBag[_]which requires memory allocations that can be removed ifIOStrategy.AsyncIOis removed.If
IOStrategy.AsyncIOcan be removed from core then the need forBag[_]would only be at API level which is optional anyway.Task
Benchmark the cost of
IOStrategy.AsyncIOand at what point it becomes useful. Check ifIOStrategy.AsyncIOcan be removed completely.