HDFGroup / hermes

Extending the HDF5 library to support intelligent I/O buffering for deep memory and storage hierarchy systems
https://grc.iit.edu/research/projects/hermes
30 stars 17 forks source link

Improve performance for extremely small I/O workloads #680

Open lukemartinlogan opened 4 months ago

lukemartinlogan commented 4 months ago

Hermes performs well at 4KB, but not so well at 300 bytes -- which is the case for certain workload types we are encountering.

  1. Make a lock-free hashmap supporting both inserts and deletes
  2. Improve shared memory allocator performance by leveraging lock-free queues for a small cache
  3. Allow clients to read data directly from the SHM hashmap