irods / irods_capability_automated_ingest

Other
12 stars 15 forks source link

Optimization for event_handler-in-Redis #161

Open alanking opened 2 years ago

alanking commented 2 years ago

Based on the work in #102...

Possible future optimization:

This should reduce network traffic by N files - M num_workers at the cost of N files * target_path_length_in_bytes in Redis.

I think an option should be provided via the CLI to skip checking Redis entirely if the user knows what path the workers are supposed to be checking (i.e. restores the option of using the historical, "manual" deployment).

trel commented 2 years ago

this should prove efficient because network traffic is a lot more expensive than a larger redis database.

but we'll test this assumption before merging.

with N = 1 million files, M = 50 workers, and target path length = 40bytes... ... estimated savings of 999,950 redis hits for the cost of 40MB of additional redis memory