Open alanking opened 2 years ago
this should prove efficient because network traffic is a lot more expensive than a larger redis database.
but we'll test this assumption before merging.
with N = 1 million files, M = 50 workers, and target path length = 40bytes... ... estimated savings of 999,950 redis hits for the cost of 40MB of additional redis memory
Based on the work in #102...
Possible future optimization:
This should reduce network traffic by N files - M num_workers at the cost of N files * target_path_length_in_bytes in Redis.
I think an option should be provided via the CLI to skip checking Redis entirely if the user knows what path the workers are supposed to be checking (i.e. restores the option of using the historical, "manual" deployment).