jqmp
commented
4 years ago Sure, that's a great idea. I'll do that this week. Let me run a short version by you to make sure it actually answers your question:
- It's probably not safe to run the same flow simultaneously from multiple threads.
- It's probably safe to run the same flow simultaneously from multiple processes. It's not really the intended usage but we've tried to design the cache in a way that makes this safe. However, this may lead to duplicated work, such as multiple processes starting the same expensive computation and then each caching their own instance of the output.
- It's probably safe to run the same flow simultaneously on multiple hosts. I know people at Square are doing this now (for distributed hyperparameter tuning) and they haven't reported any concurrency bugs, although they do get annoyed by the duplicate-work issue above.
I'd love to hear more about the use cases you have in mind. Ultimately we'd like to have Bionic handle this for you inside a single flow -- and in the next 48 hours we should be releasing 0.8.0, which will be able to compute entities in parallel (separate processes on one host).
Could you please document in Warnings and Pitfalls what the user should be aware of, if they have mutiple processes or threads open, which are using the same flow?