some speed improvements to sync

OpenMined / PySyft

Perform data science on data that remains in someone else's server

https://www.openmined.org/

Apache License 2.0

9.43k stars 1.99k forks source link

Closed abyesilyurt closed 1 month ago

abyesilyurt commented 1 month ago

6x speed up by implementing the following changes:

Keep a copy of the last sync state in memory rather than reading from db.
Use partition keys to query jobs by result id instead of filtering in python.
Do not wait for saving the sync state and do an async write. This one is still hacky, need to handle it better.