Fix mongo size issues for Job results - Githubissues

OpenMined / PySyft

Perform data science on data that remains in someone else's server

https://www.openmined.org/

Apache License 2.0

9.43k stars 1.99k forks source link

Fix mongo size issues for Job results #9102

Closed eelcovdw closed 1 month ago

eelcovdw commented 1 month ago

Description

Set MIN_SIZE_BLOB_STORAGE_MB to 1 (from 16) to reduce db performance impact
Do not save items and permissions in the database when making the syncstate snapshot, to avoid mongo size issues + improve performance
always clear result cache on job_stash.set_result, we should never save cache in db unless its a non-blobstorage actionobject

Affected Dependencies

List any dependencies that are required for this change.

How has this been tested?

Describe the tests that you ran to verify your changes.
Provide instructions so we can reproduce.
List any relevant details for your test configuration.

Checklist

[ ] I have followed the Contribution Guidelines and Code of Conduct
[ ] I have commented my code following the OpenMined Styleguide
[ ] I have labeled this PR with the relevant Type labels
[ ] My changes are covered by tests