[11.0] Performance imporvement on muk_dms_attachment force storage

katyukha commented 5 years ago

Same as muk_base#19 but related to muk_dms_attachment

This PR contains refactors force_storage method to be able to handle a lot of attachments. May be this is not ideal solution, but it helped to migrate 8000+ attachments from lobject to filestore.

The reason for these modifications is that migration operation takes too long time for large databases. Thus it is better to migrate attachments by small chunks and processing each chunks in separate transaction. Without this approach, when migration takes more than few minutes, there is possible transaction rollbacks or deadlocs due to parallel modifications, or in most frequent case timeouts. Especialy this may be important for odoo instances running on Odoo.SH or other hostings where there is no control on timeouts.

This PR do fillowing:

cleanup store_lobject shen location is not lobject. this is required to distinguish attachments that is not migrated yet. without this fix each time you press Force Storage button, odoo iterates over all attachments (including migrated attachments)
Automatically detect attachments that have to be migrated. (To not migrate already migrated attachments)
Migrate attachments by chunks of 100 elements. Each chunk processed in its own transaction. This allows to continue migration in case of transaction rollback or timeout errors.

Note, that this PR needs precise review and tests.

keshrath commented 5 years ago

Can you estimate how big the performance gain is? Or did you do a benchmark test? What was the result?

katyukha commented 5 years ago

No benchmarks. In my case main problem was timeouts and concurrent writes. And when migration was stopped because of timeout you have to migrate all attachments again. The approach with chunks allows you to continue migration (doing migration only for attachments that are not migrated yet).

keshrath commented 5 years ago

Since we use our own development platform and Github only for code publishing, issue tracking and pull requests, I can't accept this pull request directly as it would bypass our CI workflow.

The changes would be incorporated into these commits:

https://github.com/muk-it/muk_dms/commit/64eca56f9736da3415110a39791f691fba26774c

muk-it / muk_dms

[11.0] Performance imporvement on muk_dms_attachment force storage #104