Slow mongodb query in the retention module

[jobulcke]

Insertion of certain member properties in the retention module is very slow and expensive. The need of this query needs to be (re)evaluated, since some/all of the properties are now also stored in the ingest_ldesmember collection as well. If it appears to still be needed, then this query must be optimized

{"insert":"retention_member_properties","ordered":true,"$db":"vsds?retryWrites=false","documents":[{"_id":"?","collectionName":"?","views":["?"],"versionOf":"?","timestamp":{"$date":{"$numberLong":"?"}},"_class":"?"}]}

[jobulcke]

Some remarks about this issue:

• The collection retention_member_properties only contains the additional field views in comparance to ingest_ldesmember (since version 2.12.0). That field views is sort of a mapping between the member and to which views that member is part of, which would usally be in a "pivot table" in SQL databases. However, in the fetch module, there is already some kind of "pivot table" between members and fragments, and as a fragment can only have one view, the "pivot table" works also between members and views.

• The main reason why the collection retention_member_properties exists, is namely to fetch the so called "expired" members, but this already happens via an aggragation, which opens the door (especially since the last changes of version 2.12.0) to use a "lookup operation" in that aggragation, which can be compared to a join clause in SQL databases

• I see "ordered":true is present, this makes the insertion sequential instead of parallel. In sequential insertion, the insertion process stops when an error occurs, but notice the first insertions will not be undone. On the other hand, parallel insertion does not take any other result of the bulk insertion into account

Possible solutions:

Solutions that reflects a more SQL approach (which drops the retention_member_properties collection)

• (purely in Java) provide a Repository that only does those lookup aggregations, but is has no collection on its own
• Create a new collection (e.g. member_view_allocation or something like that), but also provide aggragation functionality (with the lookup actions) for the repository reflecting this collection. The advantage of this approach is that the responsibilties of each repository keeps limited to its own Maven module

Quick and easy solutions that does not change anything about the data structure

• Make the method that does the insertion into the retention_member_properties asynchronous
• Make the bulk insert not ordered (which must happen if nothing changes to the data structure in my opinion)

[jobulcke]

Resolved in https://github.com/Informatievlaanderen/VSDS-LDESServer4J/pull/1263