ElliottKasoar
commented
4 months ago Resolved by https://github.com/ElliottKasoar/abcd/commit/9b4ea426b0c639ac1b8d02ec4fbbaa470f13cbf1 for fields where aggregation/fields values can be returned (does not include text by default)
The property function, which returns a list of all values of a property in documents stored in OpenSearch is much slower than the equivalent MongoDB function, and the similar count_property function, which effectively returns a counter of the same list.
The issue appears to be affected by the size of all the data in each documents (see also: here), regardless of the specific property being requested, as reducing the size of the arrays stored for positions, forces etc. speeds up the result, even when not requesting these fields.
The rate limiting step appears to be in reading the data, rather than the query itself, as can be seen through equivalent use of
count_property, or if non-field values are returned for the same query.