search

cardiacsociety / web-services

MappCPD API and other services
0 stars 1 forks source link

Algr atomic rebuild causing too many operations #30

Closed mikedonnici closed 6 years ago

mikedonnici commented 6 years ago

The daily complete rebuild of the indexes is done in batches of 1000, however, Algolia counts each update as an operation. Given that the resource operation has over 20,000 records in it a daily update consumes 40,000 operations from the 1,000,000 allowed. Over 30 days this will exceed the limit.

Previously, back days was used to select records for update, but this did not include records that had been deleted, soft-deleted or otherwise set with values that should preclude them from inclusion in the index. For example, directory consent set to false for a member.

This is why the decision was made to completely rebuild the indexes each day - then it is just a one-way sync.

However, this issue with operation limits means a new approach is required. At least for the resources index as others are relatively small.

For now maybe could hardcode an atomic update for the resources only on a particular day.

mikedonnici commented 6 years ago

Reduced the update operations from >30,000 to less than 8,000 for a weekday. The number will be higher on Sunday and on the first of the month due to scheduled full/atomic rebuilds. Overall a good result for now and will monitor.