Closed alexashley closed 3 years ago
Lastly, some of this is too specific to
grafeas-elasticsearch
, but if we wanted to extract this out for use in Rode, then I think we could make theIndexManager
implementation more generic and pass in a function that could be used to filter indices.
Yeah definitely something for us to consider as we expand out the responsibility of Rode. For another time though
This is the second part of #58 after splitting it out.
Originally we tried to implement migrations using a lock in order to prevent multiple instances of
grafeas-elasticsearch
from attempting to do migrations all at once. However, as we started to dive into error cases, we realized that would require some manual intervention if the app crashed or was terminated in the middle of a migration.After that, we stumbled on this migration RFC for Kibana, which outlines a lockless migration that can happen in parallel with multiple instances. There's potentially some duplicate work, but the end result should be the same and shouldn't require manual intervention. We deviated some from what's outlined there since it's also concerned with data migrations, but it seems like a good template to follow if we want to add those in the future. We can also certainly go down the lock route as well, as long as we document what needs to be done if the migration is interrupted.
Some assumptions:
grafeas-elasticsearch
start withgrafeas
and have a value ofgrafeas
in the fieldmappings._meta.type
on the indexgrafeas-elasticsearch
indices have a version and it's specified in the index name (i.e.,grafeas-$VERSION)
.Migration steps:
read_only
block as well, but that would prevent us from switching the alias name over).We ran a few manual tests for this, the setup for those is in the
migration-tests
branch. The first tests runs a migration, the next starts multiple instances ofgrafeas-elasticsearch
that will all attempt to migrate, and the final one fails the migration after every step to check that the process is reentrant. Potentially if we were to extract this into its own library, we could flesh those tests out more.Lastly, some of this is too specific to
grafeas-elasticsearch
, but if we wanted to extract this out for use in Rode, then I think we could make theIndexManager
implementation more generic and pass in a function that could be used to filter indices.