In the future, there will be a redis stream (or other type) that will be used to hold stream names to use for reading current data about renku entities.
Search must only use these streams to read and ingest data. Once the value changes, it means there is a complete new dataset and the index is outdated. Search must then re-index everything from the beginning of these streams.
When starting up, get the "latest message" from this "control stream" and get the stream names for project, user and group
connect to them and handle messages
connect to the "control stream" and handle new values by restarting the provisioning with the new streams
In the future, there will be a redis stream (or other type) that will be used to hold stream names to use for reading current data about renku entities.
Search must only use these streams to read and ingest data. Once the value changes, it means there is a complete new dataset and the index is outdated. Search must then re-index everything from the beginning of these streams.