To accomplish 1 we had to thread the database transaction deep
into the dig package. I'm not totally pleased with this pattern
since it will issue N queries for each input that it needs to check.
Ideally we can batch these checks into a single query. We also have
to thread a mutex since we need to hold the lock before we query
and after we scan all the results. Nevertheless, this should work
and not be terrible. But we will make it better in the future.
For 2 we simply check the integration config to see if any filters
depend on other integrations. If they do, we add a list of integration
names as dependencies to the integration. Then, when we are determining
which blocks a task should process, in addition to using the ethereum
source's latest block, we also use the dependency blocks. This
ensures that the dependency has processed a block before the dependent
task.
Future followup items:
Add support for indexes (we already have unique indexes)
This feature commit includes 2 major things:
To accomplish 1 we had to thread the database transaction deep into the dig package. I'm not totally pleased with this pattern since it will issue N queries for each input that it needs to check. Ideally we can batch these checks into a single query. We also have to thread a mutex since we need to hold the lock before we query and after we scan all the results. Nevertheless, this should work and not be terrible. But we will make it better in the future.
For 2 we simply check the integration config to see if any filters depend on other integrations. If they do, we add a list of integration names as dependencies to the integration. Then, when we are determining which blocks a task should process, in addition to using the ethereum source's latest block, we also use the dependency blocks. This ensures that the dependency has processed a block before the dependent task.
Future followup items: