refactor(indexer): event filtering and related substate storing

[mrnaveira]

Description

• New configuration in the indexer for event filtering
• Event filtering logic in the event_scanner module
• Removal of all the code related to substate address monitoring
• If an event is filtered to be stored in the database, the related substate (if any) will also be stored in the database automatically. The related substate is extracted from the corresponding transaction result.
• Refactor of the get_non_fungibles functionality to fetch all non fungibles of a collection from the network on the fly
• Refactored the event_manager and substate_manager to simplify and properly split them, as a lot of functionality was duplicated and overlapped between the two modules.
• Updated TypeScript bindings

Motivation and Context

Up until now, the behaviour of the indexer regarding events and substates was:

• Storing in the database all events in the network, with no possibility of filtering.
• Allowing to store substates in the database via the user manually adding addresses to a watchlist using JRPC methods.

This PR refactors the indexer behaviour to be more useful:

• Allowing the user to filter which events in the network we want to store in the database.
• Automatically storing the related substate of each event in the database. Thus removing all the address watchlist functionality from the indexer, as we will be using the new event filters to decide what to store.

Regarding the event filtering:

• The list of event filters is static, it can be edited in the indexer config file. Any change in event filtering requires a restart of the indexer.
• It allows to specify a list of event filters. If a new event matches at least one filter, it will be stored in the database.
• Each filter allows to specify an AND condition of values for topic, substate_id, template_address and entity_id. All filter fields are optional. An event will match a filter if all of its fields match the values specified in the filter.
• The entity_id one can be used when we want to store everything that happens in a particular EntityId no matter the type of substate (component, vault or resource).
• The default behaviour is an empty filter which matches all events, making it store all events that happen in the network.

Regarding the substate storing:

• We can fetch the substate data from the transaction result (up_substates field) corresponding to the event we are storing. This is a handy optimization because it avoids a query to the network to fetch the related substate.
• Because we are constantly fetching new events and extracting the related substate from the transaction results, it means we don't need to keep monitoring for the latest versions of each substate.
• In the database we only store the latest version of each substate.

This PR removes functionality regarding substate monitoring and address watchlist, as now the substate storing is determined by the events.

This PR keeps existing functionality regarding:

• Arbitrary substate fetching via the JRPC get_substate method. Even if the substate is not stored in the database, the indexer will try to fetch it from the network.
• Listing all NFTs from a collection via the get_non_fungibles method. This functionality requires to fetch the non fungibles from the network on the fly, as currently we don't emit the proper events regarding NFT minting. Even if we do in a future, we would need to account for NFT burning (thus deleting the substate from the database in that case)

How Has This Been Tested?

Manually by:

• Spawning a local network using tari_swarm.
• Using different event_filters configurations in the indexer.
• Doing transactions (e.g. transfers) and inspecting which events and substates where stored in the indexer database.

What process can a PR reviewer use to test or verify this change?

See previous section.

Breaking Changes

• [x] None
• [ ] Requires data directory to be deleted
• [ ] Other - Please specify

[github-actions[bot]]

Test Results (CI)

535 tests   - 6   535 :white_check_mark:  - 6   1h 39m 38s :stopwatch: - 7m 43s  63 suites ±0     0 :zzz: ±0    2 files   ±0     0 :x: ±0 

Results for commit 0031ed88. ± Comparison against base commit ad490ae3.

This pull request removes 9 and adds 3 tests. Note that renamed tests count towards both.

``` Scenario: Indexer GraphQL filtering and pagination of events: tests/features/indexer.feature:157:3 Scenario: Indexer GraphQL requests events over network substate indexing: tests/features/indexer.feature:118:3 Scenario: Indexer GraphQL requests work: tests/features/indexer.feature:107:3 tari_indexer_client ‑ types::export_bindings_addaddressrequest tari_indexer_client ‑ types::export_bindings_addaddressresponse tari_indexer_client ‑ types::export_bindings_clearaddressesresponse tari_indexer_client ‑ types::export_bindings_deleteaddressrequest tari_indexer_client ‑ types::export_bindings_deleteaddressresponse tari_indexer_client ‑ types::export_bindings_getaddressesresponse ``` ``` Scenario: Indexer GraphQL filtering and pagination of events: tests/features/indexer.feature:147:3 Scenario: Indexer GraphQL requests events over network substate indexing: tests/features/indexer.feature:108:3 Scenario: Indexer GraphQL requests work: tests/features/indexer.feature:97:3 ```

:recycle: This comment has been updated with latest results.