apibara / dna

Apibara is the fastest platform to build production-grade indexers that connect onchain data to web2 services.
https://www.apibara.com/
Apache License 2.0
168 stars 32 forks source link

Write to multiple Mongo collections from a single indexer #278

Closed fracek closed 7 months ago

fracek commented 8 months ago

Is your feature request related to a problem? Please describe.

At the moment, one indexer can insert data only into one collection. This is mostly fine but:

Describe the solution you'd like

We should introduce a "multi collection" mode for mongo indexers.

Configuration wise, we change the mongo options to allow a single collection (like now) or multiple collections. Like all options, we want this to be configurable from inside the script, but also cli and environment variables. The easiest way is to have two keys: collectionName and collectionNames.

pub struct SinkMongoOptions {
  /// ...
  pub collection: CollectionOptions;
}

#[derive(Args)]
pub struct CollectionOptions {
    #[arg(long, env = "MONGO_COLLECTION_NAME", conflicts_with = "collection_names")]
    pub collection_name: Option<String>,
    #[arg(long, env = "MONGO_COLLECTION_NAMES")]
    pub collection_names: Option<Vec<String>>,
}

When users opt into multi-collection mode, the value returned by the transform function is expected to be different.

On chain reorganizations, data from all collections should be invalidated.

Additional context

This feature has been requested by a user and should be prioritized.