API <-> Fetcher architecture

[t4ccer]

I'm not a huge fan of current way of how API communicates with block fetcher. Maybe model when fetcher thread is always running and API somewhat sends commands to that thread to change it's behavior (opposing to current model of killing thread and restarting it with new parameters, which is inconvenient and error prone) would be easier to maintain in the long run. I think it would get even more messier when we'd integrate proposed tx caching.

I'm not entirely convinced though, so it's rather call for ideas/opinions. https://mlabs-corp.slack.com/archives/C01UAHD91PH/p1653055202402859

[t4ccer]

Proposed architecture

Split ODC into 3 components:

Block fetcher

Block fetcher will be responsible for:

• Maintaining connection to ogmios (reacting when WS to ogmios dies)
• Requesting blocks from ogmios
• Pushing parsed blocks to bounded queue

Block processor

Responsibilities:

• Reading blocks from queue - consider using TBQueue
• Extracting and filtering datums/txs from blocks
• Pushing extracted data into database

API server

Responsibilities

• Reading data from database
• Changing filters
• Changing starting block

This architecture won't require killing any threads on filters/block changes. Change in filters will be simply change in shared config between API and Block processor.

[micahkendall]

Proposed architecture

I like this for several reasons:

• current is monolithic and becoming difficult to work with

  Block fetcher ...

• can easilier remove ogmios as a dependency if it is useful to query cardano-node directly.
• privileged api which is disliked is made redundant

  Block processor ...

• separates filtering from fetching

  API server ...

• +scalability, API server can be used within a server-less architecture as a cloud function.