search

duality-labs / hapi-indexer

A Node.js based indexer for the Duality Cosmos chain
1 stars 0 forks source link

feat: add long-polling block-range updates for tick liquidity #23

Closed dib542 closed 11 months ago

dib542 commented 11 months ago

This PR adds support for requesting liquidity at a target block height range:

the endpoints:

support query parameters:

and return new body attributes:

Using no block_range query parameters will cause a default range from 0-currentHeight, effectively the same request and response as previous exiting route /liquidity/token/{tokenA}/{tokenB} created in https://github.com/duality-labs/hapi-indexer/pull/21

Optional long polling functionality with block_range.from_height

By making a request with block_range.from_height that would return an empty update (no data found since this height), the indexer will wait until data becomes available, and send that data. At that point the front end client may recursively take that response's block_range.to_height value and use it in a subsequent request as the block_range.from_height param, effectively waiting for the next block of data after every request response, i.e. long-polling.

Consistent height pagination with block_range.to_height

A concern with pagination of any data is: "what happens if the underlying data has changed in between page requests?". This can be dealt with in many ways, one being database specific solutions that can lookup database cursors using nextKey params, but if using just offset and limit as is being used currently here, the data may change between page requests.

This PR caches all pages of data for new requests against the specific height so that the same user may come back and receive the pages of data that continues their same request at the same block height. There is a specific issue that may be encountered due to this "just caching" method when run in production with multiple indexers and the user unintentionally queries different indexers between pages (due to a load balancer or other architecture reason) which is detailed in:

Fetching both tick liquidity sides of of token pair at once using /liquidity/pair/{tokenA}/{tokenB}

A new route /liquidity/pair/{tokenA}/{tokenB} exposes the ability to requests both tick sides at once. This provides a much stronger guarantee to front end clients that the data being presented for both side of liquidity for a pair is the data at the same point in time (i.e. the same block height on both sides).

Database Changes

To achieve this a new field and index in the derived.tick_state SQL table: related.block.header.height tracks the height at which the update to the tick was made, which allows us to select only the relevant data that has changed from a certain block (up to the current block), the table overwrites data rows for each tickIndex with newer height data so it is not possible to query a range that exists only in the past, #22 should resolve this by tracking much more state data and enabling this.

An event listener was also created to listen for newHeight events (when a new block height is seen in the block table).