Epic: Refactor Agent Indexing (Mar/Apr '24)

avious00 commented 7 months ago

Context

We need to improve the robustness of our agents indexing. Based on this Notion document proposal we created this epic to address the top issues.

Details

Notion document

See all issues https://github.com/hyperlane-xyz/hyperlane-monorepo/issues?q=is%3Aissue+is%3Aopen+indexing, need to sift through and decide which are important & plan out what we should do to address them

We should do these ASAP to derisk our indexing situation across the board. This is in chronological order.

### Things to do ASAP
- [ ] https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3224
- [ ] https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3213
- [ ] https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3264
- [ ] https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3267

Longer term issues for better indexing positioning have been moved to https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3414

Desired end state

Cursors:

These are the cursors I expect we'll have at the end of this all - many need renaming:

Cursor Name	Functionality	Range types?	Direction?	Relevant issue	Expected use
ForwardSequenceAwareSyncCursor	Looks at onchain sequence counts so it knows when new logs can be indexed. Ensures no sequence gaps in logs.	Block ranges or sequence ranges	Forward	https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3213	Messages & merkle tree insertions for all chains; IGP payments / deliveries when sequence-range-based indexing, like for Solana
BackwardSequenceAwareSyncCursor	Ensures no sequence gaps in logs. Stops indexing once sequence 0 has been indexed.	Block ranges or sequence ranges	Backward	https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3213	Messages & merkle tree insertions for all chains; IGP payments / deliveries when sequence-range-based indexing, like for Solana
ForwardBackwardSequenceAwareSyncCursor	Wrapper around the forward & backward sequence-aware cursors. Prioritizes the forward cursor	Block ranges or sequence ranges	Forward & Backward	https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3213	Messages & merkle tree insertions for all chains; IGP payments / deliveries when sequence-range-based indexing, like for Solana
RateLimitedContractSyncCursor (probably needs rename)	Syncs forward block ranges, if it's close to the tip then it rate limits itself so we're not making many expensive block range requests	Block ranges only	Forward	https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3264	Whenever sequence-range-based indexing isn't possible, e.g. IGP payments / deliveries for EVM chains, etc
BackwardSlidingBlockRangeSyncCursor (probably needs rename)	Syncs backward in block ranges, stopping at the index.from	Block ranges only	Backward	https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3336	Whenever sequence-range-based indexing isn't possible, e.g. IGP payments / deliveries for EVM chains, etc
ForwardBackwardSlidingBlockRangeSyncCursor (probably needs rename)	Wrapper around the forward RateLimitedContractSyncCursor & backward BackwardSlidingBlockRangeSyncCursor. Prioritizes the forward cursor	Block ranges only	Backward	https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3336	Whenever sequence-range-based indexing isn't possible, e.g. IGP payments / deliveries for EVM chains, etc

avious00 commented 7 months ago

hey @tkporter is https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3171 still relevant here? should it be pulled into this epic

tkporter commented 7 months ago

thanks for calling it out, closed as it should be fixed now

avious00 commented 7 months ago

hey @tkporter did this consider https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2308

tkporter commented 7 months ago

@avious00 yeah part of https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3264 is to think about this. Let's keep https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2308 open for now

hyperlane-xyz / hyperlane-monorepo