near / near-one-project-tracking

A repository for tracking work items that NEAR One is working on.
0 stars 0 forks source link

[ProjectTracking]: New bridge indexer #53

Open akhi3030 opened 8 months ago

akhi3030 commented 8 months ago

Description

For more than a year already, there were two separate components to provide the observability of Rainbow Bridge operations: Bridge Monitoring and Bridge Indexer. The decoupling of these components in terms of development, along with some erroneous design decisions, made the system not reliable enough and hard to support.

As a result of this, the decision was made to join two components and update the technological stack to meet the following goals.

Goals

Links to external documentation and discussions

Estimated effort

Assumptions

Pre-requisites

Initial pre-requisites:

Out of scope

kisialiou commented 8 months ago

During the last 2 weeks, we worked on the following issue.

Its main goals were:

The work resulted in the PR which is currently in the process of fixing several minor comments.

Immediate plans include:

See more details in the corresponding project roadmap.

kisialiou commented 8 months ago

March 22nd update

Completed:

Immediate plans:

  1. Deployment of updated API.
  2. Establishing monitoring for indexer/processor hangs.

See more details in the corresponding project roadmap.

kisialiou commented 7 months ago

April 5th update

Completed:

Immediate plans:

  1. Establishing monitoring for indexer/processor hangs.
  2. Integration of Ethereum indexer.

Shift in dealines: due to appeared bugs and unexpected development, the expected deadline is shifted from end of May to the beginning of June (~5-10th of June).

See more details in the corresponding project roadmap.

kisialiou commented 7 months ago

April 19th update

Completed:

Immediate plans:

  1. Finalize the enricher implementation and migrate all existing tools to the new data.

See more details in the corresponding project roadmap.

kisialiou commented 6 months ago

May 5th update

Worked on the enricher implementation. Namely:

Immediate plans:

  1. Test the current solution.
  2. Implement transaction "matching" functionality.

Unfortunately, due to underestimated feature complexity and national holidays, the development of the enricher appeared to be longer than expected, thus shifting overall project delivery date by ~10 days, i.e. to ~20th of June.

Nevertheless, I plan to finish the project still as expected ~10th of June.

kisialiou commented 6 months ago

May 17th update

Due to my short vacation, national holidays and work on other tasks, there is little to no progress on the indexer at the moment. I plan to return to it next week after accomplishing more prioritized tasks.

kisialiou commented 5 months ago

May 31st update

During the previous week, I was mostly concentrated on eth-connector migration monitoring tools which again postponed the work on the indexer. Still, in connection with the emergence of a new NEP-141 connector, support for it was added to the indexer.

kisialiou commented 5 months ago

June 14th update

Completed:

  1. Finished the implementation of enricher and transactions linking.
  2. Deployed the enricher.
  3. Completely migrated Bridge API to the new indexer data.
  4. Created new panel and alerts to monitor enrichment process.
  5. Wrote README + documentation for all components of new bridge indexer.

After 2 week vacation (back at work on 07/01):

  1. Start the implementation of Ethereum indexer.
kisialiou commented 4 months ago

July 12th update

Completed:

  1. Refined the code for processing of Aurora-related transactions.
  2. Updated enricher to work with Aurora-related transactions.
  3. Migrated and redeployed indexer to the new address of EthConnector.

Immediate plans:

  1. Finalize the support of Aurora-related transactions processing and deploy.
  2. Create a new Bridge dashboard with enhanced capabilities (e.g. search by tx hash), relying on the new indexer.
kisialiou commented 4 months ago

July 26th update

Completed:

  1. Finalized the support of Aurora-related transactions processing and deployed.
  2. Added support for linking Fast Bridge transactions.
  3. Added minor improvements and bug fixes to current indexer and processor implementations.
  4. Added minor changes, ensuring correct processes restart after server crashes.

Immediate plans:

  1. Add Fast Bridge alerts, relying on the new indexer.
  2. Create a new Bridge dashboard with enhanced capabilities (e.g. search by tx hash), relying on the new indexer.
kisialiou commented 3 months ago

August 9th update

Completed:

  1. Set up monitoring for RAM and CPU on the server.
  2. Deployed new version and resolved issues for Fast Bridge testnet indexing. Made Fast Bridge mainnet reprocessing with new data.
  3. Wrote, tested and deployed alerts for Fast Bridge with a new indexer.
  4. Explored alternatives to Grafana for dashboards: Metabase, PowerBI.
  5. Created a "Rainbow Bridge Explorer" dashboard in Grafana, providing search and filtering capabilities over Rainbow Bridge transactions. The dashboards rely on new indexer data.

Immediate plans:

  1. Optimize speed for the "Rainbow Bridge Explorer" dashboard and refine the interface.
  2. Create a dashboard with metrics for the connector's statistics.
kisialiou commented 3 months ago

August 22nd update

Completed:

  1. Finalized query optimization and formatting for the "Rainbow Bridge Explorer" dashboard. Achieved nearly instant result loads for most queries.
  2. Investigated and reported a bug in Coingecko which caused incorrect data in the indexer.
  3. Added alerting for BTC light client on testnet.
  4. Detected and investigated problems with missing data.

Immediate plans:

  1. Add support for new methods to cover the missing data.
  2. Create a dashboard with metrics for the Ethereum connector's statistics.