yExporter #2

[0xBasically]

Scope

This is a budget request for maintaining all exporter-related tasks. The exporter is a service that powers yearn.vision, calculates APYs for all our networks, serves data for various internal and external frontends, and is used to visualize current and historical partner, treasury, and transaction data.

Responsibilities

• handle up status and monitoring for:

  • staging and prod environments for yearn.vision
  • staging and prod environments for yearn-exporter-api which is the base API for partners.yearn.finance
  • prod Yearn APY data which is the base APY API for yDaemon
  • prod Zootroop revenue calculations
  • prod etherscan cache
  • prod sentry for exception tracing

• code maintenance for:

  • yearn-exporter
  • yearn-exporter-api
  • yMonitor
  • ypricemagic
    • will ensure support for all tokens necessary for y-e.
  • dank_mids
  • eth-portfolio

Plan

As of today, Crebsy and BobTheBuilder are taking care of most of the maintenance and new feature implementation to Exporter's codebase. 0xBasic is reviewing PRs, maintaining and improving grafana dashboards while CSdata1 works on integrating statistics and machine learning models into the Exporter. We will integrate new Yearn products thus push metrics users want to see to Yearn.Vision, and by cleaning the codebase get Exporter to a point where it can move into maintance only mode by the end of the year. Next to these development tasks, we'll serve and monitor the APIs that feed yDaemon, the partner scripts, and other existing and future frontends.

Tasks

• add yETH support

  • add yETH grafana dashboard

• add yVaults v3 support

  • add support for networks v3 will be deploying on
  • build new v3 vault exporter
  • add v3 support to apy script & monitoring

• consolidate data layer for open data science:

  • We're using a mixture of victoria-metrics db and postgres db. A spike to replace victoria by timescale+promscale has been closed because promscale has been officially discontinued this year.
  • The current solution with victoria-metrics is stable and works but lacks flexibility for partial updates and is not easy to integrate into BI systems like metabase.
  • A new effort will be taken to create a datamodel with postgres-based timescale db which will enable us to have more control about the indexed web3 data and let us integrate it into 3rd party tools more easily. This will also help external data scientists to work with the exporter data more easily.

• grafana dashboard statistics and enhancements

  • Building a statistics and machine learning dashboard in Grafana.
  • Enhance and update all current dashboard sections, [General, Strategists, System, Treasury, Treasury-dashboards, Yearn] to include customized graphs with statistics and metrics each team in yearn can utilize built to their specific needs.
  • More info found here. BR 143 will be closed and added into yExporter team's budget request.

• fix tests

  • many older tests are failing which delays refactorings, leads to regressions and implies a lot of manual testing
  • migrate price-related tests to ypricemagic repo to reduce run time

• yearn-exporter-api updates and improvements for partners and apps

  • explore currently unmet data needs for seafood and provide the required metrics via rest api

• replace eth-brownie with ape once ape has async support

  • ape-dankmids

• finish Balancer apys

  • Calculate forward-looking APYs like we do with curve and convex, for bal and aura strategies.

Duration

:information_source: This budget request is for a period of 3 months and will be evaluated at the end of the period.

This budget request will allow us to get Exporter to a state where we can go into "maintance mode". This is a state where we dont add new features but the current feature set should suffice and alow us to maintain it with mostly fixes and small improvements when needed. Thus lowering total costs while keeping everything working as normal: exporters, apis, monitoring. We expect this to cut costs to 15k per month moving forward.

Public Group

YFI Exporter (public) bridged to discord

Status

Past budget request

:information_source: https://github.com/yearn/budget/issues/126

New tasks

• [ ] add yETH support
• [ ] add yVaults v3 support
• [ ] fix tests
• [ ] consolidate data layer for open data science
• [ ] grafana dashboard statistics and enhancements
• [ ] yearn-exporter-api updates and improvements
• [ ] replace eth-brownie with ape once ape has async support
• [ ] Balancer apys

  Finished tasks from last budget request (3m of the 6m scope)

• [x] finish dank async refactoring
  • this lets us build and test features and fixes in a fraction of the time
• [x] finish yexporter async refactoring
• [x] fix treasury exporter
• [x] fix apy
  • [x] curve fwd-looking apy for all chains
  • [x] add velo calcs
• [x] general maintenance
• [x] migrated codebase to python 3.10
• [x] improved observability stack with sentry + ELK

Deadline

2023-10-31

People

• Cresby
• BobTheBuilder
• 0xBasic
• CSdata1

Money

Monthly breakdown:

Type	Description	DAI	YFI
HOSTING	3x server	390	0
NODES	5x networks	500	0
DEV	crebsy	7000	0.75
DEV	BobTheBuilder	7000	0.75
DEV	0xBasic	7000	0.75
DATA ANALYST	CSdata1	1500	0.20
=======	=============	========	=======
	TOTAL	23390	2.45

TOTAL AMOUNT	Month	DAI	YFI	Description
July	500	0	NODES
August	16390	1.70	HOSTING, NODES, DEVS, (cresby time off)
September	23390	2.45	HOSTING, NODES, DEVS
October	23390	2.45	HOSTING, NODES, DEVS
=========	======	====	===========
	63670	6.6

Amount

63670 DAI + 6.6 YFI

Wallet address

0xcD63C69f08bdDa7Fe96a87A2Ca3f56f3a6990a75

Reporting

Monthly

[0xBasically]

Updated maintenance mode costs for clarity