search

data-preservation-programs / RetrievalBot

A scalable framework to perform retrieval testing over Filecoin network
Other
13 stars 3 forks source link

API for filplus team to pull the retrieval bot data #25

Open xinaxu opened 1 year ago

xinaxu commented 1 year ago

https://filecoinproject.slack.com/archives/C05NRCURT6K/p1692957100158189

Use or create a new materialized view for this purpose

xmcai2016 commented 1 year ago

I am not able to see the slack message for some reason, (these messages expire after a while) could you paste the text in the description here or summarize what is needed here / timeline? @xinaxu Fil+ team should be encouraged to directly consume from Polybase and write their own logic to make sense of the data.

xinaxu commented 1 year ago

From Catalin Bara: So to sum things up, we want to persist daily data in our db and build the cumulative info ourseleves. We can work with an api that returns data in the current AWS api format, we just need to know when we run the cron job that fetches new data, how far in the past we should look to get final data.

xinaxu commented 1 year ago

The current API is not efficient as each request is a mongo aggregation query. The ask is to use a materialized view to reduce the db load