[Lake] Add retries around filling the data lake.

idiom-bytes commented 9 months ago

Motivation

As an outcome of reviewing 514/#552, we emphasized that it would be valuable to add a retry around filling the data_lake.

Outline

Any inputs/triggers that cause gql_lake (and perhaps olhcv_lake too) to fetch/sync, should offer a clean "retry" around the outer-loop such that ["failures from subgraph", "failures from ccxt"] and should provide it in the fn interface.

This should improve the UX of cli_commands/agents, such that building the lake is more resillient to subgraph/api errors.

DoD:

[x] Add retries around data_factory loops such that they resume from last_checkpoint.
[x] Continue evolving data factory & data modelling (i.e. beyond pdr analytics) example: volume ticks, f(x) w/ 100 params, improved risk models for trading execution.

trentmc commented 8 months ago

Note: commit https://github.com/oceanprotocol/pdr-backend/commit/674ca138fdd6afd34905a536498eca26a02246b9 auto-linked above accidentally pointed to this issue, rather than the correct one #653

idiom-bytes commented 4 months ago

We have various retries added and improved this overall. I have closed this ticket and have updated the core issue.

oceanprotocol / pdr-backend