search

cowprotocol / dune-sync

Components for syncing off-chain data with Dune Community Sources
4 stars 1 forks source link

[Batch Rewards] Schema & ETL Process for Sync #28

Closed bh2smith closed 1 year ago

bh2smith commented 1 year ago

Another part of #27 (based on #26)

This defined the "schema" for what will be uploaded to Dune. We reused/generalize some common code for the order rewards sync. More or less, this was a minor change. Still remaining todo for parsing the bytea[] type with participants field (since duneV2 uses lower case hex strings). We will want to replace all \x coming from postgres with 0x

bh2smith commented 1 year ago

How is this supposed to be run? As another pod running the same python script but with different arguments? Or is the same pod going to sync both order as well as batch rewards (we will need both for the transition period).

Initially this has been setup to run as another pod with different runtime arguments, but now that you mention it, we don't really need to do that since it will be phased out so soon. Perhaps we can just put both orderbook table syncs into the same job (they execute almost instantly and, use almost no computational resources and do nearly the same thing). Then later we can just drop the other job from here and version bump the production deployment!

Update: As cool as it might sound to run them both at the same time, it would require a bit of hacky refactoring (not relevant to this PR so I will not be doing that here).

bh2smith commented 1 year ago

This PR is getting pretty old. All comments have been addressed and I plan to merge this today.

bh2smith commented 1 year ago

Merging now as the general sentiment is correct. There have been some changes to the backend DB that will need to be addressed as a follow up once their structures have been deployed and populated.