cowprotocol / dune-sync

Components for syncing off-chain data with Dune Community Sources
4 stars 1 forks source link

[Schema] Order Rewards to use JSON field for variable data. #18

Closed bh2smith closed 1 year ago

bh2smith commented 1 year ago

The schema proposed here is [order_uid: str | tx_hash: str | solver: str | data ] where (currently) data takes the following form:

Data {
   surplus_fee: str (we can make make this nullable, or use zero in place of null)
   amount: float
   safe_liquidity: boolean (nullable)
}

Examples, of this are provided in the unit test.

Additional Notes

  1. Technically the solver column isn't necessary since it can be obtained from the tx_hash. Let me know if we think it should be removed.

  2. I spoke with Dune today about string types in spark SQL and their answer was as I expected:

We use decimal(38,0) or just strings for large integers in spark. DuneSQL will provide native support for large integers but it’s not directly available in spark sql

Note that, we will use strings for the surplus_fee and cast to Decimal in the spell that parses this content.

Test Plan

with all the appropriate credentials you can run:

python -m src.main --sync-table order_rewards --dry-run True

Which will process orders and write locally. A sample output with two rows would look like this:

{"order_uid": "0xa386250eda29e80d804b228908968865ab31fa8023abc5bbbf532ba965ecaa6f844eb02130e1849bf94e15ef08cdf070fbf1302b639270eb", "tx_hash": "0x2a6ccd340d7fa12d483851d0d7fa5ae6627ec852a2e3c638bec3deac1c1837be", "solver": "0x97ec0a17432d71a3234ef7173c6b48a2c0940896", "data": {"surplus_fee": "0", "amount": 40.21522326123023, "safe_liquidity": null}}
{"order_uid": "0xb0677cedb8279bfde76e6c15cc264b61386d2e2b4ddeb76d59cea0a635c59eb0b2515a7221b2654f9faae0e4ed1d0e49aa7b85dd63927116", "tx_hash": "0x7e482a5b04dc78d64a36588e90d549ecfb2d5d9f90d261848bbfc3269758cda5", "solver": "0xc9ec550bea1c64d779124b23a26292cc223327b6", "data": {"surplus_fee": "0", "amount": 38.97091536669401, "safe_liquidity": null}}