Use new database layout

[fhenneke]

This is a work-in-progress PR for using the database tables. It is not meant for review but more as a record of the current state of the implementation.

Current state

• runs locally, alongside old functionality
• stores timestamps for transactions
• stores all tokens which emitted a transfer event
  • does not store ETH yet
  • also stores tokens traded between AMMs
• stores (at most) one ALL available prices per token
• stores decimals for all new tokens

Next steps

• disentangle logic between different data fetching steps
• add correct handling of errors (data fetching, data writing, ...)

[fhenneke]

The database setup seems to be working more or less now.

Testing price fetching currently seems to require api keys. The coingecko price fetcher cannot be set up since the token list cannot be fetched. (The actual price fetching contains special treatment for not having a key, but that part of the code cannot be reached.)

[socket-security[bot]]

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher
pypi/psycopg@3.2.3	environment, eval, filesystem, network, shell	0	691 kB	piro
pypi/pycryptodome@3.20.0	environment, eval, filesystem, unsafe	0	4.51 MB	Legrandin
pypi/pydantic-core@2.23.4	None	0	0 B
pypi/pydantic@2.9.2	None	0	0 B

🚮 Removed packages: pypi/numpy@1.26.4, pypi/pandas-stubs@2.2.2.240909, pypi/pandas@2.2.1, pypi/psycopg2@2.9.9

View full report↗︎

[harisang]

Pushed a commit that comments out the old code except for the raw token imbalances calculation. The writing in the new tables is also based in the token imbalances token list, as this is more efficient (as we do not need prices for tokens with an imbalances of exactly zero).

Merging so as to test for a few hours/days. A folllow-up clean-up PR should follow.

Note: all new db tables have been manually created based on the definitions used in this PR.