Duneapi package integration

[josojo]

Unfortunately, this integration comes with a bigger refactor.

The package offers the ability to wait until a query calculation is finished. This is really nice and allows us to cut the amount of python script by a factor of 2: Previously, we had one script for executing the queries and a second one for fetching the results. Now, with the waiting feature, we can do it all in one go.

The duneapi package also allowed me to store the fetched dune data in a cleaner structure, as the parsing of the fetched result is better. Due to that I also needed to touch the rust code.

There is still a lot to be refactored to make the repo clean, but this should be okay as a first go.

testplan:

cd dune_api_scripts
python3 -m venv env
source ./env/bin/activate
pip install -r requirements.txt
source .env
export DUNE_DATA_FOLDER=./data/dune_data/
export APP_DATA_REFERRAL_RELATION_FILE=./data/referral_data/app_data_referral_relationship.json
python -m dune_api_scripts.store_query_result_all_distinct_app_data

Now, the referral data needs to be downloaded from ipfs. This takes quite long, if this should be done for all appdata. Hence, I would recommend deleting all entries, but 1, from the newly created distinct_app_data.json file in the data folder. Once it is done, one can run

cargo run

And wait until app_data_referral_relationship.json is written.

Now, one can run:

python -m dune_api_scripts.store_query_result_for_todays_trading_data
cargo run

and check that trading volume was downloaded and is served by checking:

http://127.0.0.1:8080/api/v1/profile/0x019d6ad53704fc851f1cc9effc12e89de4a1fc2c

[bh2smith]

One thing I noticed that we could probably move towards with these kinds of python projects is writing a single bash script containing all of this stuff

python3 -m venv venv
source ./venv/bin/activate
pip install -r requirements.txt
cp .env_sample .env

[bh2smith]

PS - the reason CI is failing is because duneapi requires python@3.10