demeter-fetch fails to fetch data

[GuyMcComb]

Hi team,

demeter-fetch is failing to pull data from an rpc endpoint. I am trying to use the tool with the example config provided in the docs, but the command fails to execute without errors.

env:

python --version
Python 3.10.12

(venv) :~/repos/demeter-fetch$ pip freeze
cachetools==5.5.0
certifi==2024.8.30
charset-normalizer==3.4.0
db-dtypes==1.3.0
demeter-fetch==1.1.4
google-api-core==2.21.0
google-auth==2.35.0
google-cloud-bigquery==3.26.0
google-cloud-core==2.4.1
google-crc32c==1.6.0
google-resumable-media==2.7.2
googleapis-common-protos==1.65.0
grpcio==1.67.0
grpcio-status==1.67.0
idna==3.10
numpy==2.1.2
packaging==24.1
pandas==2.2.3
proto-plus==1.24.0
protobuf==5.28.2
pyarrow==17.0.0
pyasn1==0.6.1
pyasn1_modules==0.4.1
python-dateutil==2.9.0.post0
pytz==2024.2
requests==2.32.3
rsa==4.9
six==1.16.0
toml==0.10.2
tqdm==4.66.5
tzdata==2024.2
urllib3==2.2.3

config:

[from]
chain = "ethereum"
datasource = "rpc" # change datasource from big_query to rpc
dapp_type = "uniswap"
start = "2023-11-25"
end = "2023-11-25"

[from.uniswap]
pool_address = "0x88e6a0c2ddd26feeb64f039a2c41296fcb3f5640"

[from.rpc] # remove from.big_query and set this section
end_point = "https://eth.llamarpc.com"
ignore_position_id = true

[to]
type = "minute"
save_path = "./sample"

command:

demeter-fetch -c config.sample.toml

error:

(venv):~/repos/demeter-fetch$ demeter-fetch -c config.sample.toml
Traceback (most recent call last):
  File "/home/repos/demeter-fetch/venv/bin/demeter-fetch", line 33, in <module>
    sys.exit(load_entry_point('demeter-fetch==1.1.4', 'console_scripts', 'demeter-fetch')())
  File "/home/repos/demeter-fetch/venv/bin/demeter-fetch", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/repos/demeter-fetch/venv/lib/python3.10/site-packages/demeter_fetch/__init__.py", line 1, in <module>
    from demeter_fetch.common._typing import *
  File "/home/repos/demeter-fetch/venv/lib/python3.10/site-packages/demeter_fetch/common/__init__.py", line 1, in <module>
    from ._typing import *
  File "/home/repos/demeter-fetch/venv/lib/python3.10/site-packages/demeter_fetch/common/_typing.py", line 7, in <module>
    class DataSource(enum.StrEnum):
AttributeError: module 'enum' has no attribute 'StrEnum'

---

When I upgrade my python version to 3.12 and run the same command I get:

(venv12):~/repos/demeter-fetch$ demeter-fetch -c config.sample.toml
Traceback (most recent call last):
  File "/home/repos/demeter-fetch/venv12/bin/demeter-fetch", line 5, in <module>
    from demeter_fetch.main import main
  File "/home/repos/demeter-fetch/venv12/lib/python3.12/site-packages/demeter_fetch/__init__.py", line 2, in <module>
    from demeter_fetch.core import download_by_config
  File "/home/repos/demeter-fetch/venv12/lib/python3.12/site-packages/demeter_fetch/core/__init__.py", line 8, in <module>
    from .downloader import download,download_by_config
  File "/home/repos/demeter-fetch/venv12/lib/python3.12/site-packages/demeter_fetch/core/downloader.py", line 15, in <module>
    from . import engine
  File "/home/repos/demeter-fetch/venv12/lib/python3.12/site-packages/demeter_fetch/core/engine.py", line 12, in <module>
    from ..processor_uniswap import UniUserLP, UniPositions, UniTick, UniTickNoPos, UniMinute
  File "/home/repos/demeter-fetch/venv12/lib/python3.12/site-packages/demeter_fetch/processor_uniswap/__init__.py", line 3, in <module>
    from .position import UniPositions, UniUserLP
  File "/home/repos/demeter-fetch/venv12/lib/python3.12/site-packages/demeter_fetch/processor_uniswap/position.py", line 18, in <module>
    from ..sources.rpc_utils import set_position_id
  File "/home/repos/demeter-fetch/venv12/lib/python3.12/site-packages/demeter_fetch/sources/__init__.py", line 1, in <module>
    from .source_core import (
  File "/home/repos/demeter-fetch/venv12/lib/python3.12/site-packages/demeter_fetch/sources/source_core.py", line 12, in <module>
    from .big_query import bigquery_aave, bigquery_pool, bigquery_proxy_lp, bigquery_proxy_transfer, bigquery_transaction
  File "/home/repos/demeter-fetch/venv12/lib/python3.12/site-packages/demeter_fetch/sources/big_query.py", line 11, in <module>
    from .big_query_utils import BigQueryChain, query_by_sql
  File "/home/repos/demeter-fetch/venv12/lib/python3.12/site-packages/demeter_fetch/sources/big_query_utils.py", line 13, in <module>
    class BigQueryChain(Enum):
  File "/home/repos/demeter-fetch/venv12/lib/python3.12/site-packages/demeter_fetch/sources/big_query_utils.py", line 34, in BigQueryChain
    arbitrum = {
    ^^^^^^^^
  File "/usr/lib/python3.12/enum.py", line 443, in __setitem__
    raise TypeError('%r already defined as %r' % (key, self[key]))
TypeError: 'arbitrum' already defined as {'table_name': 'bigquery-public-data.goog_blockchain_arbitrum_one_us.logs', 'tx_table_name': 'bigquery-public-data.goog_blockchain_arbitrum_one_us.transactions'}

and just for consistancy here is my python3.12 env:

(venv12) guy@guyslaptop:~/repos/demeter-fetch$ pip freeze
cachetools==5.5.0
certifi==2024.8.30
charset-normalizer==3.4.0
db-dtypes==1.3.0
demeter-fetch==1.1.4
google-api-core==2.21.0
google-auth==2.35.0
google-cloud-bigquery==3.26.0
google-cloud-core==2.4.1
google-crc32c==1.6.0
google-resumable-media==2.7.2
googleapis-common-protos==1.65.0
grpcio==1.67.0
grpcio-status==1.67.0
idna==3.10
numpy==2.1.2
packaging==24.1
pandas==2.2.3
proto-plus==1.24.0
protobuf==5.28.2
pyarrow==17.0.0
pyasn1==0.6.1
pyasn1_modules==0.4.1
python-dateutil==2.9.0.post0
pytz==2024.2
requests==2.32.3
rsa==4.9
six==1.16.0
toml==0.10.2
tqdm==4.66.5
tzdata==2024.2
urllib3==2.2.3

Any help would be appreciated!

[32ethers]

There issues on setup.py.

minimal python version should be 3.11

I have release a new version 1.1.6, please try this

I didn't reproduce the second problem. you can try again

[32ethers]

and i tried you rpc: end_point = "https://eth.llamarpc.com" and find it has query limit. if you use this you will fail when query timestamp for every block. because there will be a lot of query

you can sign up an alchemy account,