search

getamis / eth-indexer

An Ethereum project to crawl blockchain states into database
GNU Lesser General Public License v3.0
233 stars 47 forks source link

Data not in database #129

Open AmbushIV opened 5 years ago

AmbushIV commented 5 years ago

Hey guys, I am having the following problem. I got all the setup right and ran through docker-compose up

Now, I connected to my database via remote connection but I don't have any entries in any of the tables and I keep getting errors in cmd line:

indexer_1        | CRIT [02-06|11:10:58|indexer/root.go:183]       ServerCmd Execution failed               err="no contract code at given address"
indexer_1        |       --indexer.db.cred.user string         The database username to login (default "root")
indexer_1        |       --indexer.db.cred.vault.path string   Vault path for indexer database credential (default "database/creds/ethdb")
indexer_1        |       --metrics                             Enable metrics
indexer_1        |       --metrics.host string                 Metrics listening host
indexer_1        |       --metrics.port int                    Metrics listening port (default 9092)
indexer_1        |       --pprof                               Enable the pprof HTTP server
indexer_1        |       --pprof.host string                   pprof HTTP server listening interface (default "0.0.0.0")
indexer_1        |       --pprof.port int                      pprof HTTP server listening port (default 8000)
indexer_1        |       --sync.fromBlock int                  The init block number to sync to initially
indexer_1        |       --vault.ca.path string                The path of vault CA file (default "/etc/ssl/certs/amis/vault.pem")
indexer_1        |       --vault.host string                   The vault server host
indexer_1        |
indexer_1        | INFO [02-06|11:11:59|indexer/root.go:235] No config file to load as default.       info="Config File \"config\" Not Found in \"[/configs]\""
indexer_1        | INFO [02-06|11:11:59|multiclient/option.go:29] EthClients from static list              urls=[ws://geth:8546]
indexer_1        | TRACE[02-06|11:11:59|multiclient/map.go:62]    Eth client added                         url=ws://geth:8546
indexer_1        | INFO [02-06|11:11:59|multiclient/client.go:780] Connect to eth client successfully       url=ws://geth:8546
indexer_1        | DEBUG[02-06|11:11:59|indexer/root.go:149]       erc20Addresses Successfully Loaded
indexer_1        | ERROR[02-06|11:11:59|client/client.go:125]      Failed to get total supply               addr=0x86Fa049857E0209aa7D9e616F7eb3b3B78ECfdb0 err="no contract code at given address"
indexer_1        | ERROR[02-06|11:11:59|indexer/indexer.go:103]    Failed to get ERC20                      addr=0x86fa049857e0209aa7d9e616f7eb3b3b78ecfdb0 err="no contract code at given address"
indexer_1        | ERROR[02-06|11:11:59|indexer/root.go:152]       Fail to subscribe ERC20Tokens and write to database err="no contract code at given address"

Any ideas on why this is happening?

AmbushIV commented 5 years ago

After a bit of configuration I got it working but now I wonder, does it automatically parse all the transactions data and populate the transactions table? If so, when?

markya0616 commented 5 years ago

Yes, it should be. Indexer inserts the transaction data while it receives the new head event.

AmbushIV commented 5 years ago

I left it running for days, got to about 4m blocks but I am not having any data in the transactions table, nor in block_headers (I managed to start indexer later in the sync). All I got is data in transactions_dificulty, any idea on this? Thank you for your answer!

AmbushIV commented 5 years ago

Yes, it should be. Indexer inserts the transaction data while it receives the new head event.

How much space do I need to run this program of yours on my PC? I noticed I have a syncmode property on full which means over 1 TB of storage is needed. Is this true?

markya0616 commented 5 years ago

I left it running for days, got to about 4m blocks but I am not having any data in the transactions table, nor in block_headers (I managed to start indexer later in the sync). All I got is data in transactions_dificulty, any idea on this? Thank you for your answer!

Do you sync on mainnet or testnet? If you use fast sync, you cannot get new head events until the chain is synced.

How much space do I need to run this program of yours on my PC?

It depends on the number blocks, transactions and subscriptions. However, the 1TB is for the blockchain data, not for the db data.

AmbushIV commented 5 years ago

I left it running for days, got to about 4m blocks but I am not having any data in the transactions table, nor in block_headers (I managed to start indexer later in the sync). All I got is data in transactions_dificulty, any idea on this? Thank you for your answer!

Do you sync on mainnet or testnet? If you use fast sync, you cannot get new head events until the chain is synced.

How much space do I need to run this program of yours on my PC?

It depends on the number blocks, transactions and subscriptions. However, the 1TB is for the blockchain data, not for the db data.

Thank you for your reply. I sync on the mainnet and I figured that if I delete those addreses from the erc20.yaml config file it would start adding entries to my database.

Also, I've deleted the gcmode=archive parameter in order to reduce the blockchain data as I care only about having all the transactions in a database. So far so good, I think it is saving everything as intended in the database. I'm at 4.8m blocks with the chain data and 3.5m blocks + 25m transactions saved in the DB.

Are there any troubles with saving the blockchain data without the gcmode? I've read and basically I'm syncing in "full" mode but with that pruning feature so I will have about 200 GB of blockchain data in the end. It should still have all the blocks and transactions, right?

markya0616 commented 5 years ago

Are there any troubles with saving the blockchain data without the gcmode? I've read and basically I'm syncing in "full" mode but with that pruning feature so I will have about 200 GB of blockchain data in the end. It should still have all the blocks and transactions, right?

We must use archieve gcmode. archieve mode keeps all states for the new blocks in disk. If reorgs happen, indexer may need to refer to old blockchain data. In that case, we may need the archieve data.

It should still have all the blocks and transactions, right?

Yes, the data should be in DB.

AmbushIV commented 5 years ago

Are there any troubles with saving the blockchain data without the gcmode? I've read and basically I'm syncing in "full" mode but with that pruning feature so I will have about 200 GB of blockchain data in the end. It should still have all the blocks and transactions, right?

We must use archieve gcmode. archieve mode keeps all states for the new blocks in disk. If reorgs happen, indexer may need to refer to old blockchain data. In that case, we may need the archieve data.

It should still have all the blocks and transactions, right?

Yes, the data should be in DB.

Is it possible for me to use syncmode="full" until the sync is complete and then restart with the gcmode="archieve" ? If not then I will stick to the actual way I am using now. Thank you for your time and help.

jackysmith9901 commented 5 years ago

indexer_1 | INFO [02-06|11:11:59|indexer/root.go:235] No config file to load as default. info="Config File \"config\" Not Found in \"[/configs]\"" indexer_1 | INFO [02-06|11:11:59|multiclient/option.go:29] EthClients from static list urls=[ws://geth:8546] indexer_1 | TRACE[02-06|11:11:59|multiclient/map.go:62] Eth client added url=ws://geth:8546 indexer_1 | INFO [02-06|11:11:59|multiclient/client.go:780] Connect to eth client successfully url=ws://geth:8546 indexer_1 | DEBUG[02-06|11:11:59|indexer/root.go:149] erc20Addresses Successfully Loaded indexer_1 | ERROR[02-06|11:11:59|client/client.go:125] Failed to get total supply addr=0x86Fa049857E0209aa7D9e616F7eb3b3B78ECfdb0 err="no contract code at given address" indexer_1 | ERROR[02-06|11:11:59|indexer/indexer.go:103]

After a bit of configuration I got it working but now I wonder, does it automatically parse all the transactions data and populate the transactions table? If so, when?

It's great , I have the same problem, How did you confige it and it work?