search

blockchain-etl / ethereum-etl

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
https://t.me/BlockchainETL
MIT License
2.87k stars 812 forks source link

Large number of duplicate token extracted #489

Open rajeevpaudel opened 2 months ago

rajeevpaudel commented 2 months ago

While exporting all data for a certain block parallelly (6 process at a time, each for a day), I found out that the same tokens are being repeated for number of times

0x32d222e1f6386b3df7065d639870be0ef76d3599,Wrapped Ether,WETH,18,33924642310000000000, 0x32d222e1f6386b3df7065d639870be0ef76d3599,Wrapped Ether,WETH,18,33924642310000000000, 0x32d222e1f6386b3df7065d639870be0ef76d3599,Wrapped Ether,WETH,18,33924642310000000000, 0x32d222e1f6386b3df7065d639870be0ef76d3599,Wrapped Ether,WETH,18,33924642310000000000, 0x32d222e1f6386b3df7065d639870be0ef76d3599,Wrapped Ether,WETH,18,33924642310000000000, 0x32d222e1f6386b3df7065d639870be0ef76d3599,Wrapped Ether,WETH,18,33924642310000000000, 0x32d222e1f6386b3df7065d639870be0ef76d3599,Wrapped Ether,WETH,18,33924642310000000000, 0x32d222e1f6386b3df7065d639870be0ef76d3599,Wrapped Ether,WETH,18,33924642310000000000, 0x32d222e1f6386b3df7065d639870be0ef76d3599,Wrapped Ether,WETH,18,33924642310000000000, 0x32d222e1f6386b3df7065d639870be0ef76d3599,Wrapped Ether,WETH,18,33924642310000000000, 0x32d222e1f6386b3df7065d639870be0ef76d3599,Wrapped Ether,WETH,18,33924642310000000000, 0x32d222e1f6386b3df7065d639870be0ef76d3599,Wrapped Ether,WETH,18,33924642310000000000,

This is a example of extracted csv data