search

blockchain-etl / ethereum-etl

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
https://t.me/BlockchainETL
MIT License
2.97k stars 852 forks source link

Proposal - Allow to get information about tokens even if there are not ERC20 o ERC721 #240

Open alexon1234 opened 3 years ago

alexon1234 commented 3 years ago

Hi,

I would like to propose to remove or allow to override the current filter is placed in the ExtractTokensJob class that filters all the contracts that are not erc20 or erc721.
There are multiple contracts that are not erc20 neither erc721, but they expose the symbol() and name() methods, so we could still populate the tokens.csv with this information.

Currently, in the repository https://github.com/blockchain-etl/ethereum-etl-airflow, we keep the following csv. This csv is used to add manually tokens that this ETL is not able to process.

But if we take the tokens address from this csv and run

python ethereumetl.py export_tokens --token-addresses contract_addresses.txt -p https://mainnet.infura.io/v3/XXXXX -o tokens.csv

We will get the following

address,symbol,name,decimals,total_supply,block_number
0xd533a949740bb3306d119cc777fa900ba034cd52,CRV,Curve DAO Token,18,1450457314060250117622832176,
0x221657776846890989a759ba2973e427dff5c9bb,REPv2,Reputation,18,5928073097034991490330330,
0x1985365e9f78359a9b6ad760e32412f4a445e862,REP,Reputation,18,11000000000000000000000000,
0x6b175474e89094c44da98b954eedeac495271d0f,DAI,Dai Stablecoin,18,2578507524520975022044854399,
0x9f8f72aa9304c8b593d555f12ef6589cc3a579a2,,,18,995238778286468792512963,
0x7fc66500c84a76ad7e9c93437bfc5ac33e2ddae9,AAVE,Aave Token,18,16000000000000000000000000,
0x1f9840a85d5af5bf1d1762f925bdaddc4201f984,UNI,Uniswap,18,1000000000000000000000000000,
0x4fabb145d64652a948d72533023f6e7a623c7c53,BUSD,Binance USD,18,3180218425530000000000000000,
0x0000000000085d4780b73119b644ae5ecd22b376,TUSD,TrueUSD,18,284582051320000000000000000,
0x0bc529c00c6401aef6d220be8c6ea1667f6ad93e,YFI,yearn.finance,18,36666000000000000000000,
0xc00e94cb662c3520282e6f5717214004a7f26888,COMP,Compound,18,10000000000000000000000000,
0x04fa0d235c4abf4bcf4787af4cf447de572ef828,UMA,UMA Voting Token v1,18,101742296880803612603530945,
0xd26114cd6ee289accf82350c8d8487fedb8a0c07,OMG,OMGToken,18,140245398245132780789239631,
0x0000000000004946c0e9f43f4dee607b0ef1fa1c,CHI,Chi Gastoken by 1inch,0,1412443,
0x111111111117dc0aa78b770fa6a738034120c302,1INCH,1INCH Token,18,1500000000000000000000000000,
0xc944e90c64b2c07662a292be6244bdf05cda44a7,GRT,Graph Token,18,10051452321666598216635259792,
0x6b3595068778dd592e39a122f4f5a5cf09c90fe2,SUSHI,SushiToken,18,204430147688771048000835347,
0xbbbbca6a901c926f240b89eacb641d8aec7aeafd,LRC,LoopringCoin V2,18,1374378440442457437622945100,
0x8e870d67f660d95d5be530380d0ec0bd388289e1,PAX,Paxos Standard,18,746722130910000000000000000,
0x0000000000095413afc295d19edeb1ad7b71c952,LON,Tokenlon,18,110000000000000000000000000,
0xa117000000f279d81a1d3cc75430faa017fa5a2e,ANT,Aragon Network Token,18,39974728552947991110240591,
0xba100000625a3754423978a60c9317c58a424e3d,BAL,Balancer,18,40800000000000000000000000,
0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48,USDC,USD Coin,6,9209083350459550,
0x0e2298e3b3390e3b945a5456fbf59ecc3f55da16,YAM,YAM,18,574622356457309677448366755385476442247771087774958689155945907855212985,
0x967da4048cd07ab37855c090aaf366e4ce1b9f48,OCEAN,Ocean Token,18,613099141000000000000000000,
0x374CB8C27130E2c9E04F44303f3c8351B9De61C1,BAO,BaoToken,18,899592413827554209204245650437,
0xeb4c2781e4eba804ce9a9803c67d0893436bb27d,renBTC,renBTC,8,1296676639145,
0x89d24a6b4ccb1b6faa2625fe562bdd9a23260359,,,18,3729459392347553152988922,
0xc24EE3a1b395c600EB74eb8293c203c123AbF71F,ROSE,RoseToken,18,5021021233475066318957,
0x66a0f676479cee1d7373f3dc2e2952778bff5bd6,WISE,Wise Token,18,281419362318160544357525959,
0xC9cB53B48A2f3A9e75982685644c1870F1405CCb,SLP,SushiSwap LP Token,18,1400679413587875993,
0x2b591e99afE9f32eAA6214f7B7629768c40Eeb39,HEX,HEX,8,56985450880969914415,
0xb6ca7399b4f9ca56fc27cbff44f4d2e4eef1fc81,MUSE,Muse,18,531455615144573014557591,
0x030ba81f1c18d280636f32af80b9aad02cf0854e,aWETH,Aave interest bearing WETH,18,301902057190141366464508,
0x831091da075665168e01898c6dac004a867f1e1b,GFARM2,Gains V2,18,32684552263096501335880,
0x8ab98c28295ea3bd2db6ac8b3ca57a625c054bd1,SFR,SunflowerToken,18,9379510163633069805962,
0x37236cd05b34cc79d3715af2383e96dd7443dcf1,SLP,Small Love Potion,0,262814694,
0xbcca60bb61934080951369a648fb03df4f96263c,aUSDC,Aave interest bearing USDC,6,238065785095978,
0x629a673a8242c2ac4b7b8c5d8735fbeac21a6205,SOR,Sorare,,160564,
0x19d97d8fa813ee2f51ad4b4e04ea08baf4dffc28,bBADGER,Badger Sett Badger,18,2871785029514388340279733,
0x41c028a4c1f461ebfc3af91619b240004ebad216,TACO,TacoToken,18,224978823717473805791598353,
0x0ef1b8a0e726fc3948e15b23993015eb1627f210,1LP-ETH-1INCH,1inch Liquidity Pool (ETH-1INCH),18,5570184586260896399765500,
0xd1e5b0ff1287aa9f9a268759062e4ab08b9dacbe,UD,.crypto,,,
0x3a3a65aab0dd2a17e3f1947ba16138cd37d08c04,aETH,Aave Interest bearing ETH,18,434641581879893569263551,
0x028171bca77440897b824ca71d1c56cac55b68a3,aDAI,Aave interest bearing DAI,18,101794357661929117396208158,
0x9e15ad979919bb4db331bfe864475ae3bffeba93,sOM,Staked OM,18,90573401262663357758930627277911,
0xe5a3229ccb22b6484594973a03a3851dcd948756,RAE,RAE Token,18,5201456558082379540819985,
0xc9d7312a439539b5cd6b7fdacf76821100605b5d,DFI,DynamicDollar.Finance,18,171861450000000000000000,
0xe0ad1806fd3e7edf6ff52fdb822432e847411033,ONX,OnX.finance,18,5434304284933013661864813,
0x7e7e112a68d8d2e221e11047a72ffc1065c38e1a,bDIGG,Badger Sett Digg,18,514477655553346840680,
0xb4db55a20e0624edd82a0cf356e3488b4669bd27,1LP-ETH-USDC,1inch Liquidity Pool (ETH-USDC),18,18358877850186494671466,
0x8798249c2e607446efb7ad49ec89dd1865ff4272,xSUSHI,SushiBar,18,40527685876378786943612831,

And just a few errors

2021-03-11 17:10:21,943 - eth_token_service [ERROR] - An exception occurred in function symbol of contract 0x9f8F72aA9304c8B593d555F12eF6589cC3A579A2. This exception can be safely ignored.
2021-03-11 17:10:22,101 - eth_token_service [ERROR] - An exception occurred in function SYMBOL of contract 0x9f8F72aA9304c8B593d555F12eF6589cC3A579A2. This exception can be safely ignored.
2021-03-11 17:10:22,250 - eth_token_service [ERROR] - An exception occurred in function name of contract 0x9f8F72aA9304c8B593d555F12eF6589cC3A579A2. This exception can be safely ignored.
2021-03-11 17:10:22,389 - eth_token_service [ERROR] - An exception occurred in function NAME of contract 0x9f8F72aA9304c8B593d555F12eF6589cC3A579A2. This exception can be safely ignored.
2021-03-11 17:10:24,545 - eth_token_service [ERROR] - An exception occurred in function symbol of contract 0x89d24A6b4CcB1B6fAA2625fE562bDD9a23260359. This exception can be safely ignored.
2021-03-11 17:10:24,745 - eth_token_service [ERROR] - An exception occurred in function SYMBOL of contract 0x89d24A6b4CcB1B6fAA2625fE562bDD9a23260359. This exception can be safely ignored.
2021-03-11 17:10:24,899 - eth_token_service [ERROR] - An exception occurred in function name of contract 0x89d24A6b4CcB1B6fAA2625fE562bDD9a23260359. This exception can be safely ignored.
2021-03-11 17:10:25,067 - eth_token_service [ERROR] - An exception occurred in function NAME of contract 0x89d24A6b4CcB1B6fAA2625fE562bDD9a23260359. This exception can be safely ignored.
2021-03-11 17:10:26,405 - eth_token_service [ERROR] - An exception occurred in function decimals of contract 0x629A673A8242c2AC4B7B8C5D8735fbeac21A6205. This exception can be safely ignored.
2021-03-11 17:10:26,643 - eth_token_service [ERROR] - An exception occurred in function DECIMALS of contract 0x629A673A8242c2AC4B7B8C5D8735fbeac21A6205. This exception can be safely ignored.
2021-03-11 17:10:26,939 - eth_token_service [ERROR] - An exception occurred in function decimals of contract 0xD1E5b0FF1287aA9f9A268759062E4Ab08b9Dacbe. This exception can be safely ignored.
2021-03-11 17:10:27,092 - eth_token_service [ERROR] - An exception occurred in function DECIMALS of contract 0xD1E5b0FF1287aA9f9A268759062E4Ab08b9Dacbe. This exception can be safely ignored.
2021-03-11 17:10:27,225 - eth_token_service [ERROR] - An exception occurred in function totalSupply of contract 0xD1E5b0FF1287aA9f9A268759062E4Ab08b9Dacbe. This exception can be safely ignored.
2021-03-11 17:10:28,311 - ProgressLogger [INFO] - Finished work. Total items processed: 52. Took 0:00:07.246757.
2021-03-11 17:10:28,313 - CompositeItemExporter [INFO] - token items exported: 52

The cause of this errors are two things.

  1. 0x9f8F72aA9304c8B593d555F12eF6589cC3A579A2 and 0x89d24A6b4CcB1B6fAA2625fE562bDD9a23260359 are using byte32 instead of string in the name and symbol methods. This can be handle easily by just doing:
def normalize_text(str):
    return w3.toText(str) if str.find("0x") > -1 else str

print(normalize_text("0x44616920537461626c65636f696e2076312e3000000000000000000000000000"))
// Dai Stablecoin v1.0
print(normalize_text("AAVE"))
// AAVE
  1. 0x629A673A8242c2AC4B7B8C5D8735fbeac21A6205 and 0xD1E5b0FF1287aA9f9A268759062E4Ab08b9Dacbe are just NFT using the ERC165. Not having decimals or total supply makes sense, so we could just throw errors and keep the column empty

What do you think? I think it could be worth it

Thanks, Alejandro

medvedev1088 commented 3 years ago

I like this idea, it would increase the coverage for tokens that return byte32 instead of string for symbol and name.

I listed some other cases when token detection https://github.com/blockchain-etl/ethereum-etl/blob/develop/ethereumetl/jobs/export_contracts_job.py#L82 doesn't work https://github.com/blockchain-etl/ethereum-etl/issues/195