The isUserEthereumAddressInBloom is totally not working

[jackykwandesign]

This is a test block, Eth mainnet block 11722493 https://etherscan.io/block/11722493

{
    number: 11722493,
    hash: '0xba24ea487322f4c2db4e0c0b415cf4890bb3d3aa4d7082d5eb578b4e61498b68',
    parentHash: '0x4216964047968bb407a8467023620f57fe217fd5ceb4b55dfa2ef1029d4db54f',
    sha3Uncles: '0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347',
    logsBloom: '0x15ac582820ce219071f520ac98b0fc458059117dac1099491493a45ad0e48b81051e214c658816f457107a028c1a8b5d862ee083d93a2aa76c8db1b0baee4259594c408aa112a96e4dc3cc8cd3f9c76698464147114614138cc61c48da4b5551fd4089951a304361134dd841636d9cea4f4d1328442a1fd10a7e7f3a122c5da059b9d9068b682684ddeb48140206a4170c0e8c9f85383629e126bce1869e91b68329681bfe59f3027e85f2f4ba11091902458c41882a59402278a30911288256b950a063253e8ab4144c4d72b3bbac8f0465df4b574b53dae81988474c48a0a51eb86810baa2c0458c501df0f1b85300952c00d205cf2c7108ec8676039256fa',
    transactionsRoot: '0xd7c55db05b182757e28de60e05be60ecb21709c552e5e63ad2b6ef1424c32538',
    stateRoot: '0xe9210552072b6f9daa121bf9b8d82f3d71b8795d65287ae640685da9f81c8886',
    receiptsRoot: '0x162589db53c73278738eb28d2614f6aa647fbcc6fc36e69b5ec0840823a5025a',
    miner: '0x829BD824B016326A401d083B33D092293333A830',
    difficulty: '4343007843365264',
    extraData: '0x7070796520e4b883e5bda9e7a59ee4bb99e9b1bc0006',
    gasLimit: 12457317,
    gasUsed: 12448301,
    timestamp: 1611546765,
    nonce: '0xe77b5f1cbb0ded96',
    mixHash: '0x9e64c804e02560f77091372b551b672a4759b633623a8f15b47f3b7d168c8977',
    size: undefined
  }

The bloom is 0x15ac582820ce219071f520ac98b0fc458059117dac1099491493a45ad0e48b81051e214c658816f457107a028c1a8b5d862ee083d93a2aa76c8db1b0baee4259594c408aa112a96e4dc3cc8cd3f9c76698464147114614138cc61c48da4b5551fd4089951a304361134dd841636d9cea4f4d1328442a1fd10a7e7f3a122c5da059b9d9068b682684ddeb48140206a4170c0e8c9f85383629e126bce1869e91b68329681bfe59f3027e85f2f4ba11091902458c41882a59402278a30911288256b950a063253e8ab4144c4d72b3bbac8f0465df4b574b53dae81988474c48a0a51eb86810baa2c0458c501df0f1b85300952c00d205cf2c7108ec8676039256fa

Sample data 1 TX: 0xb40818abfc6a5f460922e107a108a49a041d48d29b78f914ae35ffeeccea566c Address: 0xdac17f958d2ee523a2206206994597c13d831ec7 (USDT token contract address)

Sample data 2 TX: 0xbbbfaa43ceaf7e6472ba41ffccea878027d330418d435551f206128557bc527b Address: 0x8d1f2cd09a96701a074783841be8d4ec97ec5f0c (simple address)

simple code for the case

        const activeAddress = ['0xdac17f958d2ee523a2206206994597c13d831ec7','0x8d1f2cd09a96701a074783841be8d4ec97ec5f0c']
        let bloom = '0x15ac582820ce219071f520ac98b0fc458059117dac1099491493a45ad0e48b81051e214c658816f457107a028c1a8b5d862ee083d93a2aa76c8db1b0baee4259594c408aa112a96e4dc3cc8cd3f9c76698464147114614138cc61c48da4b5551fd4089951a304361134dd841636d9cea4f4d1328442a1fd10a7e7f3a122c5da059b9d9068b682684ddeb48140206a4170c0e8c9f85383629e126bce1869e91b68329681bfe59f3027e85f2f4ba11091902458c41882a59402278a30911288256b950a063253e8ab4144c4d72b3bbac8f0465df4b574b53dae81988474c48a0a51eb86810baa2c0458c501df0f1b85300952c00d205cf2c7108ec8676039256fa'
        console.log("isBloom", isBloom(bloom))
        console.log("isUserEthereumAddressInBloom activeAddress 0", isUserEthereumAddressInBloom(bloom, activeAddress[0]))
        console.log("isUserEthereumAddressInBloom activeAddress 1", isUserEthereumAddressInBloom(bloom, activeAddress[1]))

Result:

isBloom true
isUserEthereumAddressInBloom activeAddress 0 true
isUserEthereumAddressInBloom activeAddress 1 false

So is that my understanding of isUserEthereumAddressInBloom wrong or code mistake ?

[joshstevens19]

Hey, thanks a lot for raising the issue.

So your last transaction is not a contract call its just sending eth which does not emit logs so not queryable in blooms. This issue has made me write a little section on this to explain this to users (so thanks):

Blooms do not work with eth transactions (purely sending eth), eth transactions do not emit logs so do not exist in the bloom filter. This is what ethereum did purposely but it means you should query the eth balance every block to make sure it's in sync. Blooms will only work if the transaction emits an event which then ends up in the logs. The bloom filter is there to help you find logs. A contract can be written which does not emit an event and in that case, would not be queryable from a bloom filter. The erc20 token spec requires you to fire an event on approval and transfer so blooms will work for approval and transfer for ALL erc20 tokens, this will be most people's primary use-case. Saying that this can be used in any way you want with any use-case as long as events are emitted then it's queryable.

You can now see that section in the readme https://github.com/joshstevens19/ethereum-bloom-filters#requirements-for-blooms-to-be-queryable

Thanks!

[jackykwandesign]

thanks for your reply, great explanation.

I did some research today and find out that there is no shortcut for pure eth transaction, which my job need to. Since my use case is to monitor a HD wallet with ETH, USDT and BTC, which is EOA and not supported by most of monitor service

So i left a method here for anyone come want to know how to deal with pure ETH.

The old fashion way,

1. listen to new block
2. When new block come, get latest active address list from DB and buffer to RAM
3. Query block with all tx details from Infura
4. Decode and extract only ETH or USDT transaction
5. compare to Active address list in RAM, check if exist, if yes, log in down and do further business logic

The main time consume job is Infura call and DB call, each infura call require 1-2 sec and DB call require 1 sec. So normally you have 12 - 20 secs to process each block, should be enough since only < 200 transactions @ block Each day around 6500 ETH block generated so a simple Infura free core is enough (100000 Requests @ day)

[joshstevens19]

@jackykwandesign What I would do if I was you would mix it up with both. We do this on our wallet and it works really well.

1. Listen to new blocks coming in (as you said 12-17 seconds normally)
2. Each block use the blooms for your USDT balance > if true query that contract balance and update it (not wasting node infura calls if it's false)
3. Just call eth_getBalance (JSONRPC request) on every new block which comes in with the address of the user's eth address (which you must know)
4. if the eth balance has changed with what you have already got in the state (from the last block) update it, if not keep the same
5. call BTC balance API updates every 1 minute (whatever API you're using for that)
6. keep going from step 1 to step 6 over and over again. This is the most lightweight less queryable approach for your solution.

From experience, this flow works very nicely and is not intense.

With 100000 requests a day that's 4166.66666667 an hour so 69.4444444444 a minute, say a client does 4 a minute this approach could handle 17.3 clients running at the same time constantly for 24 hours. So to make it even less intense you could only call eth_balance every 1 minute, as your USDT will be using blooms and only querying when it needs to. This would mean you're probably going to be able to run 3x 17.3 clients more like 52 clients running constantly for 24 hours without hitting a paid subscription (doubt you ever hit that as clients don't run for 24 hours ever really, user sessions are 1 hour or so). If you did not do blooms for USDT the maths above would be much much more (double) and you only be able to run 8.6 clients for the free plan running at the same time constantly for 24 hours.

Hope it helps! Josh

[joshstevens19]

Also have a look at https://github.com/joshstevens19/ethereum-multicall can group many json rpc calls in 1 to save you even more credits.