uint256[] Node.GetMempool() is not able to download more than 50 000 transaction hashes

[vonzepp]

Current Bitcoin Mempool size is about 80 000 Tx, but I can't get more than 50 000 from single Node. Am I right that problem is in InvPayload.cs? : https://github.com/MetacoSA/NBitcoin/blob/master/NBitcoin/Protocol/Payloads/InvPayload.cs#L52

        public override void ReadWriteCore(BitcoinStream stream)
        {
            var old = stream.MaxArraySize;
            stream.MaxArraySize = 50000;

[NicolasDorier]

I think this inv limit is enforced on bitcoin core side, if it is above, it should be broken in smaller inv payload, so I don't think it is that. I think the bug was related to https://github.com/MetacoSA/NBitcoin/issues/250 .

[vonzepp]

It is definetely not related to #250 . Look at my log. I requested GetMempool() from 12 nodes at the same time. All of them returned 50 000. It seems that each node don't sort them before sending. So I was able to get 76451 unique transaction identifiers after mixing received results:

10.05.2017 03:13:01.0685161 [7] [Info] [SPVClient] Trying to get mempool headers from 12 nodes
10.05.2017 03:13:01.0895173 [36] [Debug] [SPVClient] Downloaded 1 Ids of Tx from mempool in 00:00:00.0164111 from node HandShaked ([::ffff:82.116.203.240]:8333)
10.05.2017 03:13:02.9566241 [21] [Debug] [SPVClient] Downloaded 50000 Ids of Tx from mempool in 00:00:01.8862218 from node HandShaked ([::ffff:91.205.176.56]:8333)
10.05.2017 03:13:03.1366344 [22] [Debug] [SPVClient] Downloaded 50000 Ids of Tx from mempool in 00:00:02.0633999 from node HandShaked ([::ffff:178.162.198.45]:37295)
10.05.2017 03:13:03.1486351 [25] [Debug] [SPVClient] Downloaded 50000 Ids of Tx from mempool in 00:00:02.0778822 from node HandShaked ([::ffff:94.211.248.93]:8333)
10.05.2017 03:13:07.3588759 [8] [Debug] [SPVClient] Downloaded 50000 Ids of Tx from mempool in 00:00:06.2851329 from node HandShaked ([::ffff:173.212.202.33]:8337)
10.05.2017 03:13:08.5919464 [36] [Debug] [SPVClient] Downloaded 50000 Ids of Tx from mempool in 00:00:07.5015405 from node HandShaked ([::ffff:99.52.182.51]:8333)
10.05.2017 03:13:09.6100047 [38] [Debug] [SPVClient] Downloaded 50000 Ids of Tx from mempool in 00:00:08.5357925 from node HandShaked ([::ffff:45.23.235.104]:8333)
10.05.2017 03:13:13.1662081 [19] [Debug] [SPVClient] Downloaded 50000 Ids of Tx from mempool in 00:00:12.0906976 from node HandShaked ([::ffff:50.175.65.65]:8333)
10.05.2017 03:13:16.5824035 [39] [Debug] [SPVClient] Downloaded 50000 Ids of Tx from mempool in 00:00:15.5087478 from node HandShaked ([::ffff:197.242.156.160]:8333)
10.05.2017 03:13:23.8168173 [40] [Debug] [SPVClient] Downloaded 50000 Ids of Tx from mempool in 00:00:22.7466883 from node HandShaked ([::ffff:104.199.184.15]:8333)
10.05.2017 03:13:25.7009250 [52] [Debug] [SPVClient] Downloaded 50000 Ids of Tx from mempool in 00:00:23.5995308 from node HandShaked ([::ffff:112.74.76.17]:8333)
10.05.2017 03:13:27.7630430 [10] [Debug] [SPVClient] Downloaded 50000 Ids of Tx from mempool in 00:00:26.6890308 from node HandShaked ([::ffff:104.236.208.7]:8333)
10.05.2017 03:13:27.8940505 [7] [Info] [SPVClient] Network mempool contains 76451 transactions

It is close to current mempool size: https://blockchain.info/ru/charts/mempool-size

[vonzepp]

My code to download and mix mempool from different nodes:

        private uint256 [] GetAllTransactionIdsInMempool()
        {
            logger.Info($"Trying to get mempool headers from {group.ConnectedNodes.Count} nodes");

            var cts = new CancellationTokenSource();
            var poolTxIdentifiers = new ConcurrentQueue<uint256[]>();

            var tasks = group.ConnectedNodes.Select(node => Task.Run(() =>
            {
                if (!node.IsConnected) return;
                logger.Debug($"Trying to download mempool headers from node {node}...");

                try
                {
                    var sw = Stopwatch.StartNew();
                    var ids = node.GetMempool(cts.Token);
                    sw.Stop();
                    logger.Debug($"Downloaded {ids.Length} Ids of Tx from mempool in {sw.Elapsed} from node {node}");
                    poolTxIdentifiers.Enqueue(ids);
                }
                catch (OperationCanceledException)
                {
                    logger.Warn($"Failed to get mempool headers from node {node}. Operation was canceled");
                }
                catch (Exception ex)
                {
                    logger.Error($"Failed to get mempool headers from node {node}: {ex}");
                }
            })).ToArray();

            bool allTasksCompleted;
            try
            {
                allTasksCompleted = Task.WaitAll(tasks, (int)config.MaxTimeOfSingleRequest.TotalMilliseconds, pullingTreadsCTS.Token);
            }
            catch (OperationCanceledException)
            {
                allTasksCompleted = false;
            }

            if (!allTasksCompleted)
            {
                logger.Warn($"{tasks.Count(t => !t.IsCompleted)} tasks hangs");
                cts.Cancel();
            }

            var allIdentifiers = poolTxIdentifiers.SelectMany(ids => ids);
            return new HashSet<uint256>(allIdentifiers).ToArray();
        }

[vonzepp]

https://github.com/bitcoin/bitcoin/blob/master/src/net.h#L53 @NicolasDorier , Is it a correct server side limitation which was mentioned here:

I think this inv limit is enforced on bitcoin core side

If yes, can you suggest any workaround to be sure that client was downloaded all transactions from the mempool ?

[NicolasDorier]

mmh, GetMempool seems faulty.

        public uint256[] GetMempool(CancellationToken cancellationToken = default(CancellationToken))
        {
            AssertState(NodeState.HandShaked);
            using(var listener = CreateListener().OfType<InvPayload>())
            {
                this.SendMessageAsync(new MempoolPayload());
                return listener.ReceivePayload<InvPayload>(cancellationToken).Inventory.Select(i => i.Hash).ToArray();
            }
        }

Try to make your own, which receive several InvPayload like (pseudo code):

        public uint256[] GetMempool(Node node, CancellationToken cancellationToken = default(CancellationToken))
        {
            AssertState(NodeState.HandShaked);
            using(var listener = node.CreateListener().OfType<InvPayload>())
            {
                node.SendMessageAsync(new MempoolPayload());
                var invs = listener.ReceivePayload<InvPayload>(cancellationToken).Inventory.Select(i => i.Hash).ToArray();
                               var result = invs.ToList();
                               while(invs.Length == 50000)
                               {
                           invs = listener.ReceivePayload<InvPayload>(cancellationToken).Inventory.Select(i => i.Hash).ToArray();
                            result.AddRange(invs);
                                    //......
                               }
                             return result;                  
            }
        }

If it works better, I will update GetMempool

[vonzepp]

Thanks! I'll try this soon.

[NicolasDorier]

@vonzepp I just checked in bitcoin core source code, this is effectively the issue.

[NicolasDorier]

@vonzepp pushed version 3.0.2.22 it should be fixed. Try again with it and close the issue if it worked.

[vonzepp]

It works in 3.0.2.22, Thank you very much!

10.05.2017 17:44:36.0418808 [8] [Debug] [SPVClient] Downloaded 107241 Ids of Tx from mempool in 00:00:04.0130077 from node HandShaked ([::ffff:5.199.130.228]:8333)
10.05.2017 17:44:38.2440067 [7] [Debug] [SPVClient] Downloaded 64213 Ids of Tx from mempool in 00:00:06.2162502 from node HandShaked ([::ffff:176.14.240.103]:8333)
10.05.2017 17:44:40.1191140 [21] [Debug] [SPVClient] Downloaded 107059 Ids of Tx from mempool in 00:00:08.0900632 from node HandShaked ([::ffff:5.102.205.240]:8333)