[Part 1/2: Introduction] Functional tests for v2 P2P encryption #24748

[stratospher]

Session Details

• Date: 16-11-2023
• Time: IST 20:00 (UTC 14:30)
• PR: https://github.com/bitcoin/bitcoin/pull/24748
• Context: p2p test
• Difficulty: intermediate
• Language: python

Learning

This is the introductory session for a 2-part review club. The review club will focus on how the functional test framework works on current master and will not cover the changes in PR #24748. If this is your first exposure to the test framework, it is recommended to understand the basics discussed in this review club. It's a definite prerequisite to understanding PR #24748.

• Functional test framework

• P2P network

  Note: Drop your PR and review related suggestions/thoughts/questions below

Summary

• Bitcoin's functional test framework is a cool tool to spin up nodes and simulate your own network scenario in the regtest.
• Each node would be an actual bitcoind instance and managed by the python class TestNode.
• However we can't customise behaviour of bitcoind. Developers often want to be able to send their own custom response, introduce malicious behaviour and check if error messages were reported, pause and examine bitcoind process etc. The P2PInterface class serves this purpose and mocks the behaviour of a peer. It can be easily customised.
• You can connect 2 bitcoind nodes(TestNode <--> TestNode) using connect_nodes which invokes the addnode RPC.
• P2PInterface allows custom interaction. You can create inbound/outbound connections to the TestNode from your mockable peer.
  1. Inbound connections to the TestNode are made using add_p2p_connection()
     • TestNode <---------- P2PInterface
     • Here, P2PInterface is the initiator of the connection
  2. Outbound connections from the TestNode are made using add_outbound_p2p_connection()
     • TestNode ----------> P2PInterface
     • Here, TestNode is the initiator of the connection

Questions

1. What does this PR do at a high level? Why do we need this PR?
2. Can you trace the functions through which an inbound connection is established in the test framework?
   • hint: see how TestNode::add_p2p_connection makes a call to peer_connect
3. Why is the mechanism to establish an outbound connection in the test framework different? Can you trace the functions through which an outbound connection is established in the test framework?
   • hint: see how TestNode::add_outbound_p2p_connection makes a call to peer_accept_connection
4. How are P2P messages read? How are P2P messages sent?
   • hint: see data_received() and send_message()
5. What is a version handshake? What tools can you use to observe different messages that are exchanged.
6. Briefly explain initial v2 handshake in BIP 324 and when it would need to be done so that P2P messages exchanged are encrypted.

Extra reading

1. Amiti's onboarding exercise on version handshake
2. Detailed writeup about working of the functional test framework

[stratospher]

Summary

Overview

• Unit tests test a single function in isolation. Functional tests test the interaction of many functions in a system with multiple layers of the stack (network, network processing, validation). ex: P2P and RPC behaviour
• Functional tests can be found in test/functional/*
• They all have the same basic structure - lot of imports, subclassing BitcoinTestFramework (base class for all functional tests), writing test parameters in set_test_params() and writing the test in run_test()
  See very simple code demonstrating components in the functional tests for explanation purpose

  
  

#!/usr/bin/env python3
# Copyright (c) 2021-2022 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.

from test_framework.p2p import P2PInterface
from test_framework.test_framework import BitcoinTestFramework
from test_framework.util import assert_greater_than

class FunInterface(P2PInterface):
    def on_version(self, message):
        super().on_version(message)
        print(message)

class FunTest(BitcoinTestFramework):
    def set_test_params(self):
        self.num_nodes = 2

    def setup_network(self):
        self.setup_nodes()
        # Don't connect the nodes

    def run_test(self):
        # info1 = self.nodes[0].getpeerinfo()
        # print(info1)
        # self.connect_nodes(0, 1)
        # print("does anything change")
        # info2 = self.nodes[0].getpeerinfo()
        # for keys in info2[0]:
        #     print(keys, info2[0][keys])
        peer_0 = self.nodes[0].add_p2p_connection(FunInterface())
        info1 = self.nodes[0].getpeerinfo()
        assert_greater_than(info1[0]['bytesrecv_per_msg']['version'], 0)
        # assert False

if __name__ == "__main__":
    FunTest().main()

• Difference between TestNode and P2PInterface:
  1. TestNode => class which manages the bitcoind process(start/stop node, manages node's data directory, log file)
  2. P2PInterface => Python class which can speak the bitcoin protocol so that we can mock the behaviour of a peer. It's not an actual node but a dummy implementation of how a node's peer would behave.

1. What does this PR do at a high level? Why do we need this PR?

BIP 324 was recently merged and nodes can now send encrypted P2P messages to each other if they run the latest client. This PR adds support for BIP 324 to the dummy implementation of peer in python so that the python code can also talk to bitcoind using the encrypted protocol. We need this to test BIP 324 code (things like whether we're communicating on encrypted/unencrypted protocol, backward compatible with unencrypted protocol etc..) and also to confirm that existing tests work fine and behave as expected.

2. Can you trace the functions through which an inbound connection is established in the test framework? hint: see how TestNode::add_p2p_connection makes a call to peer_connect

• TestNode::add_p2p_connection makes a call to P2PInterface::peer_connect() which makes a call to P2PConnection::peer_connect()
• inside peer_connect there's a network event loop which is a big smart for loop and it runs coroutines one after another.
• coroutines are subroutines which can be paused and continued as needed (ex: you need something from network, pause and save the context in a local variable. it will wait until network result is ready and when it is ready, it is going to continue running having the same context as before). loop.create_connection is a coroutine that returns as soon as it has created a transport/protocol pair.

3. Why is the mechanism to establish an outbound connection in the test framework different? Can you trace the functions through which an outbound connection is established in the test framework? hint: see how TestNode::add_outbound_p2p_connection makes a call to peer_accept_connection

• In inbound connection, we already have a running bitcoind process. our P2PInterface can just open a TCP connection to it. In outbound connection, we need to have a running P2PInterface process, so that bitcoind had something to connect to. That's why the mechanisms differ.
• add connection RPC

4. How are P2P messages read? How are P2P messages sent? hint: see data_received() and send_message()

• P2PConnection implements low-level network operations, such as opening and closing the TCP connection to the node and reading bytes from and writing bytes to the socket.
• Callback style reading is done because you don't want to block waiting for your data to be ready. You just want a callback when you get some data.
• data_received()
• send_message()

5. What is a version handshake? why do we need it? What tools can you use to observe different messages that are exchanged.

• version handshake: after the node has identified a peer it wants to connect to, it sends a version message to the peer and waits for the peer to send a version message and a verack message back to it. It then sends a verack back to the peer to establish the connection.

• significance:

  1. each node can get information about the other node like its version number, block and current time.
  2. it's how a reliable connection can be established.(similar to how TCP-handshake establishes TCP connection) Only after the version handshake is over can the peers accept other messages.
  3. service advertising - the version message would contain the service bits a peer advertises and hence our node would know the services the peer supports.

• Tools to observe different messages - stats from getpeerinfo (bytesrecv_per_msg), test/functional/combine_logs.py

  6. Briefly explain initial v2 handshake in BIP 324 and when it would need to be done so that P2P messages exchanged are encrypted.

• suppose node1(initiator) wants to connect to node2(responder)

• In v1 protocol, node1 starts the connection by sending a version message and performing the version handshake described above.

• In v2 protocol, we want even these version messages to be encrypted

• so both node1 and node2 need to have the same keys which nobody else knows about so that they can encrypt/decrypt the P2P messages

• In v2 protocol, a more advanced form of Diffie-Hellman handshake takes place so that the initiator and responder arrive at the same key stream. Then both of them can use this keystream to encrypt/decrypt P2P messages.