[Bug] Exception handling of Invalid ProtocolMessageTypes in 1.4.0

[dcsastry]

What happened?

I upgraded my Farmer/Wallet GUI client on my main Windows machine to 1.4.0. I noticed the harvesters (running 1.0.7) running on Ubunutu machines were no longer communicating to the Windows client.

I upgraded one of the harvesters to 1.4.0 (CLI version) and restarted the harvester daemon, but still the Windows GUI client was not seeing the harvester plots.

I thought there might be a communication problem with ports or certs, but checking the harvester log shows that instead the harvester is having trouble with understanding a message type from the Farmer. During handling of this communication problem, another code exception has hit, as the communication value is not valid for the class.

From the log, I can see that harvester bans communication to the farmer (ip 192.168.1.92) for 10 seconds, and then it reconnects but ends up with the same communication error (ProtocolMessageType value 66), and the process repeats.

Version

1.4.0

What platform are you using?

Windows

What ui mode are you using?

GUI

Relevant log output

2022-07-06T10:37:10.861 harvester harvester               : INFO     Reconnecting to peer {'host': '192.168.1.92', 'port': 8447}
2022-07-06T10:37:10.862 harvester harvester_server        : WARNING  Peer 192.168.1.92 is still banned, not connecting to it
2022-07-06T10:37:13.866 harvester harvester               : INFO     Reconnecting to peer {'host': '192.168.1.92', 'port': 8447}
2022-07-06T10:37:13.890 harvester harvester_server        : INFO     Connected with farmer {'host': '192.168.1.92', 'port': 8447}
2022-07-06T10:37:13.934 harvester chia.plotting.plot_tools: INFO     Searching directories ['/media/chiaboy/easystore/farm', '/media/fiveT1/farm', '/media/fiveT2/farm', '/media/fiveT3/farm', '/mnt/Storage/chiaboy1_farm', '/media/chiaboy/9200ssd/farm', '/media/eightT1/farm', '/media/eightT2/farm']
2022-07-06T10:37:13.968 harvester chia.plotting.plot_tools: INFO     Loaded a total of 455 plots of size 45.03706495377173 TiB, in 0.07130169868469238 seconds
2022-07-06T10:37:22.428 harvester chia.server.rate_limits : WARNING  Invalid message: 66, 66 is not a valid ProtocolMessageTypes
2022-07-06T10:37:22.431 harvester harvester_server        : ERROR    Exception: 66 is not a valid ProtocolMessageTypes <class 'ValueError'>, closing connection {'host': '192.168.1.92', 'port': 8447}. ValueError: 66 is not a valid ProtocolMessageTypes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/chiaboy/chia-blockchain/chia/server/server.py", line 493, in api_call
    f"<- {ProtocolMessageTypes(full_message.type).name} from peer "
  File "/usr/lib/python3.8/enum.py", line 309, in __call__
    return cls.__new__(cls, value)
  File "/usr/lib/python3.8/enum.py", line 600, in __new__
    raise exc
  File "/usr/lib/python3.8/enum.py", line 584, in __new__
    result = cls._missing_(value)
  File "/usr/lib/python3.8/enum.py", line 613, in _missing_
    raise ValueError("%r is not a valid %s" % (value, cls.__name__))
ValueError: 66 is not a valid ProtocolMessageTypes

2022-07-06T10:37:22.433 harvester harvester_server        : INFO     Connection closed: 192.168.1.92, node id: c1b815a0f828d96e476c0fbb2711203c3200f23d4796e415225fd9bcb7060c53
2022-07-06T10:37:22.433 harvester harvester_server        : WARNING  Banning 192.168.1.92 for 10 seconds
2022-07-06T10:37:22.433 harvester chia.harvester.harvester: INFO     peer disconnected {'host': '192.168.1.92', 'port': 8447}
2022-07-06T10:37:22.901 harvester harvester               : INFO     Reconnecting to peer {'host': '192.168.1.92', 'port': 8447}
2022-07-06T10:37:22.902 harvester harvester_server        : WARNING  Peer 192.168.1.92 is still banned, not connecting to it

[ghost]

There has been breaking changes to the harvester protocol since 1.3.4; if you are running 1.4 on your farmer you will need 1.3.4+ on all your harvesters -- recommend 1.4 on all though as many improvements and fixes.

1.3.4 notes:

Significant harvester protocol update: You will need to update your farmer and all your harvesters as this is a breaking change in the harvester protocol. The new protocol solves many scaling issues. In particular, the protocol supports sending delta changes to the farmer - so for example, adding plots to a farm results in only the new plots being reported. We recommend you update your farmer first.

1.4.0 notes:

Fixed issues with harvesters not reconnecting properly.

[dcsastry]

I am running 1.4.0 on both the Farmer system and the Harvester system.

[dcsastry]

I'm synching a 1.4.0 full node on one of my Ubuntu systems, and will try connecting the other two ubuntu harvester systems to that new full node. But it will take time to synch the full node.

[emlowe]

Protocol message type 66 was actually introduced in 1.2 - your harvester would only report it's unknown if it is running a very old version. There is no other cause.

[dcsastry]

Thank you. I did discover that when I followed the update procedure using APT, the clients did not get updated. It seemed like they updated to 1.4.0, but they did not. "chia version" still showed 1.1.7.

I use GIT fetch and checkout to get the latest version and using the install script to install. Now my harvesters are properly updated and communicating properly to the Farmer system.

Thanks for the help @emlowe !!!