network: remove handshake port check against socket port

[thibault-martinez]

The handshake port should be checked against the ports in the list of neighbors, not the socket port which is random.