mattmathis
commented
2 years ago I propose that we support general server side NAT style load balancing by having the client and server annotations carry the actual public side IP addresses and ports, (such that the 4 fields in the annotations exactly mach IP headers as seen on the public wire). There should also be a 4 separate flags indicating which fields were re written. The archived data (TCPinfo, pcap, NDT records, etc) should all be archived as seen by the server (i.e. with the private addresses and ports). After the annotation join, the data in BQ should reflect the addresses and ports from the annotations and not the raw data. The flags would indicate which fields get overwritten.
Search "dynamic NAT load balancing" and "layer 4 load balancing" for many tutorials.
As an added benefit we potentially enable a low cost solution to the "/26 problem" by replacing our switches with L4 load balancers that distribute incoming requests (to a small number of IP address) to multiple servers.
Note that ICMP is trickey. and traceroute will require careful design considerations and testing.
@mattmathis reported that the
server.*annotation records for measurements to the new cloud nodes are empty in the sandbox tables.I've confirmed that siteinfo annotations https://siteinfo.mlab-sandbox.measurementlab.net/v1/sites/annotations.json have records for the cloud node regions and that the uuid-annotator is in fact writing JSON files with empty server records.
The Cloud VM public IP (and the /32 associated with the node) is different from the private IP known inside the VM. So, when the uuid-annotator checks that the given server IP is within the configured netblock for this site it always fails.
See: https://github.com/m-lab/uuid-annotator/blob/master/siteannotator/server.go#L67
Perhaps this condition is optional and can be removed.