Ndt7: Add a way to measure latency

[EKami]

Hi everyone, Is there a way to measure latency on ndt7 servers? So far I can only get results like this one:

    {
      "machine": "mlab1-bom02.mlab-oti.measurement-lab.org",
      "location": {
        "city": "Mumbai",
        "country": "IN"
      },
      "urls": {
        "ws:///ndt/v7/download": "ws://ndt-mlab1-bom02.mlab-oti.measurement-lab.org/ndt/v7/download?access_token=eyJhbGciOiJFZERTQSIsImtpZCI6ImxvY2F0ZV8yMDIwMDQwOSJ9.eyJhdWQiOlsibWxhYjEtYm9tMDIubWxhYi1vdGkubWVhc3VyZW1lbnQtbGFiLm9yZyIsIm1sYWIxLmJvbTAyLm1lYXN1cmVtZW50LWxhYi5vcmciXSwiZXhwIjoxNjAxMzU4OTMwLCJpc3MiOiJsb2NhdGUiLCJzdWIiOiJuZHQifQ.-3QOaLvbNaZ3bUD4jDNGqP17FpR5BQp8DuC5mtQ3SZnFyE2HcYbpZSuTh3N-a1Ff9XLOoBECYgE4heZFu5T1BQ",
        "ws:///ndt/v7/upload": "ws://ndt-mlab1-bom02.mlab-oti.measurement-lab.org/ndt/v7/upload?access_token=eyJhbGciOiJFZERTQSIsImtpZCI6ImxvY2F0ZV8yMDIwMDQwOSJ9.eyJhdWQiOlsibWxhYjEtYm9tMDIubWxhYi1vdGkubWVhc3VyZW1lbnQtbGFiLm9yZyIsIm1sYWIxLmJvbTAyLm1lYXN1cmVtZW50LWxhYi5vcmciXSwiZXhwIjoxNjAxMzU4OTMwLCJpc3MiOiJsb2NhdGUiLCJzdWIiOiJuZHQifQ.-3QOaLvbNaZ3bUD4jDNGqP17FpR5BQp8DuC5mtQ3SZnFyE2HcYbpZSuTh3N-a1Ff9XLOoBECYgE4heZFu5T1BQ",
        "wss:///ndt/v7/download": "wss://ndt-mlab1-bom02.mlab-oti.measurement-lab.org/ndt/v7/download?access_token=eyJhbGciOiJFZERTQSIsImtpZCI6ImxvY2F0ZV8yMDIwMDQwOSJ9.eyJhdWQiOlsibWxhYjEtYm9tMDIubWxhYi1vdGkubWVhc3VyZW1lbnQtbGFiLm9yZyIsIm1sYWIxLmJvbTAyLm1lYXN1cmVtZW50LWxhYi5vcmciXSwiZXhwIjoxNjAxMzU4OTMwLCJpc3MiOiJsb2NhdGUiLCJzdWIiOiJuZHQifQ.-3QOaLvbNaZ3bUD4jDNGqP17FpR5BQp8DuC5mtQ3SZnFyE2HcYbpZSuTh3N-a1Ff9XLOoBECYgE4heZFu5T1BQ",
        "wss:///ndt/v7/upload": "wss://ndt-mlab1-bom02.mlab-oti.measurement-lab.org/ndt/v7/upload?access_token=eyJhbGciOiJFZERTQSIsImtpZCI6ImxvY2F0ZV8yMDIwMDQwOSJ9.eyJhdWQiOlsibWxhYjEtYm9tMDIubWxhYi1vdGkubWVhc3VyZW1lbnQtbGFiLm9yZyIsIm1sYWIxLmJvbTAyLm1lYXN1cmVtZW50LWxhYi5vcmciXSwiZXhwIjoxNjAxMzU4OTMwLCJpc3MiOiJsb2NhdGUiLCJzdWIiOiJuZHQifQ.-3QOaLvbNaZ3bUD4jDNGqP17FpR5BQp8DuC5mtQ3SZnFyE2HcYbpZSuTh3N-a1Ff9XLOoBECYgE4heZFu5T1BQ"
      }
    }

and while the ICMP protocol works on mlab1-bom02.mlab-oti.measurement-lab.org it's not something that I can use in a Javascript client where only HTTP/HTTPS/Websocket is accepted. It'd be great if it can be measured through the WebSocket on the client side since that would allow the client to also measure the jitter. So far I came up with a hacky solution of using the wss:///ndt/v7/upload to upload 1 bit of data, wait for the response and measure the time between the two, then divide by 2. It's working "okay" relative to what I get with a ping command but sometimes it's completely not accurate.

Thanks!

[laiyi-ohlsen]

@stephen-soltesz

[bassosimone]

I have started working on this and have implemented ~50% of it here: https://github.com/bassosimone/ndt-server/tree/feature/roundtrip. It should take 1-2 more weeks for me to finish it 🤞

[EKami]

Awesome news! Thank you so much! And how does it work when it comes to deployment exactly once the feature is released? Is it automatically deployed to all your NDT7 servers?

[bassosimone]

@EKami if the PR is accepted upstream, I guess it should be released in a few weeks.

[stephen-soltesz]

@EKami the method you described for the "hacky solution" using upload will not work the way you want. The messages sent by the server are randomly timed. So, there is no helpful correspondence between upload start time, time of first message received from the server, and the system to system ping times.

@EKami what is the use-case for this type of latency measurement? Application level latency measurements (as @bassosimone is proposing) may be inherently larger than the equivalent ping measurement from the same systems.

[EKami]

@stephen-soltesz yeah, I knew there was something fishy about my implementation lol. The use case is simply to be able to run this with a js client since we can't use ICMP. That would also allow me to calculate the jitter which in turn, can tell if the connection is stable or has some micro-disconnections/ is unstable.

[bassosimone]

I have written a prototype and am now evaluating it (see the above commits referencing this issue). It will take a little more time to see whether this works in reasonable way in JavaScript. If I see convincing data, then I will turn this prototype into a spec and submit it to the M-Lab core team for evaluation.

[EKami]

Awesome news, thanks a lot @bassosimone :D

[nosnilmot]

ping! (excuse the pun)

is there any prospect of this feature being implemented? @bassosimone did anything come from your implementatin/evaluation?

I'm also interested in having this feature (application level latency-only test) available.

[stephen-soltesz]

FYI: @mattmathis in case you have thoughts.