robgjansen
commented
2 months ago What exactly is the "exit Transfer Error Rate" in the plot output by tornettools? In the "Once is Never Enough" paper (link), it says it's the fraction of failed over attempted downloads. But I don't know what those kinds of downloads are. Are they downloads that never succeeded, are they downloads that only succeeded after trying different circuits, or something else?
The error rates are computed using tgen client data, specifically, the "perf" clients that download 50KB, 1MB, and 5MB files. These clients are added to the simulation so we can track how the network overall is performing. From the client perspective, it starts a download and either the download succeeds, or it fails. Partial downloads count as failures. Timeouts count as failures. Whatever happens inside the Tor circuits to cause failures is orthogonal.
Here is the code that computes the errors if you want the nitty gritty details: https://github.com/shadow/tornettools/blob/9716a8682483f363e4bc9f9503f6871536e7b846/tornettools/parse_tgen.py#L159
I'm also looking at a plot to compare two simulations. In one simulation, I let some Tor clients run additional programs to try to write sensible values to the v3bw bandwidth files, at a certain frequency throughout the simulation. In the other, I added a program that writes random values to the bandwidth files at a certain frequency. In the plot, the two simulations show similar "exit Client Transfer Goodput". But the one with random values shows a higher error rate. Does that mean the one with random values have a poorer performance? And is the goodput computed only among downloads that succeeded, or among all downloads?
Yes your intuition here is correct. The Client Transfer Goodput metric is computed among succeeded 5 MB downloads, following the Tor Project's way of calculating it (from the metrics.torproject.org site). In your "bad" simulation (where you set random weights), probably many downloads are failing so yes that means much worse performance in general. But some downloads get lucky and find a fast enough circuit to succeed, and for those we compute the Client Transfer Goodput. This is probably why this metrics looks similar across your experiments.
You are of course welcome to change the tornettools parsing scripts if you'd prefer to compute the metrics differently, or to compute different metrics. These are just the ones we found useful, but who are we to tell you how to do your research!? :)
higher-order
What exactly is the "exit Transfer Error Rate" in the plot output by tornettools? In the "Once is Never Enough" paper (link), it says it's the fraction of failed over attempted downloads. But I don't know what those kinds of downloads are. Are they downloads that never succeeded, are they downloads that only succeeded after trying different circuits, or something else?
I'm also looking at a plot to compare two simulations. In one simulation, I let some Tor clients run additional programs to try to write sensible values to the v3bw bandwidth files, at a certain frequency throughout the simulation. In the other, I added a program that writes random values to the bandwidth files at a certain frequency. In the plot, the two simulations show similar "exit Client Transfer Goodput". But the one with random values shows a higher error rate. Does that mean the one with random values have a poorer performance? And is the goodput computed only among downloads that succeeded, or among all downloads?
I've attached the plot. The sensible simulation is called "orig2crowd", and the one with random values is called "dynamic random (failed)".
tornet.plot.pages.pdf