bennlich
commented
5 years ago This is awesome @thedig ! You are a speed demon :)
It just occurred to me that calculating the totals in a live scenario is not quite as simple as our tests would make it seem. The complication is that each exitnode publishes its route table information whenever it wants (more or less every 2 minutes). This means that the timestamps of the logs from one exitnode are not guaranteed to match the timestamps of the logs from any other exitnode (in fact they almost certainly will not match).
Running the simulate-activity.js script once does not make this obvious, but if you comment out one of the simulated posts (as though one of the exitnodes skipped a beat), you'll see an error on the server-side. (I'll open an issue about making simulate-activity.js a little more robust.)
I can think of two ways to deal with this timestamp issue. Let's say the logs look something like this:
| timestamps | 2 | 6 | 8 | 11 |
|---|---|---|---|---|
| exitnode reporting | A | B | A | B |
| route table | 10.0.0.1, 10.0.0.2 | 10.0.0.3 | 10.0.0.1 | 10.0.0.2, 10.0.0.3 |
Method 1: Keep a running total of nodes as you move through time. So the resulting totals would be:
| timestamps | 2 | 6 | 8 | 11 |
|---|---|---|---|---|
| unique nodes | 10.0.0.1, 10.0.0.2 | 10.0.0.1, 10.0.0.2, 10.0.0.3 | 10.0.0.1, 10.0.0.3 | 10.0.0.1, 10.0.0.2, 10.0.0.3 |
| totals | 2 | 3 | 2 | 3 |
Method 2: Bin our logs in increments of, say, 5 minutes. So the above logs would get reduced to a two data points:
| timestamps | 2-7 | 7-12 |
|---|---|---|
| unique nodes | 10.0.0.1, 10.0.0.2, 10.0.0.3 | 10.0.0.1, 10.0.0.2, 10.0.0.3 |
| totals | 3 | 3 |
I think method 2 might be a simpler implementation. It loses some granularity, but I think that's okay. Might even able to do it with a single mongo query.
Let me know what you think! Or if any part of this is unclear. Maybe there's a simpler way to deal with this? In general I think implementations that are simpler and easier to understand are better, even if they're not quite as correct.
(I definitely did not foresee this problem when I wrote the issue...)
thedig
For #35