search

tohojo / bufferbloat-net

The source repository for the bufferbloat.net web site
https://www.bufferbloat.net
Other
52 stars 32 forks source link

Add Cake performance report with illustrations. #29

Closed chromi closed 5 years ago

dtaht commented 7 years ago

excellent!

3 qs: I like having the flent data also. Are pngs smaller than svgs? why no direct comparison plot(s)?

On 3/6/17 3:51 PM, chromi wrote:

    You can view, comment on, or merge this pull request online at:

https://github.com/tohojo/bufferbloat-net/pull/29

    Commit Summary

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/tohojo/bufferbloat-net/pull/29, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGoilcvbntJ4UALrh33WB1wfuhsEDUUks5rjJv_gaJpZM4MU20v.

chromi commented 7 years ago

On 7 Mar, 2017, at 02:59, Dave Täht notifications@github.com wrote:

Are pngs smaller than svgs?

PNG format is what Flent produced by default. I didn’t see an obvious option to make SVGs.

I subsequently ran the PNGs through pngcrush to save a small amount of file size. Currently only the “small” versions are linked in the document.

why no direct comparison plot(s)?

Using the “add other data files” option failed every time. I don’t know why. Instead I used the “scale other data files” option, which did work for most of the sets.

chromi commented 7 years ago

I added the raw data files as well.

tohojo commented 7 years ago

chromi notifications@github.com writes:

On 7 Mar, 2017, at 02:59, Dave Täht notifications@github.com wrote:

Are pngs smaller than svgs?

PNG format is what Flent produced by default. I didn’t see an obvious option to make SVGs.

From the command line, you just do -o filename.svg and it'll produce an svg. From the GUI you can select the format when you save the plot.

why no direct comparison plot(s)?

Using the “add other data files” option failed every time. I don’t know why. Instead I used the “scale other data files” option, which did work for most of the sets.

Failed how? Did you get an error message, or did they just not show up? Could you turn on debug logging and post the log somewhere?

For multi-test plots I generally find it easier to generate them from the command line. e.g. flent tcp_1up*.flent.gz -o comparison.svg -p ping_cdf

I'd suggest also using --override-title when generating the plots to show the setup on each plot. Right now it's impossible to see which is which.

Also, a few comments on the page setup (I'll comment on the text later):

-Toke

chromi commented 7 years ago

why no direct comparison plot(s)?

Using the “add other data files” option failed every time. I don’t know why. Instead I used the “scale other data files” option, which did work for most of the sets.

Failed how? Did you get an error message, or did they just not show up? Could you turn on debug logging and post the log somewhere?

Something like “couldn’t find open files with matching something-or-other”. I’ll try again later.

I'd suggest also using --override-title when generating the plots to show the setup on each plot. Right now it's impossible to see which is which.

I did actually try this, but it had no effect whatsoever - at least in the GUI.

  • The graphs are way too small, almost unreadable on my screen.

They are deliberately scaled down so as not to completely dominate the layout. If there’s a way to make the small images into links to the full-size versions, that would be useful.

tohojo commented 7 years ago

chromi notifications@github.com writes:

I'd suggest also using --override-title when generating the plots to show the setup on each plot. Right now it's impossible to see which is which.

I did actually try this, but it had no effect whatsoever - at least in the GUI.

Yeah, this probably doesn't work in the GUI; generating the plots from the command line would be the way to go.

  • The graphs are way too small, almost unreadable on my screen.

They are deliberately scaled down so as not to completely dominate the layout. If there’s a way to make the small images into links to the full-size versions, that would be useful.

Not sure if it's doable with markdown, but you can use inline HTML to link to the full-size images.

-Toke

chromi commented 7 years ago

Hmm, the error message has changed and now the combined box plots actually work. The line plots don't:

Showing matplotlib pyplot viewer
ERROR: Aborting plotting due to error: Unrecognized marker style -
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flent-1.0.1_git_b97161b-py2.7.egg/flent/gui.py", line 1921, in get_plotter
    fig = self.async_fig.get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get
    raise self._value
ValueError: Unrecognized marker style -
/usr/lib/python2.7/dist-packages/matplotlib/artist.py:221: MatplotlibDeprecationWarning: This has been deprecated in mpl 1.5, please use the
axes property.  A removal date has not been set.
  warnings.warn(_get_axes_msg, mplDeprecation, stacklevel=1)
chromi commented 7 years ago

Aha - turns out Hugo's figure shortcode also has link functionality. I couldn't find that part of the documentation without knowing the keyword "figure" to search for. This will help.

But better would be to have working combined graphs. That would reduce the amount of editing I need to do now.

chromi commented 7 years ago

I've given most of the data files distinctive filenames locally, but Flent labels the series with their original filename in the legend of the combined box plots. I can't find any way to override that, except by decompressing and then editing the raw data files. This is really not helpful - can it be changed?

tohojo commented 7 years ago

chromi notifications@github.com writes:

I've given most of the data files distinctive filenames locally, but Flent labels the series with their original filename in the legend of the combined box plots. I can't find any way to override that, except by decompressing and then editing the raw data files. This is really not helpful - can it be changed?

--replace-legend can do search and replace, and --override-label can override legends completely.

As for the error, what's the output of flent --version, and which test and plot were you trying to create?

-Toke

dtaht commented 7 years ago

I note I have a strong preference for .svg output. They resize automatically, beautifully, on web browsers. I also would like to one day write a figure2 template that puts two graphs side by side, and a figurelink template that automagically creates the link back to the full size graph.

tohojo commented 7 years ago

Think I found the source of the crash; see if it works better now? :)

chromi commented 7 years ago

Yes, the line graphs now combine properly. I even managed to work out a bash recipe to coerce the right labels into a command-line plot.

I still think that using the current filename would be more useful than using the original one. In particular, that would make browsing the data using the GUI much easier.

chromi commented 7 years ago

Dave, I'm finding that SVGs end up being about the same size as the larger versions of the PNGs I've already generated. It's also proving harder to control their size on the page.

For now, I think I'll keep the PNGs, caption them and link to the larger versions. This is all about economy of effort, which I'm finding important right now.

However, I'll make box plot summaries as SVGs and see what I can do with them.

chromi commented 7 years ago

All of the existing graphs now have links to larger versions of themselves, and I've added summaries to each section using box-plots and CDFs. The new graphs' large versions are SVGs, while their thumbnails are still PNGs.

Linux seems to believe I have a 46" monitor - I wish - which may account for the huge size Firefox insists on displaying SVGs at on my machine.

And yes, I added a date header.

tohojo commented 7 years ago

Looks a lot better, and the captions help; I'll mess with the CSS a bit to make the figures appear nicer. Could probably even make them line up next to each other, if you add a div tag around each group of figures:

<div class="figure-group">
{{< figure.... >}}
{{< figure ... >}}
{{< figure ... >}}
</div>

I'll go over the text later; what's the 'multitraffic' test? Did you write a new Flent test? And if so, care to submit a pull request to Flent with that? :)

chromi commented 7 years ago

On 8 Mar, 2017, at 13:20, Toke Høiland-Jørgensen notifications@github.com wrote:

I'll go over the text later; what's the 'multitraffic' test? Did you write a new Flent test? And if so, care to submit a pull request to Flent with that? :)

The file is actually there in a subdirectory, if you’re interested. It isn’t entirely bug-free - I only got it working well enough to extract useful graphs from.

tohojo commented 7 years ago

Ah, excellent. Will look that over as well :)

chromi commented 7 years ago

I’m fairly keen to have this up this week, if possible, so that NLnet can see it.

dtaht commented 7 years ago

Is the 29MBs of commit? or 29K lines? Either way, could you rebase on just your final work (no need to have the pre-pngcrushed data in here) and submit a new pull request?

dtaht commented 7 years ago

OK, I pulled your commit and the pngs are FAR, FAR too small to read without squinting. in fact, they are unreadable also, after zooming. Let me look over what you have and see what I can do.

for the record, on my screen, they look like this: fortherecord

dtaht commented 7 years ago

and was it you that put in .cap files? I actually have no objection to having these, sometimes, but having them, and then deleting them, not so much. So please, unless you need those, don't put them in the rebase.

dtaht commented 7 years ago

also, .svg files usually compress down by a factor of 4 or so, but cloudflare just does that on the fly and thus I don't bother with actually storing them as svgz. Can you send us the script that is generating your images? or put up a commit that shows us what is going wrong? the .svg stuff I do on the cerowrt website works beautifully from mobiles to big screens, but I haven't a clue how toke did it.....

chromi commented 7 years ago

On 10 Mar, 2017, at 05:02, Dave Täht notifications@github.com wrote:

and was it you that put in .cap files?

Uh - what cap files?

chromi commented 7 years ago

Is the 29MBs of commit? or 29K lines? Either way, could you rebase on just your final work (no need to have the pre-pngcrushed data in here) and submit a new pull request?

I think it's counting the SVGs in the line count. I think Toke can do a "squash merge" to eliminate the unneeded edit history and save some space.

OK, I pulled your commit and the pngs are FAR, FAR too small to read without squinting. in fact, they are unreadable also, after zooming.

Did you try clicking on them, to see the larger versions? Some of them even lead to SVGs now.

tohojo commented 7 years ago

Okay, had some time to look it over.

First, the figures: I pushed some CSS changes that styles the figures better. This also adds support for a "small" figure class that you can use to limit the figure size (add class="small" to the figure tag). You can use this with svgs to prevent them from being scaled to full page width. So instead of having two PNGs for each figure, you can use a single SVG for both the embedded small image and the full-size that you link to. This changes the size tradeoff significantly, especially when factoring in compression (which the server will automatically apply to .svgs). So please use .svgs for the figures, and get rid of the .png commits so we don't have tens of megabytes of commits :)

As for the content. Overall comments:

It is not clear who you are targeting with this comparison. It reads a little like an academic paper, but lacks in several aspects here (most notably is the lack of discussion of related work). However, I'm guessing this is because it is not supposed to be an academic paper, but rather a showcase of all the nice features of Cake. Which makes sense, but it would be good to state explicitly in the beginning what you are trying to do, and who the target audience is.

As for showcasing Cake, your setup unfortunately conflates several aspects of Cake performance into a one-dimensional comparison between algorithms. There's the shaper, the AQM, the flow scheduler (including host fairness) and the diffserv splitting. These affect different aspects of performance, and Cake brings improvements to each one. I think your comparison would be improved by restructuring it to examine each one separately, and then having an all-out comparison at the end. This would also cut down on redundancy in the presentation. For instance, the single flow tests in the beginning, while a good introduction for someone new to the area, does not contribute any insights that are not covered by the later tests.

Finally, you are not saying anything about the limitations of Cake. While I get that this is a sales document, being honest about limitations is still important. Most of those are by design anyway (e.g., imposes a particular diffserv scheme and a particular notion of fairness, not as configurable as HTB). Restructuring the document would help with this, as it is easier to say "we designed Cake to solve these particular problems in these particular areas, and as our tests show, it does so admirably".

And some comments on specific points:

dtaht commented 7 years ago

In terms of one more nit:

SFQ is most commonly used with the default 128 depth setting. That - especially at these speeds - is a poor man's aqm.

Some of the anomolies you report are merely due to the admittedly insane 1000 txqueuelen, but a more apples to apples comparison is with SFQ at the default 128.

dtaht commented 6 years ago

I'm sorry this died, is it still needed?

chromi commented 6 years ago

On 7 Sep, 2018, at 12:43 am, Dave Täht notifications@github.com wrote:

I'm sorry this died, is it still needed?

We could simply link to the paper these days.

dtaht commented 5 years ago

How dead is this?