Performance tests

[lavelle]

I think it would be good to get some performance tests added to the test suite before spending any time implementing new methods to improve performance.

This would help us prevent regressions, and even in cases like lavelle/BrowserFS#11 where it's fairly obvious that it will improve performance, it's nice to be able to quantify and measure the improvement.

This would also allow us to make comparisons of the relative speeds of the different backends, to have some quantitative evidence for users when they're weighing up the advantages and drawbacks of each.

I couldn't find any performance tests in the Node repo, but it wouldn't be too difficult to write our own.

[jvilk]

I agree with the desire for benchmarks, but I will not have time to do this for quite awhile.

It won't be difficult to write simple benchmarks, but it would be difficult to simulate a realistic filesystem workload. Simple benchmarks are a good start, though.

One benefit of emulating the node API: We can benchmark ourselves against Node. Granted, you'll probably want to do that in a RAM disk to isolate hard drive variability.

If you want to start work on this, please go ahead!

[lavelle]

Okay, I'll try and get some simple benchmarks for append and truncate done before I begin on the HTML5 FS performance improvements.

I'll make a wiki page for benchmarks.

[lavelle]

Made a start.

[lavelle]

Wiki page.

[jvilk]

OK. Here are some notes for your use:

• Benchmarks shouldn't be part of the unit tests. Unit tests should quickly test the broad range of functionality offered by the interface. They should be minimal if possible, but should maximize code coverage. Meanwhile, it is OK for benchmarks to take awhile to run.
• We should make a benchmark target in the Makefile that starts up a simple webserver serving up a static page that contains the benchmark logic. It should have a button to start the test (meaning, it shouldn't start when the page loads), and you could maybe do a dropdown for backend selection?

Little checklist:

• [ ] Simple server that serves up static pages (could simply run python -m SimpleHttpServer or w/e from the Makefile.
• [ ] benchmark Makefile target that depends on the BFS library and runs the server.
• [ ] HTML page that contains benchmark logic, a button for starting the test, and a method for selecting the backend.
• [ ] Logic that times the execution of the tests, and presents them on the benchmark page.
• [ ] Checkbox for invoking the browser's JavaScript profiler separately for each benchmark (e.g. [ Run Tests ] [X] Run profiler ). We don't want this to be default, because the profiler slows down execution.

[lavelle]

Yeah, I agree that benchmarks and unit tests should be separate, I've already done that.

Don't see why we can't use Karma for both though, since it's already installed and configured. I've just passed in a boolean flag from the Makefile so it can decide which to run. Running a separate server for this seems kinda redundant.

[jvilk]

But with karma: How do you collect and present results? How do you easily profile the benchmarks? You need to hit debug, set a breakpoint, enable the profile, hit run again...

[lavelle]

I'd much rather stick with Karma's approach and have all input and output on the command line - tabbing to a browser and using a GUI is just going to disrupt workflow.

We can always have a command line tool for autogenerating a nicely formatted static HTML page with a summary of the benchmark results for easy sharing, since manually copying them into a wiki is cumbersome.

I hadn't thought about profiling - my use case for this is getting a quick overview of the total runtime to compare backends, rather than optimising individual lines of code. Could that be a separate tool for more intensive performance debugging, in addition to the command line tool for getting a summary?

[jvilk]

Oh, I wasn't concerned with how well it's presented.

And I'm currently suffering from profiling difficulties with Karma. Browsers hate it when you need to profile things that run when you open the page.

Feel free to continue with karma, but something like this will need to be built eventually (and I will definitely need soon; the Buffer implementation can probably be optimized...).

[lavelle]

Okay, well the vast bulk of the work (writing the benchmarks themselves) will be reusable. It's only a few lines of code being thrown away if you don't use Karma to run them, so I'll stick with this for now while I build a larger benchmark suite, and we can discuss the implementation details of the environment in which they're run further down the road.

[jvilk]

OK.

[jvilk]

What's the status of this? Is the general running-the-tests-with-karma logic ready for merging?

Here are some benchmarks that I would find useful for working on the current IE9 performance issues:

• [ ] Saving / loading a large file (programmatically generate a 2 MB file, save it to the filesystem, read it from the filesystem, verify integrity by comparing it to originally generated data).
• [ ] Saving / loading many small files (100 bytes to 1KB; ~2MB worth of files). You should pregenerate random sizes in this range so we can deterministically generate and test a set of files in this range. This is a simple simulation of Doppio's 'preloading' use case.

Backend-independent performance:

• [ ] String -> buffer on a multi-megabyte piece of data.
• [ ] Buffer -> string on a multi-megabyte piece of data.
• [ ] Normal array of bytes -> buffer on a multi-megabyte piece of data.

[jvilk]

I'm willing to write all of these, but I need something I can plug 'em into. :smile:

EDIT: And, of course, I'm not against you writing these yourself if you feel up to it. But I'd like them soonish.

[wjordan]

Just adding some potentially-helpful related resources for this item:

• feross/buffer has some browser-based performance tests in master/perf, that could be reused to benchmark buffer module performance.
• io.js/node.js has a substantial suite of benchmarks that cover certain node.js apis. Rather than building + maintaining a parallel performance test suite, perhaps it could be less work and more maintainable to hook into the subset of tests covered by this package (buffer, fs, etc). This would also provide the most direct and thorough comparison against Node as well.

[jvilk]

These are all good resources. I agree that we should re-use these whenever possible.

Note that we would need to mark certain tests are using particular portions of the Node API. Some filesystems don't support synchronous file system operations, some don't support permissions, etc.

Also, most notably, my Buffer implementation doesn't support the index operation ([]), so those need to be replaced with get/set.