Investigate slow NetCDF4 write speeds from rsaquery

[z0u]

In the latest deployment of RSA at VPAC, we noticed that large queries were running very slowly - apparently much more slowly than the previous deployment. It turns out that writing to NetCDF 3 instead of NetCDF 4 makes rsaquery run much faster for large files. We should investigate why that is.

• Library versions?
• Application configuration?
• OS/hardware configuration?

Unfortunately that previous deployment has been destroyed to make room for the new system, so we can't compare the performance.

[z0u]

There is a good report called the NetCDF-4 Performance Report, produced in 2008 by the The HDF Group, that lists various benchmarking results for netCDF-4 vs netCDF-3. These bits may be of particular interest to us:

• NetCDF-4 uses chunking for storing data, unlike netCDF-3. You can turn chunking off for netCDF-4, but not if you want to use compression.
• IO performance is greatly dependent on the operating system policy for caching. The report considers two cases: a) default caching, and b) flushing data to disk after each write.
• The problem: When using default caching, chunked netCDF-4 was less than half as fast as netCDF-3 when writing large files.
• With chunking turned off, netCDF-4 was only a little slower than netCDF-3.
• With OS caching turned off, chunked netCDF-4 was a little bit faster than netCDF-3. But it's worth noting that this is an extreme case, and not something that we would do in practice: disabling file system caching reduced all throughput by an order of magnitude.
• When the chunk size was increased, netCDF-4 performed as well as netCDF-3.
• While the report has a section on compression (4.2), it's not clear whether the tests comparing chunking and caching strategies used compression with netCDF-4. I think they probably did not.

The netCDF C library provides functions to configure chunking; we could use these to try to get better performance. Apparently netCDF-4 version 4.0 had poor defaults for chunking that were fixed in 4.2 - but this shouldn't be the cause of our current problems, because our docs recommend using version 4.3.

This shows that library versions, application configuration and OS/hardware configuration could all indeed play a part in the reduced performance that we are seeing.

[z0u]

I have found an old set of test results that directly compare nc3 and nc4 (compressed) execution time for queries. Despite giving quite good compression (65% smaller files for continuous data; 8.8MB vs 25MB output), nc4 performed about the same as nc3 (actually marginally faster in most tests).

Just running a small informal test on my machine:

• add_binary test, netCDF 3 output:

  $ time rsa data query add_binary_rsa.xml --of=nc3
  ff808181461331ae0146133345070007 0/1   100% Finished running filtersh shape (3, 1000, 2700)
  
  real 0m7.087s
  user 0m0.016s
  sys  0m0.018s
  
  $ ncdump -k add_binary_rsa.xml.nc
  classic

• add_binary, netCDF 4 output:

  $ time rsa data query add_binary_rsa.xml --of=nc4
  ff808181461331ae014613336a400008 0/1   100% Finished running filtersh shape (3, 1000, 2700)
  
  real 0m6.086s
  user 0m0.016s
  sys  0m0.017s
  
  $ ncdump -k add_binary_rsa.xml.nc
  netCDF-4

So in fact in this test netCDF 4 is a bit faster. However this is quite a small test case - not even one 5000x5000 tile in size.

[z0u]

The latest version of NetCDF-Java boasts better default chunking settings for netCDF-4 output, which is likely to improve performance. We could try upgrading, but note:

• It now requires Java 7. Not a big deal, but rsaquery currently needs Java 6. The rest of RSA uses Java 7.
• TDS needs Tomcat 7, while RSA needs Tomcat 6. I'm not sure what we need to change in RSA to get it to work with Tomcat 7, but I remember there were missing classes.

[z0u]

Actually, this might be possible to configure in the version that we're currently using. When creating a new dataset to write to, we use the ucar.nc2.NetcdfFileWriter.createNew function. By passing an extra argument we can adjust the chunking strategy.

ucar.nc2.NetcdfFileWriter.createNew(
    ucar.nc2.NetcdfFileWriter.Version,
    java.lang.String,
    ucar.nc2.jni.netcdf.Nc4Chunking)

Nc4Chunking is an interface, so we would need to implement a new class. Or maybe we can just pull in the class from NetCDF-Java 4.5 without doing a full upgrade.