Performance Benchmarks

[andersy005]

@kmpaul & @tinaok

I just put together a simple notebook demonstrating how to mount the tar archive via ratarmount. I then ran some smoke tests to see what the performance looks like:

• Native/Original netCDF files

  

• netCDF files in the tar archive

  

Here's a rendered version of the notebook: https://nbviewer.jupyter.org/github/NCAR/xtar/blob/master/notebooks/tar-netcdf.ipynb. I included dask's performance report in the notebook itself.

[kmpaul]

The Dask performance reports show that they are almost the same! That's great!

[kmpaul]

BTW, I want to write a blog post about this. 😃

[kmpaul]

So, FUSE needs to be installed for this to run. Does this work on Cheyenne? Can you install libfuse on HPC systems?

[andersy005]

BTW, I want to write a blog post about this. 😃

Me too.

Can you install libfuse on HPC systems?

libfuse is available on conda-forge: https://anaconda.org/conda-forge/libfuse

Does this work on Cheyenne?

Haven't tested it yet.

[kmpaul]

Hmm. I tried installing libfuse with conda and it failed.

☹️

[kmpaul]

I think libfuse is only installable on linux. And you need to install osxfuse on Mac by hand?

[andersy005]

Hmm. I tried installing libfuse with conda and it failed.

😔I didn't realize that the conda-forge version is linux only.

And you need to install osxfuse on Mac by hand?

I think so. I installed osxfuse on my local machine a while ago, and that's why I didn't even notice that FUSE was a dependency when I ran the notebook

[kmpaul]

Installing osxfuse is pretty easy and it works once installed. So, that's not too big of a deal. I wonder if installing ratarmount on a linux system will install libfuse automatically.

[kmpaul]

Some notes on testing this on Cheyenne:

• The system install of libfuse is old (version 2.9.3). It was not until version 2.9.9 (or version 3.3.0 for fuse3) that support for Lustre and GPFS filesystems was added to libfuse. So, you will get an error if you try ratarmount with the system install of libfuse.

• If you use conda to install the conda-forge libfuse package (currently version 3.9.1), you need to tell the fusepy package where to find this new library by setting the FUSE_LIBRARY_PATH environment variable equal to the absolute path to the libfuse3.so file.

• Currently, the libfuse conda-forge package does not install the fusermount3 utility function needed by fusepy. As a result, if you do the above, you still encounter an error:

  fuse: failed to exec fusermount3: No such file or directory

[kmpaul]

I have opened an issue in the libfuse-feedstock repo to ask if they can build the fusermount3 utility with the libfuse package:

https://github.com/conda-forge/libfuse-feedstock/issues/2

[tinaok]

Thanks a lot @kmpaul and @andersy005 Great to see just talking about it in the evening, and waking up in Europe, work is done! I'll check on the HPC site here if there is libfuse installed in the right version for lustre/gpfs.

[kmpaul]

I just wanted to chime in here and give an update.

The libfuse issue has stalled, and I suspect it will not get completed. It turns out is was substantially more complicated than I previously thought. I've asked out sysadmins to install an updated version of libfuse on our system, but they chose not to do it. So, we're stuck testing this on Cheyenne.