How to improve nfs-ganesha's I/O performance when using CEPH cluster as the backend?

[Liam-program]

I use nfs-ganesha to store 30 million small files in ceph fs, the file size is 64k. After the file is created, randomly read and write files with 25 clients through vdbench, the read and write ratio is 6:4. Then observe the I/O performance.

But I find that I/O performance is not good, so I want to ask if there is a way to improve performance?

What I currently think of is to set the parameters of MDCACHE, such as increasing the values of Entries_HWMark、 NParts and Cache_Size. But after increasing these parameters, the performance did not improve.

So in this case, can MDCACHE make the performance better? Or are there other ways and suggestions to improve performance?

The nfs-ganesha version is 3.3. The ceph cluster is three nodes.

Thanks

[dang]

There's not much you can do in Ganesha to improve I/O throughput, since Ganesha just passes I/O through to the back end. It doesn't do any kind of caching or really interfere with I/O at all. Metadata operations, however, can be affected. That's where things like Entries_HWMark will make a difference.

[davkar3n]

NFS mount is very slow on my env comparing with CEPHFS mount.

This is NFS mounted .

cephfs mount.

Any suggestion guys ?

[dang]

@jtlayton Any ideas?

[jtlayton]

I don't know of any magic bullets. This configuration is just not built for speed -- it concentrates all of the data and metadata access through the same servers, for one thing. libcephfs also has scalability problems due to it's use of a giant mutex to serialize operations.

Really, the best thing we could do to help performance would probably be to add support for pNFS, but there isn't currently a consensus on how to do that.

[davkar3n]

is it possible that you moved to NFSv4 ?

[ffilz]

Is this still active?

[davkar3n]

Yes, issue is still active

[ffilz]

FSAL_CEPH definitely needs some performance work. I am working on making FSAL_CEPH able to use Ganesha's async read/write capability which may or may not help. PNFS work has also been proposed but not committed. Beyond that, there is no commitment to work on performance. As such, I'm going to mark this an enhancement. If you have expertise and ideas to work on performance, we always welcome new contributors to the project. Performance might also be addressed directly to the Ceph project.

[davkar3n]

Hi @ffilz ,

i was investigating the NFS server performance.

when writing 4k files 10.000 and more its having such a issue.

But cephfs is has better performance comparing with NFS the same files.

i can give you example of project i was trying to copy over NFS.

i use nextcloud server and unzipped then copied over NFS

[xschlef]

Sorry to bump this old issue, but we are seeing the same issue. rand_write_4k performance is around 2-3 MB/s, rand_read_4 17MB/s. When we create a large file (5G) and mount it as loop device, format it and then run tests in this large file, we are seeing HUGE speedups. rand_read_4k jumps to 350 MB/s and rand_write_4k to 17MB/s, using default rsize/wsize.

Is there any way how we can tweak the performance of NFS here?

[ffilz]

There's also an older ganesha/cephfs performance issue: #499. I'm going to keep this newer one since it has recent activity and close the older one as a duplicate.

[blackliner]

Any ideas? Suffering from a max throughput with of about 140Mb/s, where CephFS can pull the full 25Gbit the NIC has to offer

[ffilz]

We are working on some issues here. We have done some work to address ceph memory use which might have an impact.

[jtlayton]

On Tue, 2024-02-20 at 16:44 -0800, Florian Berchtold wrote:

Any ideas? Suffering from a max throughput with of about 140Mb/s, where CephFS can pull the full 25Gbit the NIC has to offer

Expected. If you issue a WRITE to nfs-ganehsa, it then has to shard the write so it goes to correct OSD.

I think the "real" solution is to design a Ceph pNFS layout so the clients can issue reads and writes directly to the storage instead of dealing with the extra hop. -- Jeff Layton @.***>