Max filesize that can be read using libNFS

[bishnu1184]

Is there a limit of filesize that libNFS can read in nfs_read() call. I have a file with 10.8 GB in size, but when I try to read that I get crash with NFS_v3 version in first nfs_read() call. Same file I could start read without any issue if I use version NFS_v4 (yet to read complete file).

Crash location is in "nfs_v3.c" line number(4644) memcpy(&data->buffer[mdata->offset - data->offset], res->READ3res_u.resok.data.data_val, count);

Thanks, Bishnu

[bishnu1184]

Can you suggest any example for reading the large file using nfs_read(). Because In case of very large file (10GB) nfs_read() will not return complete file in one go. Then I want to know

1. What should be the minimum buffer size that we should allocate while.
2. I am getting 1 MB data in every nfs_read() call, Is there an API to get this information (if this varies)

Thanks, Bishnu

[sahlberg]

On Mon, Aug 9, 2021 at 10:22 PM bishnu1184 @.***> wrote:

Can you suggest any example for reading the large file using nfs_read(). Because In case of very large file (10GB) nfs_read() will not return complete file in one go. Then I want to know

1. What should be the minimum buffer size that we should allocate while.
2. I am getting 1 MB data in every nfs_read() call, Is there an API to get this information (if this varies)

The maximum read-size is clamped in libnfs. In NFSv3 it is clamped to the smallest of the two value NFS_MAX_XFER_SIZE (which is set to 1MB in include/libnfs-private.h, this is an arbitrary limit) and the preferred maximum readsize that the server reports in the FSINFO3 call.

In NFSv4 it is just clamped to NFS_MAX_XFER_SIZEsince I do not query the server to the best i/o size yet. This value is just set arbitrarily to prevent applications from using up "too much" memory.

You can increase these limits, by increasing this variable.

The maximum preferred read/write size for an nfs context can be found by calling: nfs_get_readmax(struct nfs_context nfs) nfs_get_writemax(struct nfs_context nfs)

So, for NFSv3, I clamp it to the smallest of 1MB and the hint for the preferred max that the server responded in to the FSINFO3 call.

But, both the "max read/write size" that the server responds in FSINFO3 as well as the NFS_MAX_XFER_SIZE size are just hints for a useful default for the max size.

An NFS server will usually respond and accept read/write calls even if they are even if they are bigger than what they advertize in FSINFO3.

I think the current limits are useful as the default values, but I can see a use case for an application to override these defaults and set other limits.

I will add two new functions to the API nfs_set_readmax() and nfs_set_writemax() where you can increase the limits.

Beware though, eventually there will be a limit where the server will start clamping the maximum sizes and in which case you would get a short read.

Thanks, Bishnu

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/sahlberg/libnfs/issues/354#issuecomment-895178615, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADY3EG7FFM23A72FQFSGNTT37CBHANCNFSM5BZM52HA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

[sahlberg]

The crash in nfsv3 is a bug. I will have a look.

[sahlberg]

For the crash. Ignore all I wrote above. I see the real issue now. I will fix it in a few days.

[sahlberg]

Please test current master. I have checked in a fix that addresses an issue where casting the size of the read buffer to a uint32 caused the buffer to not be allocated properly.

[bishnu1184]

I still get crash if I try to pass read length complete 10 GB and a buffer. Following is the callstack

Call Site VCRUNTIME140D!memcpy_repmovs+0xe [d:\agent_work\8\s\src\vctools\crt\vcruntime\src\string\amd64\memcpy.asm @ 114] nfs!nfs3_pread_mcb+0x2a8 [c:\users\administrator\downloads\libnfs-master\lib\nfs_v3.c @ 4652] nfs!rpc_process_reply+0x20c [c:\users\administrator\downloads\libnfs-master\lib\pdu.c @ 317] nfs!rpc_process_pdu+0x565 [c:\users\administrator\downloads\libnfs-master\lib\pdu.c @ 579] nfs!rpc_read_from_socket+0x407 [c:\users\administrator\downloads\libnfs-master\lib\socket.c @ 364] nfs!rpc_service+0x310 [c:\users\administrator\downloads\libnfs-master\lib\socket.c @ 509] nfs!nfs_service+0x33 [c:\users\administrator\downloads\libnfs-master\lib\libnfs.c @ 251] nfs!wait_for_nfs_reply+0xb9 [c:\users\administrator\downloads\libnfs-master\lib\libnfs-sync.c @ 197] nfs!nfs_read+0xc0 [c:\users\administrator\downloads\libnfs-master\lib\libnfs-sync.c @ 573] NFSJNIWrapperTest!LibNfsHandler::ReadNfsFile+0x2b1 [c:\code\depot\main\dev\native\src\discover\nfsjniwrapper\src\libnfshandler.cpp @ 266] NFSJNIWrapperTest!ThreadProc+0x314 [d:\work\discover\libnfs\2015\nfsjniwrappertest\nfsjniwrappertest\nfsjniwrappertest.cpp @ 64]

I have also shared google drive link with you for dmp and debug symbol file.

[bishnu1184]

I also tried to get the maximum read length using nfs_get_readmax(), but I am getting 0 only for both NFSv3 and NFSv4.