Receiving File and writing to mounted share is slow

[Arikael]

Hi

I'm using norm to send files unidirectionaly/silent to a receiver. Our receiver is a .NET Application which uses a SWIG wrapper for norm, but we are able to reproduce the problem with the example normApp.

Normally we just receive the files an use a local directory as a cache dir. This works as expected. But now we tried to use a mounted samba share and this leads to very regular failures. norm receives the files on interface1and the the samba share is connected to interface2. Those are physical interfaces.

EDIT We try to send 20 files between 250 and 1000Mbyte. It does work when we send smaller files END EDIT We already restricted sending to 250Mbit (which is rather slow, we are able to get higher when using a local directory)

What I was able to observe via iftop is, that the data is received on interface1 with about 250Mbit/s but iftop on interface2 shows only a speed of about 130Mbit/s. After some time there is a sudden spike to 500-800Mbit.

We did also some tests with udpcast, and although it works a bit different, iftop on interface2 shows a continous speed of about 500-800Mbit and doesn't seem to fail at all. There are other factors our udpcast application does different, but still the difference in speed is eye-catching.

I also tested the smb mount with dd which showed good speed in line with udpcast.

This lead me here, although I'm not sure yet if it is a problem with norm, protolib or something different. (currently it seems to point to protolib).

I will try to investigate the source code, but since I'm not a C++ developer, I wonder if you might have some ideas or insights. Any pointers where to look? Maybe some block size or caching issue when writing the file?

Thanks

tl;dr

• norm wrapper and the normApp write with about 130Mbit/s to a samba share and fails (probably because buffer are getting filled because we send with 250Mbit/s)
• if a local directory is used, it works.
• udpcast writes consistently with 500-800Mbit.

[bebopagogo]

There could be a couple of things that may be in play here:

1) On Windows, I have found performance for UDP can be limited. One issue that was identified at one point was that the default Windows UDP socket buffer size is small and I think there was a registry entry change that allowed for larger socket buffer sizes. The NORM API (and some command line options on some of the demo apps) let you set the socket buffer size but I think on WIndows some additional measure is needed. Perhaps there is a way to do that programmatically that your "udpcast" does where the NORM code doesn't do that. If that's the case, I could look into that. If the receive socket buffer size is small then even though your sender is sending 250 Mbps, there could be loss at the receiver due to UDP rx socket overflow where NORM is doing a lot of retransmissions but the "goodput" of reliable delivery is low (i.e. the 130 Mbps you see)

2) Another thing that can help using NORM ack-based option for flow control. The "normCast" code in the "examples" directory does this and provides a coding example for that. Flow control keeps the sender / receiver "in bounds" when the there are timing variations due to user-spaced process scheduling dynamics, etc.

3) If you are configuring the normApp to write to the Samba share, this could aggravate the above 2 since normApp is single-threaded and any delays while writing to the Samba share would delay the norm process reading its rx socket and lead to overflows if the buffer size is too small.

One or some combination of these issues could be in play. I haven't push on Windows testing much lately as we tend to use Unix machines quite a bit more. What command line options are you using with normApp?

[Arikael]

Thanks for your answer

1. We are mainly using Linux (CentOS and Ubuntu) for testing, but the result was the same when using a share folder from Windows. udpcast is the application from here -> http://www.udpcast.linux.lu/ Regarding Buffer size etc. I did observe a lot of receiving errors (with netstat -su), but I think they are a symptom of the slow writes to the mounted share, because the receiver side (the one which writes to share) is heavily optimized for receiving a lot of data in a short time. Usually it's now the disk which would be the bottleneck (starting at 1500Mbit/s). Like I said, when I copy a file to share manually it's quite fast. So the buffers seem to overflow because of the slow writes to the share. But the question is why are the writes so slow? I observed a lot of received duplicate block message or AppendRepairItem when I increased the debug level.

2. Unfortunately we are forced to use silent senders/receivers.

3. If I'm not mistaken the norm library itself is not single threaded, is it? If that's the case, I would assume that the problem isn't the single threaded app.

I tried the normApp with the following parameters

Sender parity 2 rate 250000000 address 239.0.0.1/5000 interface myInterface segment 8940 sendfile /data/share

Receiver address 239.0.0.1/5000 interface myInterface silentReceiver rxcachedir /data/share debug 5 address 239.0.0.1/5000 interface myInterface silentReceiver rxcachedir /data/share lowDelay rxbuffer 100000000 saveAborts debug 5

EDIT After some digging I ended up here in normObject.cpp, L2441

With some debugging, it seems to me that norm always writes in max chunks of segmen size (in our case 8940). Is that correct? If I'm not completly mistaken udpcast seems to write in much bigger chunks (max 131072 = 2 x 65536)

Maybe this explains it?

[Arikael]

I just did some (rather naive) tests to test if I notice a difference with write and fwrite and different buffersizes (I always used 1 for fwrite size parameter).

I tested 64,1024, 8940 and 65535 bytes as buffer size and just read a local file (approx 850Mbyte) and wrote it to share.

Writing to a share was extremly slow when using small batch sizes (40s with write) and way faster when doing big chunks (400-500ms). fwrite was way faster, although still very slow, when doing small batch sizes (17s for 64B). The difference was neglible with (8940 and 65535).

When writing to local files 64B was still slowest but 65535 was second slowest while the fastest was for both write and fwrite 8940B.

Testing with iftop on the samba server showed that none of the methods suffered from the 140Mbps Cap.

[bebopagogo]

The NormFileObject class is where the file i/o reads and writes are for NORM_OBJECT_FILE transfers. FYI - I am on work travel this week with limited access to Internet. I will provide some more comments as soon as I can.

[Arikael]

After some more testing, this strange cap of approx. 130Mbit vanished. I can't really tell why. Sometimes it works sometimes only sends with about 5Mbit and the process has to be killed.

Currently we are investing what the impact of segmentsize and various sysctl parameters are. It seems that, a segmentsize above 4000-5000 has a detrimental effect (strangely it did work before, I need to investigate that).

[Arikael]

We just did some additional tests with NFS and we were able to successfully transfer files and iftop showed a speed of about 930Mbit on interface2

So, it seems there are issues between samba and norm. We are still investigating.

[Arikael]

After sheding blood sweat and tears, I think we have found the culprit :D

The normApp as well as our application (which is inspired by the normApp) does create the temporary file when receiving the event RX_OBJECT_NEW and object type NormObject::FILE which happens at the beginning of receiving a new file.

See https://github.com/USNavalResearchLaboratory/norm/blob/a1bb33c3540b248f0bf2152a9ac01ee52c32c0ef/src/common/normApp.cpp#L1792

After this happened, the file gets accepted which also means a file descriptor is created (by opening the file) which will be used for all other file operations.

When receiving the event RX_OBJECT_INFO and object type NormObject::FILE the file gets renamed. And here lies the problem. While working with the local file system it's no problem to rename the file while reusing the same file descriptor. But Samba seems to trip up with that which results in slow speeds and reduced data integrity.

So the solution is either to use the event RX_OBJECT_COMPLETED to rename file or to close and reopen the file right after you renamed the it. This requires editing norm's source code. Right after https://github.com/USNavalResearchLaboratory/norm/blob/a1bb33c3540b248f0bf2152a9ac01ee52c32c0ef/src/common/normFile.cpp#L345

Close();
Open(newName, O_TRUNC | O_RWDR)

Wo got speeds up to 900Mbit without errors with both solutions.

[bebopagogo]

Thank you for your blood, sweat, and tears. You've made me so very happy you have solved this. I don't think it would be too problematic to make the change within NORM, but in the bigger scheme of things, it could still be a better practice for an application to wait for file reception completion (i.e. RX_OBJECT_COMPLETED) to avoid any possible hiccup in the middle of file reception (although if receiving a steady series of files, that may not matter). With NORM_OBJECT_FILE, the file writing happens directly in the underlying NORM operating system thread. Even though fwrite() takes advantage of file system I/O buffering (versus the low level write() call that does not), it could be the case that an additional layer of file buffering could be useful when using a remote file system (e.g., samba). For example, allow NORM write files to a local disk as a sort of cache and then let another operating system thread perform a copy of that file to the remote file system. SInce NORM is a user-level process, this would let the NORM process do its thing with minimal disruption due to external factors. I may take both of your suggestions here, making both the change to normFIle.cpp and modifying the NORM examples to defer the file renaming ...

[bebopagogo]

FYI - I updated the NormFIle code to close/reopen upon renaming. Please let me know if this resolves your issue and we can close this issue. Thanks for your input on this.

[Arikael]

I'm not sure If I currently have the time We are in the mids of finishing the project :)

But thanks for your effort.