File assembly fails when UDP packets arrive out of order

Sam, I try to start himawari-rx.bat, and use TSDuck to receive TS stream by this command: tsp -I dvb --adapter 0 --delivery-system "DVB-S2" --lnb "5150000000" --frequency 4148000000 --modulation QPSK --symbol-rate 2586148 --fec-inner "3/5" --roll-off 0.2 --polarity "horizontal" -P mpe --pid 0x03E9 --udp-forward --log -O drop It works, and receive many HRIT files. Then I can use "xrit-img.py" to generate some PNG pictures successfully, but some pictures just parts. My himawari-rx.conf is keeping default:

[rx]
path = received
format = xrit
# List of channels to ignore (e.g. "B09", "IR2")
ignored_channels = 

[udp]
ip = 239.0.0.1
port = 8001

But I found sometimes himawari-rx reports that "missing some parts" or "part before info", like this: 61481DD7EBD304786AF5ECABA4783985 I notice that "part before info" always happend before "missing some parts" at receiving the same file, and sometimes just reports "missing some parts", that makes the file failed to receive.

I also try to use EBSpro to start the "UDP Stream-server" at port 7000 with UDP, and start TSDuck by this command: tsp -I ip 7000 -P regulate --bitrate 2586148 -P mpe --pid 0x03E9 --udp-forward --log -O drop It works, too. But "missing some parts" or "part before info" still happend.

Then I keep the "UDP Stream-server" running, and start TSDuck with: tsp -I ip 7000 -P regulate --bitrate 2586148 -P mpe --pid 0x03E9 --log --output-file "udp.dump" -O drop After about 30 minutes, I terminate TSDuck and start himawari-rx by this command: python himawari-rx.py --config himawari-rx.ini --file "udp.dump" -v Finally I success to receive all parts of the pictures, and it has no "missing some parts" or "part before info" errors. Then I success to generate all PNG pictures completely that I have received by using "xrit-img.py".

Then I stop the "UDP Stream-server", and start TSDuck by this command: tsp -I dvb --adapter 0 --delivery-system "DVB-S2" --lnb "5150000000" --frequency 4148000000 --modulation QPSK --symbol-rate 2586148 --fec-inner "3/5" --roll-off 0.2 --polarity "horizontal" -P mpe --pid 0x03E9 --log --output-file "udp.dump" -O drop And also after about 30 minutes, I terminate TSDuck and start himawari-rx by this command: python himawari-rx.py --config himawari-rx.ini --file "udp.dump" -v It has no "missing some parts" or "part before info" errors, too. Then I also success to generate all PNG pictures completely that I have received.

According to UDP "do not establish end-to-end connections between communicating end systems", I guess that sometimes when the network traffic is very busy, UDP packets could be arriving out of order, or drop. If the UDP packets that send to himawari-rx is dropped, but TSDuck doesn't know, then himawari-rx would reports "missing some parts". If the UDP packets arriving out of order, himawari-rx can't rebuild the whole part, then it will lost the parts which send before "info" part, so it reports "part before info" and "missing some parts".

I suggest that himawari-rx can add TCP receive mode, and find a way to use TSDuck for send TS streams with TCP, or use another software to replace TSDuck to send TS streams with TCP, then maybe the problem can be solved.

Thanks again for you helpfull software!

I'm testing this software these days with @yanderen2

Hi, thank you very much for testing himawari-rx!

I had a feeling that packets arriving out of order would be an issue when receiving directly from the real C-band downlink. The reason most files fail is that currently the demuxer will discard any file parts which arrive before the file info packet. This can be changed but will require some re-writing of the File class.

The reason I've used UDP is that the actual packets transmitted by JMA via JCSAT-2B are multicast UDP packets. All TSDuck is doing is rebroadcasting them on the LAN for himawari-rx to receive. I do not think TCP is the answer to this problem, I should just fix the software to better handle packets which arrive out of order.

I will make these changes then push a pre-release update to GitHub and reply to this issue so you can test my changes. Thank you again for your excellent testing!

Thank you very much for your patient reply! I'm looking forward to your improvements for the software.

Hi, I have rewritten parts of the decoder to now accept file parts arriving before the information packet. I also came across another issue in the file assembly code which was causing the decoder to hang and miss UDP frames, which would cause the next file to fail.

I have now published v0.1 to GitHub. Please download himawari-rx.zip and let me know if the issue is fixed! Thanks.

Hello, thanks for the improve of the decoder! I have tested the decoder by version v0.1.1 again with TSDuck, then I found it works well most time, but sometimes still have the PART BEFORE INFO and MISSING PARTS issues. Like this: himawari-rx test

First I start himawari-rx and use bitrate 2586148 for TSDuck: tsp -I file "154.0E_4148.678_H_2587_(2020-12-27 16.04.28)_dump.ts" -P regulate --bitrate 2586148 -P mpe --pid 0x03E9 --udp-forward --log -O drop It works well and has no issues at the first time. But when I test it again, the issues caused, and the files would not be received when MISSING PARTS happend. Then I test it again and again, I found the issues caused is not very often, but is still an issue.

I think there are 2 problems needed to solve:

Any file parts arrived before the file info packet.
Some file parts arrived after the file end packet.

Then I read the script of this decoder, and found the changes of def parse_file_contents(self, p) in assembler.py. I found self.files[p.uid] = File() after if not self.file_exists(p), it is a great idea to solve problem 1! I think problem 1 is solved so the issues caused not very often.

But the problem 2 is still an issue. For this problem, I have an idea for reference: First to write the imcomplete files to disk when the end parts arrived, but still keep the informations of them and marked with "imcomplete file" (with completed parts informations in it). Then if any parts of "imcomplete file" received after the "imcomplete file" wrote to disk (if there is any missing part between two parts, just fill it with "\x00"), just read the file from disk and add the new parts to it (at the right position of the file), then write it to disk again, and update the marks of completed parts of the "imcomplete file" informations. If all parts of the file is received at any time, the file would finally be wrote completely to disk (if the file is compressed and the config "format" is "xrit", then decompress it automatic before write to disk), and then just remove the "imcomplete file" informations which is receive completed.

I found another issue, is that some files are just dropped without any notice in the decoder when receiving. And I also found the reason possibilly, is that at def parse_file_complete(self, p) in assembler.py, these lines:

    def parse_file_complete(self, p):
        """
        Parse file complete packet (type 255)
        """

        # Ignore packet without associated file object
        if not self.file_exists(p): return

        # Check file info has been set
        try:
            self.files[p.uid].name
        except AttributeError:
            # Removed complete file with missing info
            del self.files[p.uid]
            return

It means when the file end packet arrived before the file info packet, the decoder would drop the whole file directly, without any notice or data wrote.

This is the problem 3 that I've nerver found before. And my idea is:

if not self.file_exists(p) then create this file, write the empty file to disk, and mark the file as "imcomplete file".
del self.files[p.uid] modify to mark the file as "imcomplete file", and keep the informations until the file info packet arrived. When an "imcomplete file" is receive completed, just write it to disk (or if the file is compressed and the config "format" is "xrit", then decompress it automatic before write to disk) and remove the "imcomplete file" informations of it.

And don't worry about the memory of the computer, I also have an idea, is that add a time mark on each "imcomplete file" information, then remove the "imcomplete file" informations that is waitting for a long time (probbly has packets dropped really). For example, if himawari-8 transmits files every 10 minutes, then the decoder would remove all "imcomplete file" informations which is marked 10 minutes ago.

sam210723 / himawari-rx

File assembly fails when UDP packets arrive out of order #2