search

sam210723 / xrit-rx

📡 Receive images from weather satellite GEO-KOMPSAT-2A (GK-2A).
https://vksdr.com/xrit-rx
MIT License
97 stars 18 forks source link

HRIT Decoding (J2K and missing CPPDUs) #15

Open kisaa opened 4 years ago

kisaa commented 4 years ago

I got a dump of an HRIT stream and am trying to decode it using xrit-rx. I get a lot of errors like

   [TP_File] COUNTER: 0 (Other, SEGMENT:?) LENGTH: 5078090
     LENGTH: ERROR (EXPECTED: 5078090, ACTUAL: 5069900, DIFF: -8190)
     SKIPPING FILE DUE TO DROPPED PACKETS
   NO CP_PDU TO FINISH (DROPPED PACKETS?)
   [CP_PDU] EOF MARKER

Product files (.bin) are written, but their size is too small.

I have checked VCDU counter field, and I does not see any packet loss from receiver, counter leaps only on transmission start/stop:

Time               Prev   Current
12:46:39 Counter leap: e1d486 0
12:51:50 Counter leap: 1bcf2  e1d487
12:56:39 Counter leap: e371e3 0
13:01:51 Counter leap: 1be25  e371e4
13:06:39 Counter leap: e50e09 0
13:11:52 Counter leap: 1bf18  e50e0a
13:16:39 Counter leap: e6a940 0
13:21:52 Counter leap: 1bfe4  e6a941
13:26:39 Counter leap: e843ab 0
13:31:53 Counter leap: 1c081  e843ac
13:36:40 Counter leap: e9de29 0
13:41:53 Counter leap: 1c0cf  e9de2a
13:46:40 Counter leap: eb77a9 0

Links to dumps: hrit03.dmp hrit03.log hrit04.dmp hrit04.log

sam210723 commented 4 years ago

I haven't been able to test HRIT since goesrecv has not been modified for QPSK yet. What is your demodulator and decoder chain?

Thanks for the VCDU files, I'll take a look!

sam210723 commented 4 years ago

HRIT headers are intact but the JPEG/JFIF headers are broken for some reason. There does seem to be a bigger demuxing issue so I'll take a look at that first, then look at the JPEG issues.

HRIT Headers

kisaa commented 4 years ago

What is your demodulator and decoder chain?

This is a professional meteorological receiver, we used it to get HRIT / LRIT with Electro-L. It supports Viterbi and Reed-Solomon decoding and provides 1024 (full frame with sync word), 1020 (CVCDU) or 892 (VCDU) bytes packets over Ethernet. I changed the main script a bit to get VCDUs from UDP socket with the correct interleaving settings. Unfortunately, my skills are not enough to write a QPSK software demodulator ( Maybe later I can adapt https://github.com/dbdexter-dev/meteor_demod for these needs - it is written for QPSK / OQPSK.

sam210723 commented 4 years ago

Nice! is it a Dartcom receiver?

I have started work on a QPSK DSP chain using GNURadio source for reference. Using LRPT code as well is a good idea! I'll be getting back to that soon.

kisaa commented 4 years ago

is it a Dartcom receiver?

No, this is a Russian model produced by http://www.etra-plus.ru/ (D8L, could not be found on the site, possibly EOL)

sam210723 commented 4 years ago

Interesting, I thought it looked similar to the Comtech CDM modem series, then I found this on the D4L product page: image

image (Comtech CDM-600)

image (ETRA-D4L)

sam210723 commented 4 years ago

The JPEG headers in these images seem to be malformed and don't have enough info to recover the image data.

JPEG Headers

They are basically empty when compared to a normal JPEG/JFIF header. I'm going through the HRIT documentation at the moment to see if there's anything explaining this.

JPEG Header Comparison

kisaa commented 4 years ago

According to http://nmsc.kma.go.kr/resources/homepage/pdf/GK2A_HRIT_Mission_Specification_Documen_v1.0.pdf, p.24, JPEG2000 is used, not regular JPEG. But header looks untypical even for JPEG2000.

sam210723 commented 4 years ago

This makes sense as Pillow was throwing exceptions when loading the image. It needs OpenJPEG installed and in PATH to handle J2K.

It also mentions they're using J2K in a lossless configuration so that could explain the sparse headers.

kisaa commented 4 years ago

Got images from IR channels! Decompressed HRIT data using jpeg utility from https://github.com/thorfdbg/libjpeg, with intermediate conversion to PPM and regular JPG and building all segments together, by (dirty, sorry for this!) calling hrit2jpg.processHRIT() from demuxer.py (Channel.handle_xRIT) after 10th segment is saved to disk:

        else:
            # Print XRIT file info
            xrit.print_info()
            if xrit.FILE_NAME.endswith('10.hrit'):
                hrit2jpg.proceedHRIT(xrit.get_save_path(self.config.output))

Still having troubles with VIS channel ((

IMG_FD_009_IR105_20200601_012636

hrit2jpg.zip

sam210723 commented 4 years ago

Nice work! I'll look at streamlining that process then integrate it into xrit-rx.

kisaa commented 4 years ago

I left the system running overnight, and today I discovered the following: sometimes (I have no idea about the pattern) one or two segments of the VIS channel are saved successfully, and then the image is successfully (partly) decoded. The rest of the time, I get the following message from the handle_CPPDU function (I changed the line if self.config.verbose: to if True: to see only these debug messages):

[VCID 0] GK-2A: FULL DISK
  [XRIT] "IMG_FD_038_IR105_20200602_061636_01.hrit"
  [XRIT] "IMG_FD_038_SW038_20200602_061636_01.hrit"
  [XRIT] "IMG_FD_038_WV069_20200602_061636_01.hrit"
  [XRIT] "IMG_FD_038_IR123_20200602_061636_01.hrit"
    LENGTH:     ERROR (EXPECTED: 5282322, ACTUAL: 5274132, DIFF: -8190)
    SKIPPING FILE DUE TO DROPPED PACKETS
  [XRIT] "IMG_FD_038_WV069_20200602_061636_02.hrit"
  [XRIT] "IMG_FD_038_SW038_20200602_061636_02.hrit"
  [XRIT] "IMG_FD_038_IR123_20200602_061636_02.hrit"
  [XRIT] "IMG_FD_038_IR105_20200602_061636_02.hrit"
    LENGTH:     ERROR (EXPECTED: 8108234, ACTUAL: 8091854, DIFF: -16380)
    SKIPPING FILE DUE TO DROPPED PACKETS
  [XRIT] "IMG_FD_038_IR105_20200602_061636_03.hrit"
  [XRIT] "IMG_FD_038_SW038_20200602_061636_03.hrit"
  [XRIT] "IMG_FD_038_IR123_20200602_061636_03.hrit"
  [XRIT] "IMG_FD_038_WV069_20200602_061636_03.hrit"
    LENGTH:     ERROR (EXPECTED: 9413834, ACTUAL: 9397454, DIFF: -16380)
    SKIPPING FILE DUE TO DROPPED PACKETS
  [XRIT] "IMG_FD_038_IR105_20200602_061636_04.hrit"
  [XRIT] "IMG_FD_038_WV069_20200602_061636_04.hrit"
  [XRIT] "IMG_FD_038_IR123_20200602_061636_04.hrit"
  [XRIT] "IMG_FD_038_SW038_20200602_061636_04.hrit"
    LENGTH:     ERROR (EXPECTED: 9086738, ACTUAL: 9070358, DIFF: -16380)
    SKIPPING FILE DUE TO DROPPED PACKETS
  [XRIT] "IMG_FD_038_IR105_20200602_061636_05.hrit"
  [XRIT] "IMG_FD_038_SW038_20200602_061636_05.hrit"
  [XRIT] "IMG_FD_038_WV069_20200602_061636_05.hrit"
  [XRIT] "IMG_FD_038_IR123_20200602_061636_05.hrit"
    LENGTH:     ERROR (EXPECTED: 9166682, ACTUAL: 9150302, DIFF: -16380)
    SKIPPING FILE DUE TO DROPPED PACKETS
  [XRIT] "IMG_FD_038_WV069_20200602_061636_06.hrit"
  [XRIT] "IMG_FD_038_IR123_20200602_061636_06.hrit"
  [XRIT] "IMG_FD_038_IR105_20200602_061636_06.hrit"
  [XRIT] "IMG_FD_038_SW038_20200602_061636_06.hrit"
    LENGTH:     ERROR (EXPECTED: 9326610, ACTUAL: 9310230, DIFF: -16380)
    SKIPPING FILE DUE TO DROPPED PACKETS
  [XRIT] "IMG_FD_038_SW038_20200602_061636_07.hrit"
  [XRIT] "IMG_FD_038_WV069_20200602_061636_07.hrit"
  [XRIT] "IMG_FD_038_IR123_20200602_061636_07.hrit"
  [XRIT] "IMG_FD_038_IR105_20200602_061636_07.hrit"
    LENGTH:     ERROR (EXPECTED: 8890946, ACTUAL: 8874566, DIFF: -16380)
    SKIPPING FILE DUE TO DROPPED PACKETS
  [XRIT] "IMG_FD_038_IR123_20200602_061636_08.hrit"
  [XRIT] "IMG_FD_038_IR105_20200602_061636_08.hrit"
  [XRIT] "IMG_FD_038_SW038_20200602_061636_08.hrit"
  [XRIT] "IMG_FD_038_WV069_20200602_061636_08.hrit"
    LENGTH:     ERROR (EXPECTED: 7802050, ACTUAL: 7785670, DIFF: -16380)
    SKIPPING FILE DUE TO DROPPED PACKETS
  [XRIT] "IMG_FD_038_WV069_20200602_061636_09.hrit"
  [XRIT] "IMG_FD_038_SW038_20200602_061636_09.hrit"
  [XRIT] "IMG_FD_038_IR105_20200602_061636_09.hrit"
  [XRIT] "IMG_FD_038_IR123_20200602_061636_09.hrit"
    LENGTH:     ERROR (EXPECTED: 7175426, ACTUAL: 7167236, DIFF: -8190)
    SKIPPING FILE DUE TO DROPPED PACKETS
  [XRIT] "IMG_FD_038_IR123_20200602_061636_10.hrit"
  [JPEG] "IMG_FD_038_IR123_20200602_061636.jpg"
  [XRIT] "IMG_FD_038_IR105_20200602_061636_10.hrit"
  [JPEG] "IMG_FD_038_IR105_20200602_061636.jpg"
  [XRIT] "IMG_FD_038_WV069_20200602_061636_10.hrit"
  [JPEG] "IMG_FD_038_WV069_20200602_061636.jpg"
  [XRIT] "IMG_FD_038_SW038_20200602_061636_10.hrit"
  [JPEG] "IMG_FD_038_SW038_20200602_061636.jpg"
  [XRIT] "IMG_FD_038_VI006_20200602_061636_10.hrit"
  [JPEG] "IMG_FD_038_VI006_20200602_061636.jpg"

[VCID 63] GK-2A: IDLE

(lines with [JPEG] in last section is output of my script).

It is suspicious that the number of missing bytes is the same in all segments (2 CP_PDU) except the first and last, where it is one CP_PDU. Upd: for some sessions, diff can be -8190 for other segments too

IMG_FD_143_VI006_20200601_235636

kisaa commented 4 years ago

New info about CPPDU loss:

after adding CPPDU counter to TP_file and checking its value in Demuxer.Channel.handle_CPPDU, have this log:

[VCID 0] GK-2A: FULL DISK
    CPPDU COUNTER:    FIRST# 0
    CPPDU COUNTER:    LAST# 30
  [XRIT] "IMG_FD_026_SW038_20200605_041636_01.hrit"
    CPPDU COUNTER:    FIRST# 31
    CPPDU COUNTER:    LAST# 54
  [XRIT] "IMG_FD_026_WV069_20200605_041636_01.hrit"
    CPPDU COUNTER:    FIRST# 55
    CPPDU COUNTER:    LAST# 98
  [XRIT] "IMG_FD_026_IR105_20200605_041636_01.hrit"
    CPPDU COUNTER:    FIRST# 0
    CPPDU COUNTER:    LAST# 42
  [XRIT] "IMG_FD_026_IR123_20200605_041636_01.hrit"
    CPPDU COUNTER:    FIRST# 0
    CPPDU COUNTER (C): ERROR (EXPECTED: 441, ACTUAL: 442)
    CPPDU COUNTER:    LAST# 672
  [XRIT] "IMG_FD_026_VI006_20200605_041636_01.hrit"
    CPPDU COUNTER:    FIRST# 99
    CPPDU COUNTER:    LAST# 143
  [XRIT] "IMG_FD_026_SW038_20200605_041636_02.hrit"
    CPPDU COUNTER:    FIRST# 673
    CPPDU COUNTER:    LAST# 739
  [XRIT] "IMG_FD_026_IR105_20200605_041636_02.hrit"
    CPPDU COUNTER:    FIRST# 43
    CPPDU COUNTER:    LAST# 79
  [XRIT] "IMG_FD_026_WV069_20200605_041636_02.hrit"
    CPPDU COUNTER:    FIRST# 80
    CPPDU COUNTER:    LAST# 145
  [XRIT] "IMG_FD_026_IR123_20200605_041636_02.hrit"
    CPPDU COUNTER:    FIRST# 0
    CPPDU COUNTER (C): ERROR (EXPECTED: 441, ACTUAL: 442)
    CPPDU COUNTER (C): ERROR (EXPECTED: 883, ACTUAL: 884)
    CPPDU COUNTER:    LAST# 1019
  [XRIT] "IMG_FD_026_VI006_20200605_041636_02.hrit"
    CPPDU COUNTER:    FIRST# 144
    CPPDU COUNTER:    LAST# 188
  [XRIT] "IMG_FD_026_SW038_20200605_041636_03.hrit"
    CPPDU COUNTER:    FIRST# 740
    CPPDU COUNTER:    LAST# 781
  [XRIT] "IMG_FD_026_WV069_20200605_041636_03.hrit"
    CPPDU COUNTER:    FIRST# 189
    CPPDU COUNTER:    LAST# 258
  [XRIT] "IMG_FD_026_IR123_20200605_041636_03.hrit"
    CPPDU COUNTER:    FIRST# 782
    CPPDU COUNTER:    LAST# 854
  [XRIT] "IMG_FD_026_IR105_20200605_041636_03.hrit"
    CPPDU COUNTER:    FIRST# 855
    CPPDU COUNTER (C): ERROR (EXPECTED: 1296, ACTUAL: 1297)
    CPPDU COUNTER (C): ERROR (EXPECTED: 1738, ACTUAL: 1739)
    CPPDU COUNTER:    LAST# 2029
  [XRIT] "IMG_FD_026_VI006_20200605_041636_03.hrit"
    CPPDU COUNTER:    FIRST# 0
    CPPDU COUNTER:    LAST# 70
  [XRIT] "IMG_FD_026_IR105_20200605_041636_04.hrit"
    CPPDU COUNTER:    FIRST# 146
    CPPDU COUNTER:    LAST# 188
  [XRIT] "IMG_FD_026_SW038_20200605_041636_04.hrit"
    CPPDU COUNTER:    FIRST# 189
    CPPDU COUNTER:    LAST# 255
  [XRIT] "IMG_FD_026_IR123_20200605_041636_04.hrit"
    CPPDU COUNTER:    FIRST# 256
    CPPDU COUNTER:    LAST# 294
  [XRIT] "IMG_FD_026_WV069_20200605_041636_04.hrit"
    CPPDU COUNTER:    FIRST# 1020
    CPPDU COUNTER (C): ERROR (EXPECTED: 1461, ACTUAL: 1462)
    CPPDU COUNTER (C): ERROR (EXPECTED: 1903, ACTUAL: 1904)
    CPPDU COUNTER:    LAST# 2210
  [XRIT] "IMG_FD_026_VI006_20200605_041636_04.hrit"
    CPPDU COUNTER:    FIRST# 259
    CPPDU COUNTER:    LAST# 335
  [XRIT] "IMG_FD_026_IR123_20200605_041636_05.hrit"
    CPPDU COUNTER:    FIRST# 2211
    CPPDU COUNTER:    LAST# 2290
  [XRIT] "IMG_FD_026_IR105_20200605_041636_05.hrit"
    CPPDU COUNTER:    FIRST# 336
    CPPDU COUNTER:    LAST# 381
  [XRIT] "IMG_FD_026_SW038_20200605_041636_05.hrit"
    CPPDU COUNTER:    FIRST# 71
    CPPDU COUNTER:    LAST# 118
  [XRIT] "IMG_FD_026_WV069_20200605_041636_05.hrit"
    CPPDU COUNTER:    FIRST# 2030
    CPPDU COUNTER (C): ERROR (EXPECTED: 2471, ACTUAL: 2472)
    CPPDU COUNTER (C): ERROR (EXPECTED: 2913, ACTUAL: 2914)
    CPPDU COUNTER:    LAST# 3286
  [XRIT] "IMG_FD_026_VI006_20200605_041636_05.hrit"
    CPPDU COUNTER:    FIRST# 382
    CPPDU COUNTER:    LAST# 425
  [XRIT] "IMG_FD_026_WV069_20200605_041636_06.hrit"
    CPPDU COUNTER:    FIRST# 3287
    CPPDU COUNTER:    LAST# 3364
  [XRIT] "IMG_FD_026_IR105_20200605_041636_06.hrit"
    CPPDU COUNTER:    FIRST# 3365
    CPPDU COUNTER:    LAST# 3439
  [XRIT] "IMG_FD_026_IR123_20200605_041636_06.hrit"
    CPPDU COUNTER:    FIRST# 119
    CPPDU COUNTER:    LAST# 163
  [XRIT] "IMG_FD_026_SW038_20200605_041636_06.hrit"
    CPPDU COUNTER:    FIRST# 295
    CPPDU COUNTER (C): ERROR (EXPECTED: 736, ACTUAL: 737)
    CPPDU COUNTER (C): ERROR (EXPECTED: 1178, ACTUAL: 1179)
    CPPDU COUNTER:    LAST# 1561
  [XRIT] "IMG_FD_026_VI006_20200605_041636_06.hrit"
    CPPDU COUNTER:    FIRST# 1562
    CPPDU COUNTER:    LAST# 1605
  [XRIT] "IMG_FD_026_SW038_20200605_041636_07.hrit"
    CPPDU COUNTER:    FIRST# 2291
    CPPDU COUNTER:    LAST# 2330
  [XRIT] "IMG_FD_026_WV069_20200605_041636_07.hrit"
    CPPDU COUNTER:    FIRST# 2331
    CPPDU COUNTER:    LAST# 2400
  [XRIT] "IMG_FD_026_IR123_20200605_041636_07.hrit"
    CPPDU COUNTER:    FIRST# 426
    CPPDU COUNTER:    LAST# 499
  [XRIT] "IMG_FD_026_IR105_20200605_041636_07.hrit"
    CPPDU COUNTER:    FIRST# 2401
    CPPDU COUNTER (C): ERROR (EXPECTED: 2842, ACTUAL: 2843)
    CPPDU COUNTER (C): ERROR (EXPECTED: 3284, ACTUAL: 3285)
    CPPDU COUNTER:    LAST# 3615
  [XRIT] "IMG_FD_026_VI006_20200605_041636_07.hrit"
    CPPDU COUNTER:    FIRST# 500
    CPPDU COUNTER:    LAST# 541
  [XRIT] "IMG_FD_026_SW038_20200605_041636_08.hrit"
    CPPDU COUNTER:    FIRST# 3440
    CPPDU COUNTER:    LAST# 3508
  [XRIT] "IMG_FD_026_IR105_20200605_041636_08.hrit"
    CPPDU COUNTER:    FIRST# 164
    CPPDU COUNTER:    LAST# 193
  [XRIT] "IMG_FD_026_WV069_20200605_041636_08.hrit"
    CPPDU COUNTER:    FIRST# 542
    CPPDU COUNTER:    LAST# 607
  [XRIT] "IMG_FD_026_IR123_20200605_041636_08.hrit"
    CPPDU COUNTER:    FIRST# 3509
    CPPDU COUNTER (C): ERROR (EXPECTED: 3950, ACTUAL: 3951)
    CPPDU COUNTER (C): ERROR (EXPECTED: 4392, ACTUAL: 4393)
    CPPDU COUNTER:    LAST# 4617
  [XRIT] "IMG_FD_026_VI006_20200605_041636_08.hrit"
    CPPDU COUNTER:    FIRST# 4618
    CPPDU COUNTER:    LAST# 4682
  [XRIT] "IMG_FD_026_IR105_20200605_041636_09.hrit"
    CPPDU COUNTER:    FIRST# 1606
    CPPDU COUNTER:    LAST# 1641
  [XRIT] "IMG_FD_026_SW038_20200605_041636_09.hrit"
    CPPDU COUNTER:    FIRST# 608
    CPPDU COUNTER:    LAST# 670
  [XRIT] "IMG_FD_026_IR123_20200605_041636_09.hrit"
    CPPDU COUNTER:    FIRST# 1642
    CPPDU COUNTER:    LAST# 1672
  [XRIT] "IMG_FD_026_WV069_20200605_041636_09.hrit"
    CPPDU COUNTER:    FIRST# 3616
    CPPDU COUNTER (C): ERROR (EXPECTED: 4057, ACTUAL: 4058)
    CPPDU COUNTER (C): ERROR (EXPECTED: 4499, ACTUAL: 4500)
    CPPDU COUNTER:    LAST# 4596
  [XRIT] "IMG_FD_026_VI006_20200605_041636_09.hrit"
    CPPDU COUNTER:    FIRST# 4597
    CPPDU COUNTER:    LAST# 4618
  [XRIT] "IMG_FD_026_WV069_20200605_041636_10.hrit"
  [JPEG] "IMG_FD_026_WV069_20200605_041636.jpg"
    CPPDU COUNTER:    FIRST# 4619
    CPPDU COUNTER:    LAST# 4639
  [XRIT] "IMG_FD_026_SW038_20200605_041636_10.hrit"
  [JPEG] "IMG_FD_026_SW038_20200605_041636.jpg"
    CPPDU COUNTER:    FIRST# 671
    CPPDU COUNTER:    LAST# 709
  [XRIT] "IMG_FD_026_IR105_20200605_041636_10.hrit"
  [JPEG] "IMG_FD_026_IR105_20200605_041636.jpg"
    CPPDU COUNTER:    FIRST# 710
    CPPDU COUNTER:    LAST# 747
  [XRIT] "IMG_FD_026_IR123_20200605_041636_10.hrit"
  [JPEG] "IMG_FD_026_IR123_20200605_041636.jpg"
    CPPDU COUNTER:    FIRST# 4683
    CPPDU COUNTER (C): ERROR (EXPECTED: 5124, ACTUAL: 5125)
    CPPDU COUNTER:    LAST# 5162
  [XRIT] "IMG_FD_026_VI006_20200605_041636_10.hrit"

As you can see, every 442th from beginning of the TPFile CPPDU is loss. So, we have loss 8190 bytes for each 3619980 bytes (8190*442) of file length. For large file, there will be data loss after 7239960, 10859940 bytes and so on. As IR-channel's hrit files are much less than 3.6M, they have no data loss at all.

sam210723 commented 4 years ago

Fantastic work tracking down all of these issues! I finally have some time to devote to HRIT support over the next few days. I'll be implementing your findings in the hrit branch where I've done some initial HRIT product output support.

Side note: What is your RF setup for receiving HRIT? (dish size, feed, amplifiers, filters)

kisaa commented 4 years ago

For HRIT we use meteorological receiving station based on 3-meter dish and circular-polarized feed-horn with built-in LNA. I know that GK-2A uses linear polarization, but I cannot change antenna's construction. Described above ETRA receiver is used for demodulation.

LRIT is received by 2-meter dish with self-built "cantenna" feed-horn (from drainpipe :) ) with linear polarization, Nooelec SAWBird GOES LNA/filter and Airspy R2 receiver.

I have not tested yet HRIT on 2m dish (cannot powerup LNA from ETRA by coax cable, need to bring additional power to the roof), but signal on 2m dish and Airspy seems to be powerful enough: hrit0 (no data transmission, empty frames VCID=63) hrit1 (data is transmitted, VCID=0)

Also, your goesrecv-monitor shows pretty good picture for LRIT: BPSK monitor

sam210723 commented 4 years ago

I agree, HRIT on the 2m does look strong enough. Do you have an IQ recording from the 2m dish I can download? I can try running it through my GNU Radio HRIT demodulator.

You could try ETRA on the 2m dish using an external bias-tee between the ETRA and SAWbird to inject 5V DC. A cheap eBay one should do the job. This way you won't have to run separate power to the antenna.

kisaa commented 4 years ago

These are IQ-files recorded in Gqrx (freq=1695.4 MHz, rate=10MSPS). They are HUGE (approx. 450M each for 5-seconds recording), one is with VCID=63 (no data) and other is VCID=0 (FD).

https://drive.google.com/file/d/1XQQAg5-DvX-Gz_TLhViHapV3tod4B8FJ/view?usp=sharing https://drive.google.com/file/d/1IiLuatFxN7dAqIZtt4ZlimQT4LzIQnl6/view?usp=sharing

If you can not replay these files, please, give me a tip how to record.

sam210723 commented 4 years ago

QPSK constellation is definitely visible in those recordings. I'm not sure if Viterbi + RS would be enough to recover any data from this signal.

GNU Radio HRIT Demodulator

Do you have a VCDU capture of VIS006 imagery? I'd like to tackle that CP_PDU issue after getting J2K image output working.

kisaa commented 4 years ago

Do you have a VCDU capture of VIS006 imagery?

Do you mean dumps by script --dump option (links in the first message, includes all five channels) or something else? I had re-process them, and there are exactly the same errors/losses.

sam210723 commented 4 years ago

Ah sorry, you are right. For some reason I was thinking they were not transmitted at night like old COMS-1 LRIT.

In that case I'll try adding an APID filter to the demuxer so I can work on just VIS006, then move on to J2K stuff.

kisaa commented 4 years ago

Something strange is happening with APID. It seems that all the channels are spread across all APIDs:

[VCID 0] GK-2A: FULL DISK
    APID#: 6
    CPPDU COUNTER:    FIRST# 0
    CPPDU COUNTER:    LAST# 24
  [XRIT] "IMG_FD_046_WV069_20200610_073636_01.hrit"
    APID#: 2
    CPPDU COUNTER:    FIRST# 0
    CPPDU COUNTER:    LAST# 41
  [XRIT] "IMG_FD_046_IR123_20200610_073636_01.hrit"
    APID#: 14
    CPPDU COUNTER:    FIRST# 0
    CPPDU COUNTER:    LAST# 26
  [XRIT] "IMG_FD_046_SW038_20200610_073636_01.hrit"
    APID#: 6
    CPPDU COUNTER:    FIRST# 25
    CPPDU COUNTER:    LAST# 67
  [XRIT] "IMG_FD_046_IR105_20200610_073636_01.hrit"
    APID#: 6
    CPPDU COUNTER:    FIRST# 68
    CPPDU COUNTER:    LAST# 692
  [XRIT] "IMG_FD_046_VI006_20200610_073636_01.hrit"
    APID#: 6
    CPPDU COUNTER:    FIRST# 693
    CPPDU COUNTER:    LAST# 760
  [XRIT] "IMG_FD_046_IR123_20200610_073636_02.hrit"
    APID#: 14
    CPPDU COUNTER:    FIRST# 27
    CPPDU COUNTER:    LAST# 65
  [XRIT] "IMG_FD_046_WV069_20200610_073636_02.hrit"
    APID#: 12
    CPPDU COUNTER:    FIRST# 0
    CPPDU COUNTER:    LAST# 41
  [XRIT] "IMG_FD_046_SW038_20200610_073636_02.hrit"
    APID#: 6
    CPPDU COUNTER:    FIRST# 761
    CPPDU COUNTER:    LAST# 829
  [XRIT] "IMG_FD_046_IR105_20200610_073636_02.hrit"
    APID#: 2
    CPPDU COUNTER:    FIRST# 42
    CPPDU COUNTER:    LAST# 1012
  [XRIT] "IMG_FD_046_VI006_20200610_073636_02.hrit"
    APID#: 6
    CPPDU COUNTER:    FIRST# 830
    CPPDU COUNTER:    LAST# 874
  [XRIT] "IMG_FD_046_WV069_20200610_073636_03.hrit"
    APID#: 14
    CPPDU COUNTER:    FIRST# 66
    CPPDU COUNTER:    LAST# 112
  [XRIT] "IMG_FD_046_SW038_20200610_073636_03.hrit"
    APID#: 6
    CPPDU COUNTER:    FIRST# 875
    CPPDU COUNTER:    LAST# 950
  [XRIT] "IMG_FD_046_IR123_20200610_073636_03.hrit"
    APID#: 8
    CPPDU COUNTER:    FIRST# 0
    CPPDU COUNTER:    LAST# 78
  [XRIT] "IMG_FD_046_IR105_20200610_073636_03.hrit"
    APID#: 14
    CPPDU COUNTER:    FIRST# 113
    CPPDU COUNTER:    LAST# 1248
  [XRIT] "IMG_FD_046_VI006_20200610_073636_03.hrit"
    APID#: 6
    CPPDU COUNTER:    FIRST# 951
    CPPDU COUNTER:    LAST# 999
  [XRIT] "IMG_FD_046_WV069_20200610_073636_04.hrit"
    APID#: 12
    CPPDU COUNTER:    FIRST# 42
    CPPDU COUNTER:    LAST# 123
  [XRIT] "IMG_FD_046_IR105_20200610_073636_04.hrit"
    APID#: 12
    CPPDU COUNTER:    FIRST# 124
    CPPDU COUNTER:    LAST# 168
  [XRIT] "IMG_FD_046_SW038_20200610_073636_04.hrit"
    APID#: 6
    CPPDU COUNTER:    FIRST# 1000
    CPPDU COUNTER:    LAST# 1078
  [XRIT] "IMG_FD_046_IR123_20200610_073636_04.hrit"
    APID#: 2
    CPPDU COUNTER:    FIRST# 1013
    CPPDU COUNTER:    LAST# 2068
  [XRIT] "IMG_FD_046_VI006_20200610_073636_04.hrit"

and so on

sam210723 commented 4 years ago

KMA have a habit of not following their own specifications. I noticed this when working on LRIT.

It could still be a demuxer issue since I'm completely ignoring APIDs at the moment. It was done this way because the APID was useless with COMS-1 LRIT when the demuxer core was written. It used to be the VCID*64 so was of no real benefit.

I will need to track APIDs now with HRIT because the FD wavelength segments are interleaved which complicates product generation.

APID Table

sam210723 commented 4 years ago

Confirmed CP_PDU #441 is having issues, and the APIDs are completely useless. The CP_PDU header is there but it never gets marked as completed which is needed to trigger the payload length and CRC check, then its meant to be appended to the current TP_File.

Time to delve into the demuxer core...

[VCID 0] GK-2A: FULL DISK
  [XRIT] "IMG_FD_026_IR123_20200514_041636_01.hrit"
  [XRIT] "IMG_FD_026_SW038_20200514_041636_01.hrit"
  [XRIT] "IMG_FD_026_WV069_20200514_041636_01.hrit"
  [XRIT] "IMG_FD_026_IR105_20200514_041636_01.hrit"

  [CP_PDU] APID: 14   SEQ: CONTINUE   #439   LEN: 8192
    [TP_File]  CURRENT LEN: 3603590 (71%)     EXPECTED LEN: 5079906     DIFF: 1476316

  [CP_PDU] APID: 14   SEQ: CONTINUE   #440   LEN: 8192
    [TP_File]  CURRENT LEN: 3611780 (71%)     EXPECTED LEN: 5079906     DIFF: 1468126

  [CP_PDU] APID: 14   SEQ: CONTINUE   #442   LEN: 8192
    [TP_File]  CURRENT LEN: 3619970 (71%)     EXPECTED LEN: 5079906     DIFF: 1459936

  [CP_PDU] APID: 14   SEQ: CONTINUE   #443   LEN: 8192
    [TP_File]  CURRENT LEN: 3628160 (71%)     EXPECTED LEN: 5079906     DIFF: 1451746

  [CP_PDU] APID: 14   SEQ: CONTINUE   #444   LEN: 8192
    [TP_File]  CURRENT LEN: 3636350 (72%)     EXPECTED LEN: 5079906     DIFF: 1443556
    SKIPPING FILE DUE TO DROPPED PACKETS
  [XRIT] "IMG_FD_026_SW038_20200514_041636_02.hrit"
  [XRIT] "IMG_FD_026_IR123_20200514_041636_02.hrit"
[CP_PDU] APID: 14   SEQ: CONTINUE   #439   LEN: 8192
    HEADER:     0x080E01B71FFF
    OFFSET:     0x9E
    .........
    LENGTH:     OK
    CRC:        OK

    [TP_File]  CURRENT LEN: 3603590 (71%)     EXPECTED LEN: 5079906     DIFF: 1476316

  [CP_PDU] APID: 14   SEQ: CONTINUE   #440   LEN: 8192
    HEADER:     0x080E01B81FFF
    OFFSET:     0x190
    .........
    LENGTH:     OK
    CRC:        OK

    [TP_File]  CURRENT LEN: 3611780 (71%)     EXPECTED LEN: 5079906     DIFF: 1468126

  [CP_PDU] APID: 14   SEQ: CONTINUE   #441   LEN: 8192
    HEADER:     0x080E01B91FFF
    OFFSET:     0x282
    ..........  [CP_PDU] APID: 14   SEQ: CONTINUE   #442   LEN: 8192
    HEADER:     0x080E01BA1FFF
    OFFSET:     0x0
    .........
    LENGTH:     OK
    CRC:        OK

    [TP_File]  CURRENT LEN: 3619970 (71%)     EXPECTED LEN: 5079906     DIFF: 1459936

  [CP_PDU] APID: 14   SEQ: CONTINUE   #443   LEN: 8192
    HEADER:     0x080E01BB1FFF
    OFFSET:     0xF2
    .........
    LENGTH:     OK
    CRC:        OK

    [TP_File]  CURRENT LEN: 3628160 (71%)     EXPECTED LEN: 5079906     DIFF: 1451746

  [CP_PDU] APID: 14   SEQ: CONTINUE   #444   LEN: 8192
    HEADER:     0x080E01BC1FFF
    OFFSET:     0x1E4
    .........
    LENGTH:     OK
    CRC:        OK

    [TP_File]  CURRENT LEN: 3636350 (72%)     EXPECTED LEN: 5079906     DIFF: 1443556
sam210723 commented 4 years ago

Found it! The pointer field in the M_PDU header points to the location of the CP_PDU header within the current M_PDU data field.

I was foolishly using that header value to trigger the start of a new TP_File, which works for small files. But once the CP_PDU header lines up again with the start of the M_PDU data field (which it does-so every 442 CP_PDUs) the previous CP_PDU is discarded and a new one is started.

TP_File trigger

I've re-arranged that section of the demuxer to now check if there's an unfinished TP_File rather than assume an offset of zero is the start of a new TP_File. All 50 HRIT files for that FD are now being saved correctly.

File output

sam210723 commented 4 years ago

And we have complete Full Disks! I've re-written hrit-img.py based on lrit-img.py but using a pre-compiled Windows binary of libjpeg to convert the J2K payload to PPM, which is then loaded by Pillow in Python for segment merging and saving as a JPG.

Full disk images

Couldn't help myself. I had to render a false colour with GeoSatSignal

False colour

False colour crop False colour crop

I'll adapt the changes in hrit-img.py into the Products class of xrit-rx tomorrow so the image generation and merging is automatic.

kisaa commented 4 years ago

Wow! That's fantastic! :D

sam210723 commented 4 years ago

I've now got xrit-rx saving assembled HRIT full disk images without saving .hrit files to disk. It does save .jp2 files temporarily but they're deleted once libjpeg creates the .ppm file. Once Pillow has loaded the .ppm image into memory it is also deleted from disk, leaving only the assembled FDs of all five wavelengths.

File output

All that's left for HRIT support is to update the progress bar indicator to handle multiple wavelengths.

I may add a UDP socket input option before next release so you can easily try it on the real-time downlink using the ETRA modem. I'll also need to work out licensing of libjpeg since it's GPLv3 and xrit-rx is MIT.

sam210723 commented 4 years ago

Progress bar now shows each wavelength individually.

Progress bar

Changes merged into dev branch in PR #17 . Will be included in next release.

sam210723 commented 4 years ago

Just released v1.3 with all the HRIT changes and a UDP input option for your hardware modem. Let me know how the UDP input goes!

sergsero commented 4 years ago

Hello, Based on the actual experience of receiving signal from GK-2A, what is the minimum diameter of the satellite dish suitable for receiving a HRIT signal at the frequency 1695.4 MHz?

My attempts have not yet been successful with the prime focus 2.3 m mesh antenna, although the signal looks quite sufficient in sdrsharp: 2020-09-04T06:14:37Z [monitor] gain: 41.09, freq: 183474.8, omega: 3.333, vit(avg): 2112, rs(sum): 0, packets: 0, drops: 112 2020-09-04T06:14:38Z [monitor] gain: 41.11, freq: 182697.4, omega: 3.333, vit(avg): 2111, rs(sum): 0, packets: 0, drops: 110 2020-09-04T06:14:39Z [monitor] gain: 41.13, freq: 176119.0, omega: 3.333, vit(avg): 2109, rs(sum): 0, packets: 0, drops: 105 ...

sam210723 commented 4 years ago

Although xrit-rx is ready for HRIT, goesrecv is not yet. The HRIT downlink from GK-2A uses QPSK modulation and LRIT uses BPSK. I have been working on adding the necessary components to goesrecv for QPSK demodulation but have not finished this yet.

I've been posting progress updates to Twitter: https://twitter.com/sam210723/status/1274632866526191616. Here I'm testing with FY-4A LRIT which is also QPSK.

2.3m may be large enough for HRIT but I won't be able to say for sure until we have a working receive chain.

sergsero commented 4 years ago

Thank you so much for your response and for your brilliant work! I successfully use your programs to get images from GK-2A in LRIT and will follow the news about goesrecv with QPSK demodulation with great interest.

20200904_goesrecv_gk-2a_lrit

20200903_GEO-KOMPSAT-2A

sam210723 commented 4 years ago

Glad to hear xrit-rx and goesrecv-monitor are working well for you! Great looking signal there on your 2.3m dish. I will try remember to reply to this issue when HRIT demodulation/decoding is working.

adamg261 commented 3 years ago

Hi Sam, and everyone who has contributed to this fantastic outcome. Great work and talent!

Cheers