Open kisaa opened 4 years ago
I haven't been able to test HRIT since goesrecv has not been modified for QPSK yet. What is your demodulator and decoder chain?
Thanks for the VCDU files, I'll take a look!
HRIT headers are intact but the JPEG/JFIF headers are broken for some reason. There does seem to be a bigger demuxing issue so I'll take a look at that first, then look at the JPEG issues.
What is your demodulator and decoder chain?
This is a professional meteorological receiver, we used it to get HRIT / LRIT with Electro-L. It supports Viterbi and Reed-Solomon decoding and provides 1024 (full frame with sync word), 1020 (CVCDU) or 892 (VCDU) bytes packets over Ethernet. I changed the main script a bit to get VCDUs from UDP socket with the correct interleaving settings. Unfortunately, my skills are not enough to write a QPSK software demodulator ( Maybe later I can adapt https://github.com/dbdexter-dev/meteor_demod for these needs - it is written for QPSK / OQPSK.
Nice! is it a Dartcom receiver?
I have started work on a QPSK DSP chain using GNURadio source for reference. Using LRPT code as well is a good idea! I'll be getting back to that soon.
is it a Dartcom receiver?
No, this is a Russian model produced by http://www.etra-plus.ru/ (D8L, could not be found on the site, possibly EOL)
Interesting, I thought it looked similar to the Comtech CDM modem series, then I found this on the D4L product page:
(Comtech CDM-600)
(ETRA-D4L)
The JPEG headers in these images seem to be malformed and don't have enough info to recover the image data.
They are basically empty when compared to a normal JPEG/JFIF header. I'm going through the HRIT documentation at the moment to see if there's anything explaining this.
According to http://nmsc.kma.go.kr/resources/homepage/pdf/GK2A_HRIT_Mission_Specification_Documen_v1.0.pdf, p.24, JPEG2000 is used, not regular JPEG. But header looks untypical even for JPEG2000.
This makes sense as Pillow was throwing exceptions when loading the image. It needs OpenJPEG installed and in PATH to handle J2K.
It also mentions they're using J2K in a lossless configuration so that could explain the sparse headers.
Got images from IR channels!
Decompressed HRIT data using jpeg
utility from https://github.com/thorfdbg/libjpeg, with intermediate conversion to PPM and regular JPG and building all segments together, by (dirty, sorry for this!) calling hrit2jpg.processHRIT()
from demuxer.py (Channel.handle_xRIT) after 10th segment is saved to disk:
else:
# Print XRIT file info
xrit.print_info()
if xrit.FILE_NAME.endswith('10.hrit'):
hrit2jpg.proceedHRIT(xrit.get_save_path(self.config.output))
Still having troubles with VIS channel ((
Nice work! I'll look at streamlining that process then integrate it into xrit-rx.
I left the system running overnight, and today I discovered the following:
sometimes (I have no idea about the pattern) one or two segments of the VIS channel are saved successfully, and then the image is successfully (partly) decoded.
The rest of the time, I get the following message from the handle_CPPDU
function (I changed the line if self.config.verbose:
to if True:
to see only these debug messages):
[VCID 0] GK-2A: FULL DISK
[XRIT] "IMG_FD_038_IR105_20200602_061636_01.hrit"
[XRIT] "IMG_FD_038_SW038_20200602_061636_01.hrit"
[XRIT] "IMG_FD_038_WV069_20200602_061636_01.hrit"
[XRIT] "IMG_FD_038_IR123_20200602_061636_01.hrit"
LENGTH: ERROR (EXPECTED: 5282322, ACTUAL: 5274132, DIFF: -8190)
SKIPPING FILE DUE TO DROPPED PACKETS
[XRIT] "IMG_FD_038_WV069_20200602_061636_02.hrit"
[XRIT] "IMG_FD_038_SW038_20200602_061636_02.hrit"
[XRIT] "IMG_FD_038_IR123_20200602_061636_02.hrit"
[XRIT] "IMG_FD_038_IR105_20200602_061636_02.hrit"
LENGTH: ERROR (EXPECTED: 8108234, ACTUAL: 8091854, DIFF: -16380)
SKIPPING FILE DUE TO DROPPED PACKETS
[XRIT] "IMG_FD_038_IR105_20200602_061636_03.hrit"
[XRIT] "IMG_FD_038_SW038_20200602_061636_03.hrit"
[XRIT] "IMG_FD_038_IR123_20200602_061636_03.hrit"
[XRIT] "IMG_FD_038_WV069_20200602_061636_03.hrit"
LENGTH: ERROR (EXPECTED: 9413834, ACTUAL: 9397454, DIFF: -16380)
SKIPPING FILE DUE TO DROPPED PACKETS
[XRIT] "IMG_FD_038_IR105_20200602_061636_04.hrit"
[XRIT] "IMG_FD_038_WV069_20200602_061636_04.hrit"
[XRIT] "IMG_FD_038_IR123_20200602_061636_04.hrit"
[XRIT] "IMG_FD_038_SW038_20200602_061636_04.hrit"
LENGTH: ERROR (EXPECTED: 9086738, ACTUAL: 9070358, DIFF: -16380)
SKIPPING FILE DUE TO DROPPED PACKETS
[XRIT] "IMG_FD_038_IR105_20200602_061636_05.hrit"
[XRIT] "IMG_FD_038_SW038_20200602_061636_05.hrit"
[XRIT] "IMG_FD_038_WV069_20200602_061636_05.hrit"
[XRIT] "IMG_FD_038_IR123_20200602_061636_05.hrit"
LENGTH: ERROR (EXPECTED: 9166682, ACTUAL: 9150302, DIFF: -16380)
SKIPPING FILE DUE TO DROPPED PACKETS
[XRIT] "IMG_FD_038_WV069_20200602_061636_06.hrit"
[XRIT] "IMG_FD_038_IR123_20200602_061636_06.hrit"
[XRIT] "IMG_FD_038_IR105_20200602_061636_06.hrit"
[XRIT] "IMG_FD_038_SW038_20200602_061636_06.hrit"
LENGTH: ERROR (EXPECTED: 9326610, ACTUAL: 9310230, DIFF: -16380)
SKIPPING FILE DUE TO DROPPED PACKETS
[XRIT] "IMG_FD_038_SW038_20200602_061636_07.hrit"
[XRIT] "IMG_FD_038_WV069_20200602_061636_07.hrit"
[XRIT] "IMG_FD_038_IR123_20200602_061636_07.hrit"
[XRIT] "IMG_FD_038_IR105_20200602_061636_07.hrit"
LENGTH: ERROR (EXPECTED: 8890946, ACTUAL: 8874566, DIFF: -16380)
SKIPPING FILE DUE TO DROPPED PACKETS
[XRIT] "IMG_FD_038_IR123_20200602_061636_08.hrit"
[XRIT] "IMG_FD_038_IR105_20200602_061636_08.hrit"
[XRIT] "IMG_FD_038_SW038_20200602_061636_08.hrit"
[XRIT] "IMG_FD_038_WV069_20200602_061636_08.hrit"
LENGTH: ERROR (EXPECTED: 7802050, ACTUAL: 7785670, DIFF: -16380)
SKIPPING FILE DUE TO DROPPED PACKETS
[XRIT] "IMG_FD_038_WV069_20200602_061636_09.hrit"
[XRIT] "IMG_FD_038_SW038_20200602_061636_09.hrit"
[XRIT] "IMG_FD_038_IR105_20200602_061636_09.hrit"
[XRIT] "IMG_FD_038_IR123_20200602_061636_09.hrit"
LENGTH: ERROR (EXPECTED: 7175426, ACTUAL: 7167236, DIFF: -8190)
SKIPPING FILE DUE TO DROPPED PACKETS
[XRIT] "IMG_FD_038_IR123_20200602_061636_10.hrit"
[JPEG] "IMG_FD_038_IR123_20200602_061636.jpg"
[XRIT] "IMG_FD_038_IR105_20200602_061636_10.hrit"
[JPEG] "IMG_FD_038_IR105_20200602_061636.jpg"
[XRIT] "IMG_FD_038_WV069_20200602_061636_10.hrit"
[JPEG] "IMG_FD_038_WV069_20200602_061636.jpg"
[XRIT] "IMG_FD_038_SW038_20200602_061636_10.hrit"
[JPEG] "IMG_FD_038_SW038_20200602_061636.jpg"
[XRIT] "IMG_FD_038_VI006_20200602_061636_10.hrit"
[JPEG] "IMG_FD_038_VI006_20200602_061636.jpg"
[VCID 63] GK-2A: IDLE
(lines with [JPEG] in last section is output of my script).
It is suspicious that the number of missing bytes is the same in all segments (2 CP_PDU) except the first and last, where it is one CP_PDU.
Upd: for some sessions, diff
can be -8190 for other segments too
New info about CPPDU loss:
after adding CPPDU counter to TP_file
and checking its value in Demuxer.Channel.handle_CPPDU
, have this log:
[VCID 0] GK-2A: FULL DISK
CPPDU COUNTER: FIRST# 0
CPPDU COUNTER: LAST# 30
[XRIT] "IMG_FD_026_SW038_20200605_041636_01.hrit"
CPPDU COUNTER: FIRST# 31
CPPDU COUNTER: LAST# 54
[XRIT] "IMG_FD_026_WV069_20200605_041636_01.hrit"
CPPDU COUNTER: FIRST# 55
CPPDU COUNTER: LAST# 98
[XRIT] "IMG_FD_026_IR105_20200605_041636_01.hrit"
CPPDU COUNTER: FIRST# 0
CPPDU COUNTER: LAST# 42
[XRIT] "IMG_FD_026_IR123_20200605_041636_01.hrit"
CPPDU COUNTER: FIRST# 0
CPPDU COUNTER (C): ERROR (EXPECTED: 441, ACTUAL: 442)
CPPDU COUNTER: LAST# 672
[XRIT] "IMG_FD_026_VI006_20200605_041636_01.hrit"
CPPDU COUNTER: FIRST# 99
CPPDU COUNTER: LAST# 143
[XRIT] "IMG_FD_026_SW038_20200605_041636_02.hrit"
CPPDU COUNTER: FIRST# 673
CPPDU COUNTER: LAST# 739
[XRIT] "IMG_FD_026_IR105_20200605_041636_02.hrit"
CPPDU COUNTER: FIRST# 43
CPPDU COUNTER: LAST# 79
[XRIT] "IMG_FD_026_WV069_20200605_041636_02.hrit"
CPPDU COUNTER: FIRST# 80
CPPDU COUNTER: LAST# 145
[XRIT] "IMG_FD_026_IR123_20200605_041636_02.hrit"
CPPDU COUNTER: FIRST# 0
CPPDU COUNTER (C): ERROR (EXPECTED: 441, ACTUAL: 442)
CPPDU COUNTER (C): ERROR (EXPECTED: 883, ACTUAL: 884)
CPPDU COUNTER: LAST# 1019
[XRIT] "IMG_FD_026_VI006_20200605_041636_02.hrit"
CPPDU COUNTER: FIRST# 144
CPPDU COUNTER: LAST# 188
[XRIT] "IMG_FD_026_SW038_20200605_041636_03.hrit"
CPPDU COUNTER: FIRST# 740
CPPDU COUNTER: LAST# 781
[XRIT] "IMG_FD_026_WV069_20200605_041636_03.hrit"
CPPDU COUNTER: FIRST# 189
CPPDU COUNTER: LAST# 258
[XRIT] "IMG_FD_026_IR123_20200605_041636_03.hrit"
CPPDU COUNTER: FIRST# 782
CPPDU COUNTER: LAST# 854
[XRIT] "IMG_FD_026_IR105_20200605_041636_03.hrit"
CPPDU COUNTER: FIRST# 855
CPPDU COUNTER (C): ERROR (EXPECTED: 1296, ACTUAL: 1297)
CPPDU COUNTER (C): ERROR (EXPECTED: 1738, ACTUAL: 1739)
CPPDU COUNTER: LAST# 2029
[XRIT] "IMG_FD_026_VI006_20200605_041636_03.hrit"
CPPDU COUNTER: FIRST# 0
CPPDU COUNTER: LAST# 70
[XRIT] "IMG_FD_026_IR105_20200605_041636_04.hrit"
CPPDU COUNTER: FIRST# 146
CPPDU COUNTER: LAST# 188
[XRIT] "IMG_FD_026_SW038_20200605_041636_04.hrit"
CPPDU COUNTER: FIRST# 189
CPPDU COUNTER: LAST# 255
[XRIT] "IMG_FD_026_IR123_20200605_041636_04.hrit"
CPPDU COUNTER: FIRST# 256
CPPDU COUNTER: LAST# 294
[XRIT] "IMG_FD_026_WV069_20200605_041636_04.hrit"
CPPDU COUNTER: FIRST# 1020
CPPDU COUNTER (C): ERROR (EXPECTED: 1461, ACTUAL: 1462)
CPPDU COUNTER (C): ERROR (EXPECTED: 1903, ACTUAL: 1904)
CPPDU COUNTER: LAST# 2210
[XRIT] "IMG_FD_026_VI006_20200605_041636_04.hrit"
CPPDU COUNTER: FIRST# 259
CPPDU COUNTER: LAST# 335
[XRIT] "IMG_FD_026_IR123_20200605_041636_05.hrit"
CPPDU COUNTER: FIRST# 2211
CPPDU COUNTER: LAST# 2290
[XRIT] "IMG_FD_026_IR105_20200605_041636_05.hrit"
CPPDU COUNTER: FIRST# 336
CPPDU COUNTER: LAST# 381
[XRIT] "IMG_FD_026_SW038_20200605_041636_05.hrit"
CPPDU COUNTER: FIRST# 71
CPPDU COUNTER: LAST# 118
[XRIT] "IMG_FD_026_WV069_20200605_041636_05.hrit"
CPPDU COUNTER: FIRST# 2030
CPPDU COUNTER (C): ERROR (EXPECTED: 2471, ACTUAL: 2472)
CPPDU COUNTER (C): ERROR (EXPECTED: 2913, ACTUAL: 2914)
CPPDU COUNTER: LAST# 3286
[XRIT] "IMG_FD_026_VI006_20200605_041636_05.hrit"
CPPDU COUNTER: FIRST# 382
CPPDU COUNTER: LAST# 425
[XRIT] "IMG_FD_026_WV069_20200605_041636_06.hrit"
CPPDU COUNTER: FIRST# 3287
CPPDU COUNTER: LAST# 3364
[XRIT] "IMG_FD_026_IR105_20200605_041636_06.hrit"
CPPDU COUNTER: FIRST# 3365
CPPDU COUNTER: LAST# 3439
[XRIT] "IMG_FD_026_IR123_20200605_041636_06.hrit"
CPPDU COUNTER: FIRST# 119
CPPDU COUNTER: LAST# 163
[XRIT] "IMG_FD_026_SW038_20200605_041636_06.hrit"
CPPDU COUNTER: FIRST# 295
CPPDU COUNTER (C): ERROR (EXPECTED: 736, ACTUAL: 737)
CPPDU COUNTER (C): ERROR (EXPECTED: 1178, ACTUAL: 1179)
CPPDU COUNTER: LAST# 1561
[XRIT] "IMG_FD_026_VI006_20200605_041636_06.hrit"
CPPDU COUNTER: FIRST# 1562
CPPDU COUNTER: LAST# 1605
[XRIT] "IMG_FD_026_SW038_20200605_041636_07.hrit"
CPPDU COUNTER: FIRST# 2291
CPPDU COUNTER: LAST# 2330
[XRIT] "IMG_FD_026_WV069_20200605_041636_07.hrit"
CPPDU COUNTER: FIRST# 2331
CPPDU COUNTER: LAST# 2400
[XRIT] "IMG_FD_026_IR123_20200605_041636_07.hrit"
CPPDU COUNTER: FIRST# 426
CPPDU COUNTER: LAST# 499
[XRIT] "IMG_FD_026_IR105_20200605_041636_07.hrit"
CPPDU COUNTER: FIRST# 2401
CPPDU COUNTER (C): ERROR (EXPECTED: 2842, ACTUAL: 2843)
CPPDU COUNTER (C): ERROR (EXPECTED: 3284, ACTUAL: 3285)
CPPDU COUNTER: LAST# 3615
[XRIT] "IMG_FD_026_VI006_20200605_041636_07.hrit"
CPPDU COUNTER: FIRST# 500
CPPDU COUNTER: LAST# 541
[XRIT] "IMG_FD_026_SW038_20200605_041636_08.hrit"
CPPDU COUNTER: FIRST# 3440
CPPDU COUNTER: LAST# 3508
[XRIT] "IMG_FD_026_IR105_20200605_041636_08.hrit"
CPPDU COUNTER: FIRST# 164
CPPDU COUNTER: LAST# 193
[XRIT] "IMG_FD_026_WV069_20200605_041636_08.hrit"
CPPDU COUNTER: FIRST# 542
CPPDU COUNTER: LAST# 607
[XRIT] "IMG_FD_026_IR123_20200605_041636_08.hrit"
CPPDU COUNTER: FIRST# 3509
CPPDU COUNTER (C): ERROR (EXPECTED: 3950, ACTUAL: 3951)
CPPDU COUNTER (C): ERROR (EXPECTED: 4392, ACTUAL: 4393)
CPPDU COUNTER: LAST# 4617
[XRIT] "IMG_FD_026_VI006_20200605_041636_08.hrit"
CPPDU COUNTER: FIRST# 4618
CPPDU COUNTER: LAST# 4682
[XRIT] "IMG_FD_026_IR105_20200605_041636_09.hrit"
CPPDU COUNTER: FIRST# 1606
CPPDU COUNTER: LAST# 1641
[XRIT] "IMG_FD_026_SW038_20200605_041636_09.hrit"
CPPDU COUNTER: FIRST# 608
CPPDU COUNTER: LAST# 670
[XRIT] "IMG_FD_026_IR123_20200605_041636_09.hrit"
CPPDU COUNTER: FIRST# 1642
CPPDU COUNTER: LAST# 1672
[XRIT] "IMG_FD_026_WV069_20200605_041636_09.hrit"
CPPDU COUNTER: FIRST# 3616
CPPDU COUNTER (C): ERROR (EXPECTED: 4057, ACTUAL: 4058)
CPPDU COUNTER (C): ERROR (EXPECTED: 4499, ACTUAL: 4500)
CPPDU COUNTER: LAST# 4596
[XRIT] "IMG_FD_026_VI006_20200605_041636_09.hrit"
CPPDU COUNTER: FIRST# 4597
CPPDU COUNTER: LAST# 4618
[XRIT] "IMG_FD_026_WV069_20200605_041636_10.hrit"
[JPEG] "IMG_FD_026_WV069_20200605_041636.jpg"
CPPDU COUNTER: FIRST# 4619
CPPDU COUNTER: LAST# 4639
[XRIT] "IMG_FD_026_SW038_20200605_041636_10.hrit"
[JPEG] "IMG_FD_026_SW038_20200605_041636.jpg"
CPPDU COUNTER: FIRST# 671
CPPDU COUNTER: LAST# 709
[XRIT] "IMG_FD_026_IR105_20200605_041636_10.hrit"
[JPEG] "IMG_FD_026_IR105_20200605_041636.jpg"
CPPDU COUNTER: FIRST# 710
CPPDU COUNTER: LAST# 747
[XRIT] "IMG_FD_026_IR123_20200605_041636_10.hrit"
[JPEG] "IMG_FD_026_IR123_20200605_041636.jpg"
CPPDU COUNTER: FIRST# 4683
CPPDU COUNTER (C): ERROR (EXPECTED: 5124, ACTUAL: 5125)
CPPDU COUNTER: LAST# 5162
[XRIT] "IMG_FD_026_VI006_20200605_041636_10.hrit"
As you can see, every 442th from beginning of the TPFile CPPDU is loss. So, we have loss 8190 bytes for each 3619980 bytes (8190*442) of file length. For large file, there will be data loss after 7239960, 10859940 bytes and so on. As IR-channel's hrit files are much less than 3.6M, they have no data loss at all.
Fantastic work tracking down all of these issues! I finally have some time to devote to HRIT support over the next few days. I'll be implementing your findings in the hrit
branch where I've done some initial HRIT product output support.
Side note: What is your RF setup for receiving HRIT? (dish size, feed, amplifiers, filters)
For HRIT we use meteorological receiving station based on 3-meter dish and circular-polarized feed-horn with built-in LNA. I know that GK-2A uses linear polarization, but I cannot change antenna's construction. Described above ETRA receiver is used for demodulation.
LRIT is received by 2-meter dish with self-built "cantenna" feed-horn (from drainpipe :) ) with linear polarization, Nooelec SAWBird GOES LNA/filter and Airspy R2 receiver.
I have not tested yet HRIT on 2m dish (cannot powerup LNA from ETRA by coax cable, need to bring additional power to the roof), but signal on 2m dish and Airspy seems to be powerful enough:
(no data transmission, empty frames VCID=63)
(data is transmitted, VCID=0)
Also, your goesrecv-monitor
shows pretty good picture for LRIT:
I agree, HRIT on the 2m does look strong enough. Do you have an IQ recording from the 2m dish I can download? I can try running it through my GNU Radio HRIT demodulator.
You could try ETRA on the 2m dish using an external bias-tee between the ETRA and SAWbird to inject 5V DC. A cheap eBay one should do the job. This way you won't have to run separate power to the antenna.
These are IQ-files recorded in Gqrx (freq=1695.4 MHz, rate=10MSPS). They are HUGE (approx. 450M each for 5-seconds recording), one is with VCID=63 (no data) and other is VCID=0 (FD).
https://drive.google.com/file/d/1XQQAg5-DvX-Gz_TLhViHapV3tod4B8FJ/view?usp=sharing https://drive.google.com/file/d/1IiLuatFxN7dAqIZtt4ZlimQT4LzIQnl6/view?usp=sharing
If you can not replay these files, please, give me a tip how to record.
QPSK constellation is definitely visible in those recordings. I'm not sure if Viterbi + RS would be enough to recover any data from this signal.
Do you have a VCDU capture of VIS006 imagery? I'd like to tackle that CP_PDU issue after getting J2K image output working.
Do you have a VCDU capture of VIS006 imagery?
Do you mean dumps by script --dump
option (links in the first message, includes all five channels) or something else? I had re-process them, and there are exactly the same errors/losses.
Ah sorry, you are right. For some reason I was thinking they were not transmitted at night like old COMS-1 LRIT.
In that case I'll try adding an APID filter to the demuxer so I can work on just VIS006, then move on to J2K stuff.
Something strange is happening with APID. It seems that all the channels are spread across all APIDs:
[VCID 0] GK-2A: FULL DISK
APID#: 6
CPPDU COUNTER: FIRST# 0
CPPDU COUNTER: LAST# 24
[XRIT] "IMG_FD_046_WV069_20200610_073636_01.hrit"
APID#: 2
CPPDU COUNTER: FIRST# 0
CPPDU COUNTER: LAST# 41
[XRIT] "IMG_FD_046_IR123_20200610_073636_01.hrit"
APID#: 14
CPPDU COUNTER: FIRST# 0
CPPDU COUNTER: LAST# 26
[XRIT] "IMG_FD_046_SW038_20200610_073636_01.hrit"
APID#: 6
CPPDU COUNTER: FIRST# 25
CPPDU COUNTER: LAST# 67
[XRIT] "IMG_FD_046_IR105_20200610_073636_01.hrit"
APID#: 6
CPPDU COUNTER: FIRST# 68
CPPDU COUNTER: LAST# 692
[XRIT] "IMG_FD_046_VI006_20200610_073636_01.hrit"
APID#: 6
CPPDU COUNTER: FIRST# 693
CPPDU COUNTER: LAST# 760
[XRIT] "IMG_FD_046_IR123_20200610_073636_02.hrit"
APID#: 14
CPPDU COUNTER: FIRST# 27
CPPDU COUNTER: LAST# 65
[XRIT] "IMG_FD_046_WV069_20200610_073636_02.hrit"
APID#: 12
CPPDU COUNTER: FIRST# 0
CPPDU COUNTER: LAST# 41
[XRIT] "IMG_FD_046_SW038_20200610_073636_02.hrit"
APID#: 6
CPPDU COUNTER: FIRST# 761
CPPDU COUNTER: LAST# 829
[XRIT] "IMG_FD_046_IR105_20200610_073636_02.hrit"
APID#: 2
CPPDU COUNTER: FIRST# 42
CPPDU COUNTER: LAST# 1012
[XRIT] "IMG_FD_046_VI006_20200610_073636_02.hrit"
APID#: 6
CPPDU COUNTER: FIRST# 830
CPPDU COUNTER: LAST# 874
[XRIT] "IMG_FD_046_WV069_20200610_073636_03.hrit"
APID#: 14
CPPDU COUNTER: FIRST# 66
CPPDU COUNTER: LAST# 112
[XRIT] "IMG_FD_046_SW038_20200610_073636_03.hrit"
APID#: 6
CPPDU COUNTER: FIRST# 875
CPPDU COUNTER: LAST# 950
[XRIT] "IMG_FD_046_IR123_20200610_073636_03.hrit"
APID#: 8
CPPDU COUNTER: FIRST# 0
CPPDU COUNTER: LAST# 78
[XRIT] "IMG_FD_046_IR105_20200610_073636_03.hrit"
APID#: 14
CPPDU COUNTER: FIRST# 113
CPPDU COUNTER: LAST# 1248
[XRIT] "IMG_FD_046_VI006_20200610_073636_03.hrit"
APID#: 6
CPPDU COUNTER: FIRST# 951
CPPDU COUNTER: LAST# 999
[XRIT] "IMG_FD_046_WV069_20200610_073636_04.hrit"
APID#: 12
CPPDU COUNTER: FIRST# 42
CPPDU COUNTER: LAST# 123
[XRIT] "IMG_FD_046_IR105_20200610_073636_04.hrit"
APID#: 12
CPPDU COUNTER: FIRST# 124
CPPDU COUNTER: LAST# 168
[XRIT] "IMG_FD_046_SW038_20200610_073636_04.hrit"
APID#: 6
CPPDU COUNTER: FIRST# 1000
CPPDU COUNTER: LAST# 1078
[XRIT] "IMG_FD_046_IR123_20200610_073636_04.hrit"
APID#: 2
CPPDU COUNTER: FIRST# 1013
CPPDU COUNTER: LAST# 2068
[XRIT] "IMG_FD_046_VI006_20200610_073636_04.hrit"
and so on
KMA have a habit of not following their own specifications. I noticed this when working on LRIT.
It could still be a demuxer issue since I'm completely ignoring APIDs at the moment. It was done this way because the APID was useless with COMS-1 LRIT when the demuxer core was written. It used to be the VCID*64 so was of no real benefit.
I will need to track APIDs now with HRIT because the FD wavelength segments are interleaved which complicates product generation.
Confirmed CP_PDU #441 is having issues, and the APIDs are completely useless. The CP_PDU header is there but it never gets marked as completed which is needed to trigger the payload length and CRC check, then its meant to be appended to the current TP_File.
Time to delve into the demuxer core...
[VCID 0] GK-2A: FULL DISK
[XRIT] "IMG_FD_026_IR123_20200514_041636_01.hrit"
[XRIT] "IMG_FD_026_SW038_20200514_041636_01.hrit"
[XRIT] "IMG_FD_026_WV069_20200514_041636_01.hrit"
[XRIT] "IMG_FD_026_IR105_20200514_041636_01.hrit"
[CP_PDU] APID: 14 SEQ: CONTINUE #439 LEN: 8192
[TP_File] CURRENT LEN: 3603590 (71%) EXPECTED LEN: 5079906 DIFF: 1476316
[CP_PDU] APID: 14 SEQ: CONTINUE #440 LEN: 8192
[TP_File] CURRENT LEN: 3611780 (71%) EXPECTED LEN: 5079906 DIFF: 1468126
[CP_PDU] APID: 14 SEQ: CONTINUE #442 LEN: 8192
[TP_File] CURRENT LEN: 3619970 (71%) EXPECTED LEN: 5079906 DIFF: 1459936
[CP_PDU] APID: 14 SEQ: CONTINUE #443 LEN: 8192
[TP_File] CURRENT LEN: 3628160 (71%) EXPECTED LEN: 5079906 DIFF: 1451746
[CP_PDU] APID: 14 SEQ: CONTINUE #444 LEN: 8192
[TP_File] CURRENT LEN: 3636350 (72%) EXPECTED LEN: 5079906 DIFF: 1443556
SKIPPING FILE DUE TO DROPPED PACKETS
[XRIT] "IMG_FD_026_SW038_20200514_041636_02.hrit"
[XRIT] "IMG_FD_026_IR123_20200514_041636_02.hrit"
[CP_PDU] APID: 14 SEQ: CONTINUE #439 LEN: 8192
HEADER: 0x080E01B71FFF
OFFSET: 0x9E
.........
LENGTH: OK
CRC: OK
[TP_File] CURRENT LEN: 3603590 (71%) EXPECTED LEN: 5079906 DIFF: 1476316
[CP_PDU] APID: 14 SEQ: CONTINUE #440 LEN: 8192
HEADER: 0x080E01B81FFF
OFFSET: 0x190
.........
LENGTH: OK
CRC: OK
[TP_File] CURRENT LEN: 3611780 (71%) EXPECTED LEN: 5079906 DIFF: 1468126
[CP_PDU] APID: 14 SEQ: CONTINUE #441 LEN: 8192
HEADER: 0x080E01B91FFF
OFFSET: 0x282
.......... [CP_PDU] APID: 14 SEQ: CONTINUE #442 LEN: 8192
HEADER: 0x080E01BA1FFF
OFFSET: 0x0
.........
LENGTH: OK
CRC: OK
[TP_File] CURRENT LEN: 3619970 (71%) EXPECTED LEN: 5079906 DIFF: 1459936
[CP_PDU] APID: 14 SEQ: CONTINUE #443 LEN: 8192
HEADER: 0x080E01BB1FFF
OFFSET: 0xF2
.........
LENGTH: OK
CRC: OK
[TP_File] CURRENT LEN: 3628160 (71%) EXPECTED LEN: 5079906 DIFF: 1451746
[CP_PDU] APID: 14 SEQ: CONTINUE #444 LEN: 8192
HEADER: 0x080E01BC1FFF
OFFSET: 0x1E4
.........
LENGTH: OK
CRC: OK
[TP_File] CURRENT LEN: 3636350 (72%) EXPECTED LEN: 5079906 DIFF: 1443556
Found it! The pointer
field in the M_PDU
header points to the location of the CP_PDU
header within the current M_PDU
data field.
I was foolishly using that header value to trigger the start of a new TP_File, which works for small files. But once the CP_PDU header lines up again with the start of the M_PDU
data field (which it does-so every 442 CP_PDUs) the previous CP_PDU is discarded and a new one is started.
I've re-arranged that section of the demuxer to now check if there's an unfinished TP_File rather than assume an offset of zero is the start of a new TP_File. All 50 HRIT files for that FD are now being saved correctly.
And we have complete Full Disks! I've re-written hrit-img.py
based on lrit-img.py
but using a pre-compiled Windows binary of libjpeg
to convert the J2K payload to PPM, which is then loaded by Pillow in Python for segment merging and saving as a JPG.
Couldn't help myself. I had to render a false colour with GeoSatSignal
I'll adapt the changes in hrit-img.py
into the Products
class of xrit-rx tomorrow so the image generation and merging is automatic.
Wow! That's fantastic! :D
I've now got xrit-rx saving assembled HRIT full disk images without saving .hrit
files to disk. It does save .jp2
files temporarily but they're deleted once libjpeg
creates the .ppm
file. Once Pillow
has loaded the .ppm
image into memory it is also deleted from disk, leaving only the assembled FDs of all five wavelengths.
All that's left for HRIT support is to update the progress bar indicator to handle multiple wavelengths.
I may add a UDP socket input option before next release so you can easily try it on the real-time downlink using the ETRA modem. I'll also need to work out licensing of libjpeg
since it's GPLv3 and xrit-rx is MIT.
Progress bar now shows each wavelength individually.
Changes merged into dev
branch in PR #17 . Will be included in next release.
Just released v1.3
with all the HRIT changes and a UDP input option for your hardware modem. Let me know how the UDP input goes!
Hello, Based on the actual experience of receiving signal from GK-2A, what is the minimum diameter of the satellite dish suitable for receiving a HRIT signal at the frequency 1695.4 MHz?
My attempts have not yet been successful with the prime focus 2.3 m mesh antenna, although the signal looks quite sufficient in sdrsharp: 2020-09-04T06:14:37Z [monitor] gain: 41.09, freq: 183474.8, omega: 3.333, vit(avg): 2112, rs(sum): 0, packets: 0, drops: 112 2020-09-04T06:14:38Z [monitor] gain: 41.11, freq: 182697.4, omega: 3.333, vit(avg): 2111, rs(sum): 0, packets: 0, drops: 110 2020-09-04T06:14:39Z [monitor] gain: 41.13, freq: 176119.0, omega: 3.333, vit(avg): 2109, rs(sum): 0, packets: 0, drops: 105 ...
Although xrit-rx is ready for HRIT, goesrecv is not yet. The HRIT downlink from GK-2A uses QPSK modulation and LRIT uses BPSK. I have been working on adding the necessary components to goesrecv for QPSK demodulation but have not finished this yet.
I've been posting progress updates to Twitter: https://twitter.com/sam210723/status/1274632866526191616. Here I'm testing with FY-4A LRIT which is also QPSK.
2.3m may be large enough for HRIT but I won't be able to say for sure until we have a working receive chain.
Thank you so much for your response and for your brilliant work! I successfully use your programs to get images from GK-2A in LRIT and will follow the news about goesrecv with QPSK demodulation with great interest.
Glad to hear xrit-rx and goesrecv-monitor are working well for you! Great looking signal there on your 2.3m dish. I will try remember to reply to this issue when HRIT demodulation/decoding is working.
Hi Sam, and everyone who has contributed to this fantastic outcome. Great work and talent!
Cheers
I got a dump of an HRIT stream and am trying to decode it using xrit-rx. I get a lot of errors like
Product files (.bin) are written, but their size is too small.
I have checked VCDU counter field, and I does not see any packet loss from receiver, counter leaps only on transmission start/stop:
Links to dumps: hrit03.dmp hrit03.log hrit04.dmp hrit04.log