Data sometimes looks corrupt at the amsat.org

ac2cz commented 2 years ago

Well done getting this forwarder working. We really appreciate the data. However I do see very occasional issues at our end. We have received a few records that look corrupt. They have been spotted because the reset and uptime are not valid. It is way in the future. It is possible for frames to get corrupted for many reasons, so at first I did not suspect anything was wrong with this forwarder. But recently we got two frames from two different stations with the same error. The data is shown below.

For reset: 524 at Groundstation: IZ1VWE

STP Date	Uptime	Estimated T0
Sat, 12 Nov 2022 10:12:02	25165824	Tue Jan 25 03:41:38 UTC 2022

T0 Est: Tue Jan 25 03:41:38 UTC 2022
T0: 1643082098000

The two frames that caused an issue here linked here: https://amsat.org/tlm/fox1c/20221112101707.628.stp.processed https://amsat.org/tlm/fox1c/20221112102540.633.stp.processed

Any thoughts on what could be causing this?

dernasherbrezon commented 2 years ago

Hi,

I found this data in leosatdata.com:

{
  "header": {
    "type": 3,
    "uptime": 25165824,
    "resetCount": 524,
    "foxId": 3
  },
  "payloadMin": {
    "softError": {
      "dacOverflows": 18,
      "i2c1Retries": 8,
      "i2c2Retries": 3,
      "spiRetries": 64,
      "mramCRCs": 54
    },
    "maxTimestampResetCount": 8192,
    "maxTimestampUptime": 30526703,
    "safeModeIndication": false,
    "autoSafeModeActive": true,
    "autoSafeModeAllowed": false,
    "batteryAVolt": 0.006713867,
    "batteryBVolt": 0.0,
    "batteryCVolt": 0.0,
    "batteryATemperature": -207.0,
    "batteryBTemperature": -207.0,
    "batteryCTemperature": 21.756098,
    "totalBatteryCurrent": 1002.0642,
    "batteryBoardTemperature": 59.44,
    "panelPlusXVolt": 5.5421934,
    "panelMinusXVolt": 1.7335849,
    "panelPlusYVolt": 5.9825997,
    "panelMinusYVolt": 3.788815,
    "panelPlusZVolt": 4.524475,
    "panelMinusZVolt": 0.011546236,
    "panelPlusXTemperature": -201.9683,
    "panelMinusYTemperature": -128.622,
    "pcuTemperature": -203.0,
    "spin": 6.5625,
    "txPaCurrent": 98.950195,
    "txTemperature": 63.6,
    "rxTemperature": -45.2,
    "rssi": -69.71252,
    "ihuTemperature": 168.18182,
    "xAngularVelocity": -9.609611,
    "yAngularVelocity": -6.0810804,
    "zAngularVelocity": -34.572075,
    "exp4Temperature": 51.6,
    "psuCurrent": 3.90625
  }
}

This is parsed by jradio ( https://github.com/dernasherbrezon/jradio/blob/master/src/main/java/ru/r2cloud/jradio/fox/Fox1CBeacon.java#L21 ) and indeed uptime is abnormal.

Some info:

time is populated when ground station received the frame. It is 10:12:02 UTC so should be valid
I noticed 2 more frames were received during the same observation: 10:12:01 (before) and at 10:14:11 (after). They both have valid uptimes: 816530 and 816660 respectively.
amsatTlmClient forwards raw frame and it doesn't modify the header
So the issue might have happened during demodulation/de-framing in jradio or on the satellite.

Do you have telemetry from other users around that time?

dernasherbrezon commented 2 years ago

I'm looking into jradio Fox decoder: https://github.com/dernasherbrezon/jradio/blob/master/src/main/java/ru/r2cloud/jradio/fox/Fox.java#L37

It looks like there is 8b10b code + Reed Solomon. This alone might not be enough for message correctness. Hypothesis: random byte array passed Reed Solomon decoding and was forwarded further.

Have you seen such abnormal messages from other demodulators?
This message has "type" 3 which is very unusual. This might indicate that whole frame contains random data or something happened on the spacecraft while transmitting this particular type

ac2cz commented 2 years ago

I don't have other telemetry at amsat.org for that time. Is the leosat data from another decoder? Or from Satnogs?

We have seen corrupted data before and it very occasionally gets past the RS Decoder. We assumed they are from a distorted signal that happens to decode. We have not seen two corrupted records with the same data but from different stations before. That was what interested me. If we can conclude it is a corruption on the spacecraft then it may help us find a bug which may or may not be present in future spacecraft code.

I see some recordings of passes at that time on satnogs but I can't get the ogg data to decode.

dernasherbrezon commented 2 years ago

Yes, this is coming from jradio. Completely different decoder. leosatdata.com gets the raw data from different stations across the world and sends it to AMSAT + SatNOGS (if user configured integration). Also jradio + r2cloud are very aggressive in scheduling and getting the data automatically. So more data from them, higher chances to get past RS decoder.

I can check my own stations for FOX-1C abnormal messages and get raw I/Q data. Chances are slim as the local disk is small and raw I/Q stored for short periods of time.

ac2cz commented 1 year ago

We received two more corrupt frames.

For reset: 512 at Groundstation: M7RED

STP Date	Uptime	Estimated T0
Tue, 22 Nov 2022 07:30:20	20971520	Thu Mar 24 14:05:00 UTC 2022
Wed, 23 Nov 2022 07:35:53	12582912	Thu Jun 30 16:20:41 UTC 2022

Usually I receive about 1 corrupt frame a year across all the AMSAT spacecraft, so this is very unusual. There are no corrupt frames for Fox-1C so far other than from this amsatTlmClient. We now have at least 6 corrupt frame from this source. I have 680,000 frames from FoxTelem for Fox-1C and only 10,000 from amsatTlmClient. So something is causing a problem.

It is possible that the RS Decoder is configured to occasionally over correct the frame and then the corrected bytes are being forwarded. I have seen this with FoxTelem historically. For example if the erasure limit is set too high then the RS Decoder will allow corrupt frames to be passed through. In FoxTelem I lowered the erasure limit to 15 to prevent this sort of issue. If you have it set at 16 or higher, then lower it to 15. We can then see if that stops the issue.

Again, thanks for sending the data and for writing a forwarder.

dernasherbrezon commented 1 year ago

Also 15 in jradio: https://github.com/dernasherbrezon/jradio/blob/master/src/main/java/ru/r2cloud/jradio/fox/Fox.java#L39 I'm using tolerance 2 while finding syncword https://github.com/dernasherbrezon/r2cloud/blob/master/src/main/java/ru/r2cloud/satellite/decoder/FoxSlowDecoder.java#L31 Maybe it is picking up additional noise.

I can see only 1 or 2 beacons received on my stations during last weeks. I will setup an automated script to detect corrupted frame and backup raw I/Q for analysis.

dernasherbrezon / amsatTlmClient

Data sometimes looks corrupt at the amsat.org #2