search

projecthorus / radiosonde_auto_rx

Automatically Track Radiosonde Launches using RTLSDR
GNU General Public License v3.0
498 stars 125 forks source link

Radiosonde Support - Intermet iMet-4 (and narrow-band versions of the iMet-1) #94

Closed darksidelemm closed 3 years ago

darksidelemm commented 5 years ago

Placeholder issue for adding support for some other sonde types. Things to do to add a new sonde:

Viproz commented 5 years ago

Update on the M10 support :

I am still very much in a testing phase, I wasn't able to decode a sonde yet and I only have one shot per day so if anyone wants to test it it it here https://github.com/Viproz/radiosonde_auto_rx, note that you should whitelist the frequency the sonde is on, I am not sure that the peak detection is working

TheSkorm commented 5 years ago

Sweet!

@Viproz you'll probably want to keep the APRS name beginning with RS_ due to filtering setup on other projects.

darksidelemm commented 5 years ago

The M10xxxxx is just for the APRS object ID - the callsign sent to Habitat will still have the RS prefix on it.

Viproz commented 5 years ago

The decoding is now fully working for the M10 ptu, I was able to detect it and decode it today, I did have to change the bandwidth for the scan from 15k to 22k since the signal is about 20k large for the M10 and it wasn't always detection the header correctly so some testing should be done to check that it is not an issue with the other RS (Darkside tested it with a RS41 today UTC and working so far but that's a sample of 1). Also I have observed a deviation (with a TX0 0.5ppm sdr) of about 3kHz changing during the flight (went from +3kHz to ~-1kHz), I need to do more testing to see if this is an issue and a way to calibrate the frequency needs to be thought of or if it's okay to just use a slightly larger listening bandwidth (FM is pretty tolerant with off center decoding but the added noise might be an issue).

I'll continue testing in the following weeks and continue on improving what I can.

oe3jtb commented 5 years ago

Hi tested the version but the result is

2018-12-22T11:39:30Z ERROR Unsupported sonde type: M10
2018-12-22T11:39:30Z INFO Detected new M10 sonde on 401.000 MHz!

its a new type started in Vienna Austria tnx 73

darksidelemm commented 5 years ago

It's not completely integrated yet... Give it a while :-)

oe3jtb commented 5 years ago

yeah ok :-) its a nice work at all tnx

darksidelemm commented 5 years ago

Whoops - will keep this issue open until the changes are merged...

Viproz commented 5 years ago

@oe3jtb Some testing on another setup would actually be welcome, I'm guessing you didn't build the right repository if you have this issue, I did check my repo and there should be no way for it to give this error.

What you need to do is "git clone https://github.com/Viproz/radiosonde_auto_rx.git" and then just run the build.sh file inside, if you got it working for other RS it should already be setup and work for M10s

If you have some more issues you can also contact me directly on #highaltitude on freenode, I'm often there.

oe3jtb commented 5 years ago

Sri Vincent, my fault, you are absolutly Right, wrong reposititory Now I installed yours and I will get next Chance to test 12Z UTC Tnx 73 Alex

Gesendet von Mail für Windows 10

Von: Vincent Gesendet: Samstag, 22. Dezember 2018 16:18 An: projecthorus/radiosonde_auto_rx Cc: oe3jtb; Mention Betreff: Re: [projecthorus/radiosonde_auto_rx] M10 / iMet Support (#94)

@oe3jtb Some testing on another setup would actually be welcome, I'm guessing you didn't build the right repository if you have this issue, I did check my repo and there should be no way for it to give this error. What you need to do is "git clone https://github.com/Viproz/radiosonde_auto_rx.git" and then just run the build.sh file inside, if you got it working for other RS it should already be setup and work for M10s If you have some more issues you can also contact me directly on #highaltitude on freenode, I'm often there. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Viproz commented 5 years ago

Today I tried to remove the whitelist and see what it does without changing anything to the algorithm and it worked ! I was able to track two RS because they sent two today for some reason so I followed them one after the other.

image

The date is wrong, I don't know why but I don't remember it being an issue before so I'll do a bit more testing. Was a bug in my code, fixed. The main thing to decide before pushing is going to be the ID format, on APRS some people are uploading M10s with dxlAPRS but they give them the name ME025131 which does not correspond directly to the serial number and the code is way too toxic to be read by me, so if it's fine by you guys I think we can impose ours.

To give you some details : 803 2 11196 803 changes with the sonde hardware, 2 I have no clue (always the same) and 11196 is unique to a RS from what I gathered. The ID I proposed would be "M10_11196".

oe3jtb commented 5 years ago

HI I have the Vienna M10 sonde in the Grey list

Same behavior as Vincent Shows

And I have also troubles reconizing RS41 on 402.7 it is shown as M10, but no decoding of course 😊

73 Alex

Gesendet von Mail für Windows 10

Von: Vincent Gesendet: Montag, 24. Dezember 2018 18:43 An: projecthorus/radiosonde_auto_rx Cc: oe3jtb; Mention Betreff: Re: [projecthorus/radiosonde_auto_rx] M10 / iMet Support (#94)

Today I tried to remove the whitelist and see what it does without changing anything to the algorithm and it worked ! I was able to track two RS because they sent two today for some reason so I followed them one after the other.

The date is wrong, I don't know why but I don't remember it being an issue before so I'll do a bit more testing. The main thing to decide before pushing is going to be the ID format, on APRS some people are uploading M10s with dxlAPRS but they give them the name ME025131 which does not correspond directly to the serial number and the code is way too toxic to be read by me, so if it's fine by you guys I think we can impose ours. To give you some details : 803 2 11196 803 changes with the sonde hardware, 2 I have no clue (always the same) and 11196 is unique to a RS from what I gathered. The ID I proposed would be "M10_11196". — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

oe3jtb commented 5 years ago

PS when not in Grey list Decoding and recogintion is starting

Gesendet von Mail für Windows 10

Von: J.Alexander Tschapka Gesendet: Montag, 24. Dezember 2018 19:54 An: projecthorus/radiosonde_auto_rx Betreff: AW: [projecthorus/radiosonde_auto_rx] M10 / iMet Support (#94)

HI I have the Vienna M10 sonde in the Grey list

Same behavior  as Vincent Shows

And I have also troubles reconizing RS41 on 402.7 it is shown as M10, but no decoding of course 😊

73 Alex

Gesendet von Mail für Windows 10

Von: Vincent Gesendet: Montag, 24. Dezember 2018 18:43 An: projecthorus/radiosonde_auto_rx Cc: oe3jtb; Mention Betreff: Re: [projecthorus/radiosonde_auto_rx] M10 / iMet Support (#94)

Today I tried to remove the whitelist and see what it does without changing anything to the algorithm and it worked ! I was able to track two RS because they sent two today for some reason so I followed them one after the other.

The date is wrong, I don't know why but I don't remember it being an issue before so I'll do a bit more testing. The main thing to decide before pushing is going to be the ID format, on APRS some people are uploading M10s with dxlAPRS but they give them the name ME025131 which does not correspond directly to the serial number and the code is way too toxic to be read by me, so if it's fine by you guys I think we can impose ours. To give you some details : 803 2 11196 803 changes with the sonde hardware, 2 I have no clue (always the same) and 11196 is unique to a RS from what I gathered. The ID I proposed would be "M10_11196". — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Viproz commented 5 years ago

I'm not sure I understand, the M10 was on which frequency and the RS41 was on 402.7 ? It told you that it was an M10 instead of telling you it was an RS41 ?

Edit : Fixed the issue with the day of the month, git pull and build.sh to fix it

darksidelemm commented 5 years ago

I have seen some cases where a RS41 is falsely detected as a M10, however this only seemed to occur when the RS41s SNR was low. I think there's definitely some improvements to be made in the rs_detect code, but it's a bit beyond my skill to do at the moment :-(

oe3jtb commented 5 years ago

Hi Vincent yes on 402.7 there was only a RS41 and Auto RX means „found M10“ I also have dxlaprs in the same antenna line with the scanner from DF7PN there was a RS41 found I talked also to the owner from dxlaprs, because ist not Decoding so far distances as RS92 or RS41, he told me, that FEC is not known yet, so more Signal is needed at the receiver, and AFC must be Always 0 because of the second carrier in the Signal M10 which is 3 khz higher

73 Alex

Gesendet von Mail für Windows 10

Von: Vincent Gesendet: Montag, 24. Dezember 2018 23:59 An: projecthorus/radiosonde_auto_rx Cc: oe3jtb; Mention Betreff: Re: [projecthorus/radiosonde_auto_rx] M10 / iMet Support (#94)

I'm not sure I understand, the M10 was on which frequency and the RS41 was on 402.7 ? It told you that it was an M10 instead of telling you it was an RS41 ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Viproz commented 5 years ago

I made the code compatible with dxlAPRS since they are already used, the ID is not meaningful but people wouldn't update and tracking the same sonde with two IDs is a nightmare.

I added some finishing touches, I'm going to test it tonight and tomorrow and it will be ready for PR.

darksidelemm commented 5 years ago

Given there are a few changes to make M10 scanning/decoding work that may impact performance for other sondes, i'd suggest the following:

I can see this being a bit messy to implement in the short term, but it would allow those users who know they have M10 sondes in their area (and those that don't!) to test out the changes over a longer period (maybe a few weeks) so we can evaluate making the changes permanent (except for the quantisation one... that's a bit more painful to deal with).

darksidelemm commented 5 years ago

Looks like M10 support is progressing nicely and should be released shortly!

A question for @rs1729 - I'd like to add support for the newer 1200 baud iMet sondes (iMet1 and iMet4). The imet1rs_dft decodes these nicely, however there's currently no way of detecting them using either rs_detect or dft_detect. Do you have plans to add support for these, or does their modulation (1200 baud AFSK) make it too hard to auto-detect?

rs1729 commented 5 years ago

There is the iMet-1-AB and the iMet-1-RS. The latter is Bell202, FSK over FM-audio (AFSK), some 60kHz bandwidth. The iMet-4 has the same AFSK/Bell202, but with much less bandwidth, so FM-demodulated signal looks very similar. First I wanted to put it in dft_detect until I remembered how annoying the 2200/1200 tone-ratio is... Maybe it is enough to look for the 1200 Hz preamble, this could be done with the methods already used in rs_detect. So it is not the first thing on my list, but it is on the list. (and I'm sure the AFSK-decoder could be improved, if it wouldn't be such a retro/vintage modulation.)

darksidelemm commented 5 years ago

Interesting. I have a box labelled as an iMet-1-AB (though it has a sticker on it saying '6 kHz bandwidth' which is transmitting the 1200 baud AFSK modulation. I also have an iMet-4 which transmits the same modulation. So it looks like there's iMet-1-ABs with similar bandwidth too. These units were sent to me by the Perlan Project, who are interested in alternative means of tracking these sondes when they release them.

rs1729 commented 5 years ago

Oh, i have seen perlan imet4 sounding logs.

I only know the iMet-1-AB they used to launch in Belgium every Tuesday. https://www.flickr.com/photos/116298535@N06/24264799404/ Bandwidth 30 kHz. https://www.youtube.com/watch?v=3YQ6diSSlIM Modulation was something like 2400/4800 tones, 1200Hz preamble (and lead in/out). And it did not have the open frame format of iMet-1-RS. Although according to their website they were all Bell202. Now it is sometimes the iMet with 6 (rather 10) kHz bandwidth and Bell202, iMet-4.

rs1729 commented 5 years ago

I'd like to add support for the newer 1200 baud iMet sondes (iMet1 and iMet4). The imet1rs_dft decodes these nicely, however there's currently no way of detecting them using either rs_detect or dft_detect. Do you have plans to add support for these, or does their modulation (1200 baud AFSK) make it too hard to auto-detect?

dft_detect includes now both imet-formats and c34/c50, for testing. First there were many imet false positives for rs92-recordings. I hope it is better now, testet some 3500 recordings, didn't check every single one, but it looked promising. However, the more radiosonde types are included, the higher the possibility for false positives, I guess. And if you want to detect weak signals with low thresholds, the possibility is high to detect the wrong type. (I have some satellite signals that were detected as radiosondes...) If the signal is strong enough for decoding, the detection should be reliable. Still needs some testing. And it is much slower than rs_detect, of course.

darksidelemm commented 5 years ago

That's awesome! I'll give it a test...

On Wed., 6 Feb. 2019, 10:32 rs1729 <notifications@github.com wrote:

I'd like to add support for the newer 1200 baud iMet sondes (iMet1 and iMet4). The imet1rs_dft decodes these nicely, however there's currently no way of detecting them using either rs_detect or dft_detect. Do you have plans to add support for these, or does their modulation (1200 baud AFSK) make it too hard to auto-detect?

dft_detect includes now both imet-formats and c34/c50, for testing. First there were many imet false positives for rs92-recordings. I hope it is better now, testet some 3500 recordings, didn't check every single one, but it looked promising. However, the more radiosonde types are included, the higher the possibility for false positives, I guess. And if you want to detect weak signals with low thresholds, the possibility is high to detect the wrong type. (I have some satellite signals that were detected as radiosondes...) If the signal is strong enough for decoding, the detection should be reliable. Still needs some testing. And it is much slower than rs_detect, of course.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/projecthorus/radiosonde_auto_rx/issues/94#issuecomment-460853444, or mute the thread https://github.com/notifications/unsubscribe-auth/AA_fkxQokjQ5UEzchJDUfUlCstHgBpvhks5vKht6gaJpZM4ZFL_C .

HoshenKadosh commented 5 years ago

I got some recordings of the imet sonde launched in my country if this will help you. https://drive.google.com/open?id=1r-UC99l4ZAaRARcmBSbVcEq04J_v0PES

darksidelemm commented 5 years ago

Looking into the iMet telemetry format a bit more, it looks as though there is no unique serial number sent down in the telemetry (for the newer iMet4 sondes). Do you concur with this @rs1729 ?

If this is truly the case, then I don't see any easy way that we can support iMet sondes in auto_rx. (Apart from a user-override for the iMet serial number, which is not ideal). We need a unique identifier for each sonde, otherwise we will immediately hit issues when stations are receiving multiple sondes.

rs1729 commented 5 years ago

I think imet4 (as imet1rs) follow this open packet definitions https://github.com/rs1729/RS/blob/master/imet/imet1rs-binary.pdf more pkt_ids: http://harbor.weber.edu/Hardware/Ozonesonde/O3_manual.pdf So perhaps if there exists a pck_id with administrative data and if they would start using it. (The imet1ab has a serial number... but different format anyway)

What about DFM as long as the ID is not established?

darksidelemm commented 5 years ago

In https://github.com/projecthorus/radiosonde_auto_rx/commit/044466f5d9d7a61fd9031faef00b8782a32bee92 :

Still need to think about the logic required for creating imet unique IDs based on a user-input and time.

I'm thinking something along the lines of: RS_IMET--%y%m%d%HZ, where the hour is clipped to the nearest 6-hour launch time, i.e.: 22 - 03 -> 00Z 04 - 09 -> 06Z 10 -> 15 -> 12Z 16 -> 21 -> 18Z

Not yet sure how to condense this sufficiently for use on APRS.

This is only going to work if each receive station observed only one iMet sonde at a time. If a receive station can see multiple iMet sondes, then bad things will occur on the map.

TheSkorm commented 5 years ago

We could probably use the turn on time (calculated by reported gps time - frame number, taking care of the 0Z roll over). There might be enough variance between launches that this is suitable enough. A fairly lazy serial could be just unix time (seconds since 1970), a hashed version of that, or just decimal to hex.

The spec seems to say only a second, so this might be a problem with auto sonde launches.

TheSkorm commented 5 years ago

We should also add to the serial the frequency as well as another point of entropy

rs1729 commented 5 years ago

Generating unique IDs should not be a Problem, but when two observers receive the same iMet4, should they report the same ID? (On other tracking sites sometimes you could see multiple DFMs at the same position, although it is only one radiosonde in the air.) When different users send the same frames of a certain radiosonde, the server reports it once and you know all the contributors to the data of that radiosonde? So for iMet4 perhaps you should transmit the frame-CRCs as well. Then the server knows, if it already has reported a particular frame with the time stamp and the CRCs. Probably best to sent both packets, 0102 (time,position) and 0104 (frame number, ptu) with both CRCs, the whole frame is not very long after all. There might be some problems when putting all the frames together in one track, when there are multiple iMet4s.

There are only a few possible frequencies for iMet4. So it is easy to derive the right frequency, and the frequency is important to find the signal in the spectrum. On the other hand if all the iMet4 in the air are e.g. on 403MHz, the frequency is not separating them.

darksidelemm commented 5 years ago

The idea would be for each user to report the same ID, yes. I would rather not have to do this kind of matching on the server side - it's gets very complicated very quickly.

I think Michael's idea of using the power-on time of the iMet (derived from the current reported time and the frame count) is probably the best approach to start with, with the frequency and perhaps a user-supplied location ID included. We could use a short hash of this data as the APRS callsign.

The ID could be re-calculated for every new frame fairly easily - if somehow we end up with multiple iMets on the same frequency, then if they have different power-on times, they will have different calculated IDs.

rs1729 commented 5 years ago

Ah, you mean the difference GPS.time - ePTU.pcknum from the same frame!? Yes, this could be unique enough, for that day.

Have to take another look in the source to make sure, the frame is the same for both pcks, maybe the time is off by a second sometimes, but in principle should be a good idea.

EDIT: so if pcknum counts packets and there are always this to pcks 01_02 and 01_04, then pcknum increases by 2 every frame or second. However if one of the pcks is retransmitted (first I thought the time-resolution was the problem), the full example (2x 13:07:29) should be:

(13:07:26) lat: 47.345547° lon: 9.256685° alt: 16436m # CRC: 837F - 837F [OK] [11237] P:98.18mb T:-54.34°C U:4.61% bat:5.2V # CRC: 575F - 575F [OK]

(13:07:27) lat: 47.345543° lon: 9.256677° alt: 16439m # CRC: 70C0 - 70C0 [OK] [11239] P:98.13mb T:-54.34°C U:4.59% bat:5.2V # CRC: 2D86 - 2D86 [OK]

(13:07:29) lat: 47.345543° lon: 9.256650° alt: 16445m # CRC: 35BA - 35BA [OK] [11241] P:98.07mb T:-54.30°C U:4.65% bat:5.2V # CRC: A664 - A664 [OK]

(13:07:29) lat: 47.345543° lon: 9.256650° alt: 16445m # CRC: 35BA - 35BA [OK] [11243] P:98.00mb T:-54.31°C U:4.60% bat:5.2V # CRC: F3CF - F3CF [OK]

(13:07:30) lat: 47.345547° lon: 9.256651° alt: 16448m # CRC: 8134 - 8134 [OK] [11245] P:97.94mb T:-54.36°C U:4.68% bat:5.0V # CRC: A635 - A635 [OK]

(13:07:31) lat: 47.345554° lon: 9.256662° alt: 16452m # CRC: 050C - 050C [OK] [11247] P:97.89mb T:-54.37°C U:4.58% bat:5.0V # CRC: 1655 - 1655 [OK]

(13:07:32) lat: 47.345573° lon: 9.256669° alt: 16456m # CRC: CCC3 - CCC3 [OK] [11249] P:97.83mb T:-54.40°C U:4.69% bat:5.0V # CRC: 922A - 922A [OK]

(13:07:33) lat: 47.345592° lon: 9.256662° alt: 16459m # CRC: 4B87 - 4B87 [OK] [11251] P:97.79mb T:-54.37°C U:4.58% bat:5.0V # CRC: 8C45 - 8C45 [OK]

EDIT2: This is an example of iMet-1-RS. In the imet-4-data of Hoshen-4Z7HKA the pcknum advances by 1 each frame (second) and I didn't see retransmitted GPS-pcks. So maybe for imet4 (small bandwidth) this is not such an issue anymore.

darksidelemm commented 5 years ago

Hmm, if we end up seeing sondes with the above repeating / incrementing by more than one behaviour, then that is going to break my current ID approach badly (a new ID will be generated for each new packet!)

rs1729 commented 5 years ago

It is still possible that the new imet-4 behaves differently, the imet4-recordings I have seen increment the pck-num only by one each second (and right now you only want to support imet4 with narrow bandwidth?).

Maybe you could do the following: watch the difference time - pck_num over 3 or 4 frames (seconds), and if it stays constant over the last 3 frames, accept it for ID-generation. (With imet-4 I didn't observe gps-pck-retransmission either.) Following frames should match the same constant "time - pck_num" value.

Perhaps https://github.com/bobasaurus http://www.allenjordan.info/skysonde.html who seems to know the imet-1-rs, can say something more about the pck-enumeration and if there is a difference to imet-4.

darksidelemm commented 5 years ago

Initial support released in https://github.com/projecthorus/radiosonde_auto_rx/releases/tag/v1.0.1

Keeping this issue open until we see some iMet sondes in the wild and are sure it all works correctly!

jonomakepeace commented 5 years ago

I have iMet sondes in my area (Cape Town South Africa). I am not sure what type they are. I have done some scripting using the RS decoders and I'm ready to test that on the launch happening now (previous version of auto_rx didn't work when I tried last weekend), but I will try with the latest version today too and will report my results.

darksidelemm commented 5 years ago

It will be interesting to see if auto_rx detects them accurately. If you run auto_rx in verbose mode (python auto_rx.py -v) you will see a lot more debug information about the sonde type detected, and in the case of the iMet sondes, the sub-type. If it's one of the older 'wideband' sondes, then we don't currently support it, and probably won't as they are being phased out. If it's the newer narrowband type, then hopefully it will work!

darksidelemm commented 5 years ago

If you can't get it going, then a snapshot of the sonde signals spectrum from something like GQRX or SDR# would be useful, perhaps along with either an audio or IQ recording. That will at least confirm what type of sonde it is, and if it's even something we support.

rs1729 commented 5 years ago

Interestingly https://www.intermetafrica.com/category/radiosondes/ seems to offer slightly different imet-models. Don't know where they are in use, but the company is based in Cape Town, South Africa. The specifications of imet-54 look e.g. say 4800 bit/s data rate...

@jonomakepeace, if the imets in your area are different, maybe you can provide some signal recordings.

jonomakepeace commented 5 years ago

Unfortunately today's attempts were a bit of a fail.

I seem to have a problem with my RTL-SDR v3 or at least my setup. For some reason when using SDR#, I can't see the sonde signal (at 404 MHz almost on the dot) on any of the higher sample rates (0.25 is the only one I can use to get a clearish sounding), which seems a bit strange. I'm using just a DIY ~18mm 1/4 wave ground plane so far on about 2m of coax. No filter or LNA yet but I think I might need to go that route. I'm about 30km from the launch site.

Interesting you point that out. I have a contact at our weather institution so I will find out exactly which model they launch in Cape Town.

When I next get a chance I will be sure to at least record a sound clip (the signature I heard compares quite well with the one in the imet/wav folder in the RS repo) and hopefully the spectrum too.

jonomakepeace commented 5 years ago

Investigation concludes its an iMet 2 AA. 20kHz bandwidth, 2400 baud. So I imagine that this will not be supported by the current iMet decoders being used.

rs1729 commented 5 years ago

Interesting, I came across the intermetafrica-website some time ago, but did never hear/see a signal of their radiosondes. imet-2 looks like a small version of the old imet-1. And imet-54 is more like the imet-4, at least from the outside.

If you have audio or even iq-recordings that you can upload somewhere, I would be interested. And keep listening, maybe you receive a signal from an imet-54 some day.

darksidelemm commented 5 years ago

In that case they might not be iMet-4 sondes, and instead are the other variants of iMet-1s. Could you perhaps get us an audio recording of a sonde when they are in the air next? This would help identify what's going on.

darksidelemm commented 3 years ago

I think we can safely close this one now...