projecthorus / radiosonde_auto_rx

Automatically Track Radiosonde Launches using RTLSDR
GNU General Public License v3.0
477 stars 124 forks source link

strip_sonde_serial does not support MEISEI RS-11G #727

Closed AstroTaka closed 1 year ago

AstroTaka commented 1 year ago

Today, I received MEISEI RS-11G in my 1st time. I could not see the Main page because I was not in front of my PC. Probarbly Type is MEISEI and ID is RS11G-7135153. About Sonde List, type is iMS1-00 and serial is RS11G-7135153 also.

So please update the strip_sonde_serial to support serial of RS11G same as iMS-100. And please fix type of Sonde List to display RS41G.

I uploaded RS11G and RS41 log, because JMA launched them at the same time at the same baloon. https://drive.google.com/file/d/1LAnK6VuAASSHUHAj3iLmvUp4ZdM7DcjI/view?usp=share_link

P.S. According to a web page of Japan Meteorological Agency, they use iMS-100, RS-11G and RS41-SG now. https://www.jma.go.jp/jma/kishou/know/upper/kaisetsu.html

AstroTaka commented 1 year ago

About Sonde List, the following patch solve this issue for me.

diff --git a/auto_rx/autorx/log_files.py b/auto_rx/autorx/log_files.py
index aa52955..3ebb51c 100644
--- a/auto_rx/autorx/log_files.py
+++ b/auto_rx/autorx/log_files.py
@@ -67,6 +67,9 @@ def log_filename_to_stats(filename, quicklook=False):

         # Third field is the sonde type, in 'shortform'
         _type = _fields[2]
+        if _type=="MEISEI":
+            _type = _fields[1].split("-")[0]
+
         _type_str = short_type_lookup(_type)
         _short_type = short_short_type_lookup(_type)

diff --git a/auto_rx/autorx/utils.py b/auto_rx/autorx/utils.py
index 5489fd5..f5e0229 100644
--- a/auto_rx/autorx/utils.py
+++ b/auto_rx/autorx/utils.py
@@ -141,7 +141,7 @@ def strip_sonde_serial(serial):
     """ Strip off any leading sonde type that may be present in a serial number """

     # Look for serials with prefixes matching the following known sonde types.
-    _re = re.compile("^(DFM|M10|M20|IMET|IMET5|IMET54|MRZ|LMS6|IMS100|MTS01)-")
+    _re = re.compile("^(DFM|M10|M20|IMET|IMET5|IMET54|MRZ|LMS6|IMS100|MTS01|RS11G)-")

     # If we have a match, return the trailing part of the serial, re-adding
     # any - separators if they exist.
@@ -181,6 +181,10 @@ def short_type_lookup(type_name):
         return "Intermet Systems iMet-5x"
     elif type_name == "MEISEI":
         return "Meisei iMS-100/RS-11"
+    elif type_name == "IMS100":
+        return "Meisei iMS-100"
+    elif type_name == "RS11G":
+        return "Meisei RS-11G"
     elif type_name == "MRZ":
         return "Meteo-Radiy MRZ"
     elif type_name == "MTS01":
@@ -217,6 +221,10 @@ def short_short_type_lookup(type_name):
         return "iMet-5x"
     elif type_name == "MEISEI":
         return "iMS-100"
+    elif type_name == "IMS100":
+        return "iMS-100"
+    elif type_name == "RS11G":
+        return "RS-11G"
     elif type_name == "MRZ":
         return "MRZ"
     elif type_name == "MTS01":
AstroTaka commented 1 year ago

I uploaded a screen capture of Sonde List of after apply my previous posted patch. https://drive.google.com/file/d/1d4qdfG2Z6QySIpXWfwnbxilGch2ZeISL/view?usp=share_link

AstroTaka commented 1 year ago

I checked the latest beta21.

The type of RS-11G is "iMS-100" in Sonde List. I applied the following patch.

diff --git a/auto_rx/autorx/log_files.py b/auto_rx/autorx/log_files.py
index aa52955..b9eca94 100644
--- a/auto_rx/autorx/log_files.py
+++ b/auto_rx/autorx/log_files.py
@@ -67,6 +67,10 @@ def log_filename_to_stats(filename, quicklook=False):

         # Third field is the sonde type, in 'shortform'
         _type = _fields[2]
+
+        if _type=="MEISEI":
+            _type = _fields[1].split("-")[0]
+
         _type_str = short_type_lookup(_type)
         _short_type = short_short_type_lookup(_type)

diff --git a/auto_rx/autorx/utils.py b/auto_rx/autorx/utils.py
index 580a356..5afca67 100644
--- a/auto_rx/autorx/utils.py
+++ b/auto_rx/autorx/utils.py
@@ -181,6 +181,10 @@ def short_type_lookup(type_name):
         return "Intermet Systems iMet-5x"
     elif type_name == "MEISEI":
         return "Meisei iMS-100/RS-11"
+    elif type_name == "IMS100":
+        return "Meisei iMS-100"
+    elif type_name == "RS11G":
+        return "Meisei RS-11G"
     elif type_name == "MRZ":
         return "Meteo-Radiy MRZ"
     elif type_name == "MTS01":
@@ -217,6 +221,10 @@ def short_short_type_lookup(type_name):
         return "iMet-5x"
     elif type_name == "MEISEI":
         return "iMS-100"
+    elif type_name == "IMS100":
+        return "iMS-100"
+    elif type_name == "RS11G":
+        return "RS-11G"
     elif type_name == "MRZ":
         return "MRZ"
     elif type_name == "MTS01":

This a screen capture of Sonde List and SkewT of iMS-100 and RS-11G of beta21 https://drive.google.com/file/d/1ZOizjSA7uf2LDsHJNCaPXX3syVZ6DAa8/view?usp=share_link https://drive.google.com/file/d/1dd3srLD3aMkSmiheYq7uFbLVD8mOZ2PS/view?usp=share_link https://drive.google.com/file/d/1ScqUJBr2RJ3CZ_bOUapLFQizco-25mGl/view?usp=share_link

This a screen capture of Sonde List and SkewT of iMS-100 and RS-11G of patched beta21 https://drive.google.com/file/d/1dcs6kHp51vZmUaFuggGsuWJxS73xegZn/view?usp=share_link https://drive.google.com/file/d/1wfZuT5pMz-LQzWBzOCea2gVAUBZvXwzi/view?usp=share_link https://drive.google.com/file/d/1nHMApf2RDW8sKRJEKNnb6_fULDlinQW1/view?usp=share_link

darksidelemm commented 1 year ago

https://github.com/projecthorus/radiosonde_auto_rx/pull/734 has added in better (hopefully) handling of these two sonde types. Note that historical sondes with the 'MEISEI' type will not have the correct subtype displayed, this will only work for new sondes from now on.

Please test.

AstroTaka commented 1 year ago

OK, Thanks I will try the latest beta23.

AstroTaka commented 1 year ago

I checked beta24. I had a problem about decode iMS-100.

2022-12-25 14:39:40,073 WARNING:Payload ID IMS100-2105324 is invalid.
2022-12-25 14:39:41,076 DEBUG:Decoder (RTLSDR 00000002) MEISEI 404.503 - Couldn't generate APRS ID for IMS100-2105324

After apply the following patch, I have no issue to decode iMS-100.

diff --git a/auto_rx/auto_rx.py b/auto_rx/auto_rx.py
index 8450c26..bbb320e 100644
--- a/auto_rx/auto_rx.py
+++ b/auto_rx/auto_rx.py
@@ -621,7 +621,8 @@ def telemetry_filter(telemetry):

     # Check Meisei sonde callsigns for validity.
     # meisei_ims returns a callsign of IMS100-xxxxxx until it receives the serial number, so we filter based on the x's being present or not.
-    if "MEISEI" in telemetry["type"]:
+    logging.info("type:: %s, serial: %s" % (telemetry["type"],_serial))
+    if "MEISEI" in telemetry["type"] or "IMS100" in telemetry["type"] or "RS11G" in telemetry["type"]:
         meisei_callsign_valid = "x" not in _serial.split("-")[1]
     else:
         meisei_callsign_valid = False
2022-12-25 15:14:48,
113 INFO:type:: IMS100, serial: IMS100-2105324
2022-12-25 15:14:48,476 DEBUG:Telemetry Logger - Wrote line: 2022-12-25T06:14:46.000Z,IMS100-2105324,8944,36.8695
5,139.81265,12754.1,4.2,42.8,96.0,-48.4,16.7,-1.0,IMS100,404.503,16.8,-187,-1,-1.0,-1,-1

Please check.

AstroTaka commented 1 year ago

After applied my patch, I found another problem.

2022-12-26 09:04:19,105 INFO:type:: IMS100, serial: IMS100-2102060
2022-12-26 09:04:19,154 DEBUG:Telemetry Logger - Wrote line: 2022-12-26T00:04:17.020Z,IMS100-2102060,9220,35.73878,140.98130,12317.3,4.8,67.6,111.3,-46.9,0.8,-1.0,IMS100,405.704,8.3,1125,-1,-1.0,-1,-1
2022-12-26 09:04:21,011 DEBUG:Decoder (RTLSDR 00000002) MEISEI 405.704 - Couldn't generate APRS ID for IMS100-2102060

So I applied another patch.

diff --git a/auto_rx/autorx/utils.py b/auto_rx/autorx/utils.py
index 88023c0..8e502fa 100644
--- a/auto_rx/autorx/utils.py
+++ b/auto_rx/autorx/utils.py
@@ -285,7 +285,7 @@ def generate_aprs_id(sonde_data):
             _id_hex = hex(_id_suffix).upper()
             _object_name = "LMS6" + _id_hex[-5:]

-        elif "MEISEI" in sonde_data["type"]:
+        elif "MEISEI" in sonde_data["type"] or "IMS100" in sonde_data["type"] or "RS11G" in sonde_data["type"]:
             # Convert the serial number to an int
             _meisei_id = int(sonde_data["id"].split("-")[-1])
             _id_suffix = hex(_meisei_id).upper().split("0X")[1]

I will check the next launch.

darksidelemm commented 1 year ago

As usual, I change one thing and break another... I'll look at this later tonight.

AstroTaka commented 1 year ago

I have another information.

After applied two patch, my auto_rx try to upload to sondehub. But sondehub rejected. Please check the following error result.

2022-12-26 10:14:58,011 DEBUG:Sondehub Uploader - Sondehub reported issue when adding packets to DB. Status Code: 202 {"message": "some or all payloads could not be processed", "errors": [{"error_message": "Payload ID 2102060 from Sonde
type iMS-100 is invalid.", "payload": {"software_name": "radiosonde_auto_rx", "software_version": "1.6.0-beta24", "uploader_callsign": "AstroTaka-1", "uploader_position": null, "uploader_antenna": "5/8 wave mobile whip + Nooelec LNA", "t
ime_received": "2022-12-26T01:14:42.420068Z", "datetime": "2022-12-26T01:14:40.010000Z", "manufacturer": "Meisei", "type": "iMS-100", "serial": "2102060", "frame": 17666, "lat": 35.46686, "lon": 141.57031, "alt": 14246.8, "humidity": 0.7
, "vel_v": -10.95769, "vel_h": 62.65948, "heading": 102.9, "frequency": 405.702, "ref_position": "MSL", "ref_datetime": "UTC", "snr": 8.8, "user-agent": "Amazon CloudFront", "position": "35.46686,141.57031"}}, {"error_message": "Payload
ID 2102060 from Sonde type iMS-100 is invalid.", "payload": {"software_name": "radiosonde_auto_rx", "software_version": "1.6.0-beta24", "uploader_callsign": "AstroTaka-1", "uploader_position": null, "uploader_antenna": "5/8 wave mobile w
hip + Nooelec LNA", "time_received": "2022-12-26T01:14:44.025161Z", "datetime": "2022-12-26T01:14:41.010000Z", "manufacturer": "Meisei", "type": "iMS-100", "serial": "2102060", "frame": 17668, "lat": 35.46674, "lon": 141.57098, "alt": 14
236.1, "humidity": 0.7, "vel_v": -10.80336, "vel_h": 62.4537, "heading": 102.0, "frequency": 405.702, "ref_position": "MSL", "ref_datetime": "UTC", "snr": 10.1, "user-agent": "Amazon CloudFront", "position": "35.46674,141.57098"}}, {"err
or_message": "Payload ID 2102060 from Sonde type iMS-100 is invalid.", "payload": {"software_name": "radiosonde_auto_rx", "software_version": "1.6.0-beta24", "uploader_callsign": "AstroTaka-1", "uploader_position": null, "uploader_antenn
a": "5/8 wave mobile whip + Nooelec LNA", "time_received": "2022-12-26T01:14:44.627639Z", "datetime": "2022-12-26T01:14:42.010000Z", "manufacturer": "Meisei", "type": "iMS-100", "serial": "2102060", "frame": 17670, "lat": 35.46662, "lon"
: 141.57165, "alt": 14225.6, "humidity": 0.7, "vel_v": -10.54614, "vel_h": 62.65948, "heading": 102.6, "frequency": 405.702, "ref_position": "MSL", "ref_datetime": "UTC", "snr": 10.1, "user-agent": "Amazon CloudFront", "position": "35.46
662,141.57165"}}, {"error_message": "Payload ID 2102060 from Sonde type iMS-100 is invalid.", "payload": {"software_name": "radiosonde_auto_rx", "software_version": "1.6.0-beta24", "uploader_callsign": "AstroTaka-1", "uploader_position":
 null, "uploader_antenna": "5/8 wave mobile whip + Nooelec LNA", "time_received": "2022-12-26T01:14:45.531278Z", "datetime": "2022-12-26T01:14:43.010000Z", "manufacturer": "Meisei", "type": "iMS-100", "serial": "2102060", "frame": 17672,
 "lat": 35.46649, "lon": 141.57232, "alt": 14214.9, "humidity": 0.7, "vel_v": -10.49469, "vel_h": 62.60803, "heading": 103.6, "frequency": 405.702, "ref_position": "MSL", "ref_datetime": "UTC", "snr": 10.1, "user-agent": "Amazon CloudFro
nt", "position": "35.46649,141.57232"}}, {"error_message": "Payload ID 2102060 from Sonde type iMS-100 is invalid.", "payload": {"software_name": "radiosonde_auto_rx", "software_version": "1.6.0-beta24", "uploader_callsign": "AstroTaka-1
", "uploader_position": null, "uploader_antenna": "5/8 wave mobile whip + Nooelec LNA", "time_received": "2022-12-26T01:14:53.252515Z", "datetime": "2022-12-26T01:14:50.010000Z", "manufacturer": "Meisei", "type": "iMS-100", "serial": "21
02060", "frame": 17686, "lat": 35.46566, "lon": 141.5769, "alt": 14142.8, "humidity": 0.6, "vel_v": -9.9288, "vel_h": 60.80747, "heading": 102.3, "frequency": 405.702, "ref_position": "MSL", "ref_datetime": "UTC", "snr": 8.8, "tx_frequen
cy": 405.7, "user-agent": "Amazon CloudFront", "position": "35.46566,141.5769"}}, {"error_message": "Payload ID 2102060 from Sonde type iMS-100 is invalid.", "payload": {"software_name": "radiosonde_auto_rx", "software_version": "1.6.0-b
eta24", "uploader_callsign": "AstroTaka-1", "uploader_position": null, "uploader_antenna": "5/8 wave mobile whip + Nooelec LNA", "time_received": "2022-12-26T01:14:55.159996Z", "datetime": "2022-12-26T01:14:52.010000Z", "manufacturer": "
Meisei", "type": "iMS-100", "serial": "2102060", "frame": 17690, "lat": 35.46543, "lon": 141.57821, "alt": 14122.4, "humidity": 0.6, "vel_h": 60.85892, "heading": 102.5, "frequency": 405.702, "ref_position": "MSL", "ref_datetime": "UTC",
 "snr": 8.1, "tx_frequency": 405.7, "user-agent": "Amazon CloudFront", "position": "35.46543,141.57821"}}]}.

"type": "iMS-100" seems to be root cause I think. So I checked a source code of sondehub.

https://github.com/projecthorus/sondehub-tracker/blob/fc465b0cd4baf725436b1570cc946d7a7f6e3dfa/js/tracker.js#L143

There is "MEISEI" and "IMS100" but no exist "iMS-100" and "RS-11G".

darksidelemm commented 1 year ago

That issue is actually elsewhere, in the sondehub data input infrastructure, but i've got a fix in the works.

AstroTaka commented 1 year ago

OK, Thanks!

darksidelemm commented 1 year ago

OK, i've added in your patches. No version bump.

AstroTaka commented 1 year ago

There is no problem about upload Sondehub now. Thank you very much.

AstroTaka commented 1 year ago

I checked SondeList with the latest iMS-100 receive combination. Type of SondeList and SkewT is OK.

I checked SondeList with renamed RS11-G log combination. I renamed a file from 20221125-115209_RS11G-7135153_MEISEI_405197_sonde.log to 20221125-115209_RS11G-7135153_RS11G_405197_sonde.log. Type of SondeList and SkewT is OK also.

So I think that we can close this issue.

AstroTaka commented 1 year ago

@darksidelemm I have one question about utils.py.

This line seems to be not correct.

https://github.com/projecthorus/radiosonde_auto_rx/blob/68a3a296697441b75797b6839c208982d31eafda/auto_rx/autorx/utils.py#L288

This is correct I think.

elif "MEISEI" in sonde_data["type"] or "IMS100" in sonde_data["type"] or "RS11G" in sonde_data["type"]:

darksidelemm commented 1 year ago

Yep that's a bug, I've just merged a fix.

AstroTaka commented 1 year ago

Thanks