Open janvgils opened 3 years ago
I've just verified that the telemetry submitter is created using the correct URL in case this was some silly error when parsing the SatYAML to get the URL. Other than this, I don't really have a clue, since the code is the same that is working well with SatNOGS DB (and also your own test server, I guess).
Would this work over HTTP instead of HTTPS? If so, that would make it much easier to see what is the difference between Mike's request and gr-satellites'. If you feel like debugging further, you could even take a look at the code in submit.py. It's quite simple, and you can easily change some of the request parameters, such as the version, to match Mike's, or print the parameters (params
) of the request.
Thanks for the update, I will have a look.
In meantime I will also reach out to the UVSQ-Sat team, maybe they can also help.
I'm taking a look at this right now. I have changed the URL to HTTP in the .yml
to be able to see with Wireshark what's happening. The POST request sent by gr_satellites looks completely broken due to some problem with Python types:
POST /api/V2/SIDS?b'noradID=99991&source=EA4GPZ&locator=longLat&longitude=3.0W&latitude=40.0N&version=1.6.6&frame=9882A89A9EA6E09882A89A9EA66303F00801C000008920031900000002387006C4000000151A05A1018007FC037220010228006C2009010900480067000080000BE82001141007FC20000095004F140A001C0005140D01670038FF55000100000D6101BC00300D630044000800DA00680180004D00EF005E017F004C00E90026018000291A05410180010000000C393B0000021200A40006000000130000A52F×tamp=2021-02-06T21%3A08%3A56.809Z' HTTP/1.1
Note the Python b''
that encodes a bytes()
object.
I wonder how this is working with SatNOGS or other servers.
The full HTTP request is as follows:
POST /api/V2/SIDS?b'noradID=99991&source=EA4GPZ&locator=longLat&longitude=3.0W&latitude=40.0N&version=1.6.6&frame=9882A89A9EA6E09882A89A9EA66303F00801C000008920031900000002387006C4000000151A05A1018007FC037220010228006C2009010900480067000080000BE82001141007FC20000095004F140A001C0005140D01670038FF55000100000D6101BC00300D630044000800DA00680180004D00EF005E017F004C00E90026018000291A05410180010000000C393B0000021200A40006000000130000A52F×tamp=2021-02-06T21%3A08%3A56.809Z' HTTP/1.1
Accept-Encoding: identity
Content-Type: application/x-www-form-urlencoded
Content-Length: 453
Host: amsat.electrolab.fr
User-Agent: Python-urllib/3.7
Connection: close
noradID=99991&source=EA4GPZ&locator=longLat&longitude=3.0W&latitude=40.0N&version=1.6.6&frame=9882A89A9EA6E09882A89A9EA66303F00801C000008920031900000002387006C4000000151A05A1018007FC037220010228006C2009010900480067000080000BE82001141007FC20000095004F140A001C0005140D01670038FF55000100000D6101BC00300D630044000800DA00680180004D00EF005E017F004C00E90026018000291A05410180010000000C393B0000021200A40006000000130000A52F×tamp=2021-02-06T21%3A08%3A56.809Z
So I guess that the UVSQ-SAT server is choking at the malformed URL, while the other servers ignore it and look it directly at the POST content.
Great find and an example of thinking out of the box, I need to to remember this one. The server receiving the strings still finds the required fields and just processes the data.
I have pushed 081e7d2cd78ebdfbfb107ebdcefd953bee6d26c0, which fixes the problem with b''
appearing into the URL. Now the request looks reasonable.
POST /api/V2/SIDS?noradID=99991&source=EA4GPZ&locator=longLat&longitude=3.0W&latitude=40.0N&version=1.6.6&frame=9882A89A9EA6E09882A89A9EA66303F00801C000008920031900000002387006C4000000151A05A1018007FC037220010228006C2009010900480067000080000BE82001141007FC20000095004F140A001C0005140D01670038FF55000100000D6101BC00300D630044000800DA00680180004D00EF005E017F004C00E90026018000291A05410180010000000C393B0000021200A40006000000130000A52F×tamp=2021-02-06T21%3A19%3A55.905Z HTTP/1.1
Accept-Encoding: identity
Content-Type: application/x-www-form-urlencoded
Content-Length: 453
Host: amsat.electrolab.fr
User-Agent: Python-urllib/3.7
Connection: close
noradID=99991&source=EA4GPZ&locator=longLat&longitude=3.0W&latitude=40.0N&version=1.6.6&frame=9882A89A9EA6E09882A89A9EA66303F00801C000008920031900000002387006C4000000151A05A1018007FC037220010228006C2009010900480067000080000BE82001141007FC20000095004F140A001C0005140D01670038FF55000100000D6101BC00300D630044000800DA00680180004D00EF005E017F004C00E90026018000291A05410180010000000C393B0000021200A40006000000130000A52F×tamp=2021-02-06T21%3A19%3A55.905Z
However I'm still getting errors:
Error while submitting telemetry: HTTP Error 405:
I can't really debug this using HTTP instead of HTTPS because the server returns a 301 Moved Permanently redirect to the HTTPS URL.
I noted two differences in the request from DK3WN that you pasted here. The noradID
was different (so I've tested also the NORAD used by DK3WN, and it doesn't fix the problem), and the colons (:
) in the timestamp are not escaped (I'm not sure if this could be a reason why the UVSQ-Sat server is choking).
I will test this with a recording to see if that makes any difference, my system is building as we speak. After the test I will share the results.
Nope, just tested it and the same result.
Error while submitting telemetry: HTTP Error 400: Bad Request
Error while submitting telemetry: HTTP Error 500:
Tweaking a bit the code I'm now able to print the body of the HTTP responses sent by the server:
For a 400 Bad Request, I get
Server: nginx
Date: Sat, 06 Feb 2021 21:52:27 GMT
Content-Length: 0
Connection: close
Vary: Accept, Cookie
Allow: GET, POST, HEAD, OPTIONS
Content-Security-Policy: img-src 'self' https://*.gravatar.com https://*.mapbox.com https://*.google-analytics.com data: blob: https://db-satnogs.freetls.fastly.net; worker-src 'self' blob:; default-src 'self' https://*.mapbox.com 'unsafe-inline' https://db-satnogs.freetls.fastly.net; child-src blob:; script-src 'self' https://*.google-analytics.com 'unsafe-eval' https://db-satnogs.freetls.fastly.net; frame-src blob:
X-Frame-Options: DENY
For a 500 I get
Server: nginx/1.15.9
Date: Sat, 06 Feb 2021 21:52:27 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: close
I'm inclined to think that the UVSQ-Sat server is broken. After fixing the problem with the b''
's, the HTTP request sent by gr_satellites looks completely reasonable as far as I can tell.
I had a better look at the cap files that I created and found the following difference:
When using the UVSQsatDecoder the time field is different:
Form item: "timestamp" = "2021-02-06T22:17:46Z"
Key: timestamp
Value: 2021-02-06T22:17:46Z
This structure isn't recognised as a valid field value and therefor the test SIDS database is rejecting it.
Comparing this to GetKISS+ I have the following:
Form item: "timestamp" = "2021-02-06T22:24:24.090Z"
Key: timestamp
Value: 2021-02-06T22:24:24.090Z
It could be possible that the UVSQ-SAT sids database is using a different timestamp field check and therefor rejects the frames.
I guess we need some help from the UVSQ-SAT team.
I tried to reproduce this error with other SIDS database servers but I am unable to. So it seems to be specific to the UVSQ-Sat SIDS configuration and I can't check anything further without a clear understanding of there SIDS server/database setup.
I am out of options.
I'm also out of options, other than rewriting the gr-satellites submit block using requests
instead of urllib
, which might fix things. I can't see anything wrong with the HTTP POST requests sent by gr-satellites, but for some reason the UVSQ-Sat server doesn't like them.
This is something to consider for further developments of the SIDS protocol. Maybe a reference implementation would help prevent these sorts of problems?
I'll keep the issue open in case someone from the UVSQ-Sat team wants to weigh in.
Good evening,
From the beginning of the UVSQ-SAT launch and activation I have tried to forward gr_satellite decoded frames to there own SIDS database, but without any luck. This evening I tried again and with the same outcome.
Every time I use gr_satellites with the
submit_tlm=yes
in${HOME}/.gr_satellites/config.ini
including the callsign, latitude and longitude I received anerror while submitting telemetry: HTTP Error 500
To make sure it isn't a database issue on the UVSQ-SAT server side, I have configured the SIDS forwarder made by DK3WN to forward the data to
https://amsat.electrolab.fr/api/V2/SIDS
and this is working as expected, no error 500.So for some reason the SIDS config in the satyaml file or the underlying code is causing an error 500.
Below, more information regarding the commands, files and output that I used or is generated.
gr_satellites UVSQ-SAT --wavfile UVSQ-Sat_9k6_satnogs_3585560_2021-02-03T09-49-28.wav --samp_rate 48e3 --f_offset 12e3 --hexdump
Below a string that is send by the DK3WN forwarder and that is excepted by the UVSQ-SAT database (I removed some private information)
Based on the earlier WSL issues I have also tested this on a Laptop running gr_satellites and there is a small difference, this is also generating
Error while submitting telemetry: HTTP Error 400: Bad Request
As far as I know, I have used all the options I could think of to debug this issue, if there are any others please let me know. I even created a cap file but forgot the transport is ssl encrypted 👎