Update Open-Sky Network Private Single Flight API Example

DJDevon3 commented 3 months ago

Requires Open-Sky Network website account. Instructions in code comments. Allows for more daily calls than public API.

Added

pretty serial hierarchy view
general code cleanup and improvements
improved endpoints with server side failure messages
improved and streamlined debugging
shouty constants to make Pylint happy.
imports sorted
switched from circuitpython_base64 to built-in module binascii for base64 credential encoding
doc strings for functions
uses settings.toml by default
adafruit_circuitpython_requests now using Connection Manager by default

Serial output example:


Connecting to WiFi...
✅ Wifi!
 | Attempting to GET OpenSky-Network Single Private Flight JSON!
 | Website Credentials Required! Allows more daily calls than Public.
 | ✅ OpenSky-Network JSON!
 |  | Last Contact: 03/19/2024 09:49:10
 |  | Transponder: 4b1806
 |  | Callsign: SWR30M  
 |  | Squawk: 1000
 |  | Origin: Switzerland
 |  | Longitude: 8.7047
 |  | Latitude: 48.1189
 |  | Barometric Altitude: 3962.4
 |  | Velocity: 176.06
 |  | Vertical Rate: 0
✂️ Disconnected from OpenSky-Network API

Finished!
Board Uptime: 1.2 days
Next Update: 30 minutes
===============================

DJDevon3 commented 3 months ago

I messed up and published both the private and public API examples here. commit 8a34d2a is the private example commit 231e2eb is the public example The commits didn't merge because I published them from 2 branches. sigh, I don't know how to correct this.

dhalbert commented 3 months ago

The commits didn't merge because I published them from 2 branches. sigh, I don't know how to correct this.

You can revert a commit. Or fix it locally and do a push -f (force push).

justmobilize commented 3 months ago

@DJDevon3 any reason not to merge these into a single example? And base if there are creds, print out a different thing:

Connecting to WiFi...
✅ Wifi!
 | ✅ OpenSky credentials found, using private
 | Attempting to GET OpenSky-Network Single Flight JSON!

or

Connecting to WiFi...
✅ Wifi!
 | ❌ OpenSky credentials not found, using public
 | Attempting to GET OpenSky-Network Single Flight JSON!

FoamyGuy commented 3 months ago

I think I am in favor of keeping them separate. I wouldn't say that it makes a huge difference either way, but I like that the seperate public API one can be a bit less complex (losing the chunk of code responsible for base 64 encoding the auth), as well as drop the requirement for adafruit_binascii

I often start my projects by copying the most relavent library example for the first feature that I decide to work on, I appreciate having the seperated minimal examples because it tends to result in less unused stuff to find and cut out of the code.

FoamyGuy commented 3 months ago

@DJDevon3 I'm don't think I understand your comment about the two different commits for public / private versions of the examples.

Is that something that needs to get resolved before this PR can be considered for merging?

Maybe your intention was originally to do public and private in separate PRs? but they accidentally got both put into this one? If that is the case I think it's okay to just move forward with them both in this PR.

The changes look mostly fine to me, I'll go ahead and test them out but will hold off on approval / merging just in case there is something that needs to be done prior to that.

DJDevon3 commented 3 months ago

About "\n" appended to base64 hash:

I switched this and many other examples that relied on base64 encoding from the community library circuitpython_base64 to adafruit_binascii as suggested by Dan. I thought the "\n" on the end was part of the hash. Oops! Obviously I didn't do enough testing. Open-Skynetwork accepts it with the \n appended with or without the slice! for me anyway, and that's quite odd.

This isn't a real login, just an example base64 hash example

VXNlcm5hbWU6MDEyMzQ1Njc4OQ==
VXNlcm5hbWU6MDEyMzQ1Njc4OQ==\n

Open-Sky Network will accept either of these for me... and that's not good. I'm glad it fails for you. It should fail for me but it doesn't.

Issue with binascii library:

The binascii simpletest doesn't include a way to spit out the hashed string without b'encapsulation\n'. The library is adding the \n... WHY?!?! It's even demonstrated in the binascii simpletest. Both b2a and a2b decoding still retains it inside the byte encapsulation and requires padding! It's extremely picky about the format which makes me think binascii was originally designed for a very specific purpose.

Problem with adafruit_binascii for web base64 encoding:

If they would have provided an example to drill it down to a pure string I'm sure they would have also used slice[2:-3] or equivalent method to output to a pure string. I don't see a method to do that. This is a problem for passing the exact correct string into the header. As far as I can tell it's presumed that the developer (you and me) must use slice to get it the rest of the way. I have a lot of branch splitting and recommits to do today anyway, will definitely fix this which should be as simple as changing TRUNCATED_BASE64_STRING = BASE64_STRING[2:-1] to TRUNCATED_BASE64_STRING = BASE64_STRING[2:-3]

I might make a PR to adafruit_binascii so that all you have to do is pass a username:password string into it and it spits out a base64 encoded string. With http/s server getting a lot of work in Circuit Python it's just a matter of time before this becomes a necessary feature. If you know of a better way that already does this I'm all ears. This is the first time I'm using binascii, previously I'd always used circuitpython_base64.

FoamyGuy commented 3 months ago

Interesting about binascii adding the newline. It appears that behavior is inherieted from CPython: https://docs.python.org/3/library/binascii.html#binascii.b2a_base64 although in CPYthon the eventually added a newline argument to b2a_base64 in order to disable the newline being added https://github.com/python/cpython/issues/69544

I think we should add same newline argument and behavior into adafruit_binascii so it can match the CPython implementation.

I don't necessarily think that adafruit_binascii would be the appropriate place for a function that takes username:password and spits out the string ready to be added to a header unless the CPYthon version has that functionality. For the circuitpython libraries that match CPython named ones they should be a subset of the CPython one with matching behavior whenever possible. I do think the functionality itself would be valuable but should get added somewhere that matches a CPython API if possible.

In CPython requests it looks like there is HTTPBasicAuth which handles that https://requests.readthedocs.io/en/latest/user/authentication/ we could add that class here in the adafruit_requests library and then have matching APIs between circuitpython and CPython more closely which will make code more portable.

FoamyGuy commented 3 months ago

Actually it looks like there is also a core module binascii and it does already have the newline argument.

>>> import binascii
>>> dir(binascii)
['__class__', '__name__', '__dict__', 'a2b_base64', 'b2a_base64', 'crc32', 'hexlify', 'unhexlify']
>>> binascii.b2a_base64(b"hello")
b'aGVsbG8=\n'
>>> binascii.b2a_base64(b"hello", newline=False)
b'aGVsbG8='

I ran the above on a M5 Cardputer (ESP32 S3 based)

So it looks like you could use this and won't even need to have adafruit_binascii as a required library if I am understanding correctly (assuming this module is enabled in the core on all or most devices, I only tried 1 so far)

DJDevon3 commented 3 months ago

I did see that last night when looking into the library but was unsure how to use it or if it was a usable parameter. Thank you for the example. I'll take a look into updating them after the dust settles from all the PR's.

DJDevon3 commented 3 months ago

Figured out why I'm still able to login even with \n appended. It only checks the first 18 characters of the hash, ignores anything after lol. So \n should have also worked for you unless your hash is less than 20 characters.

VXNlcm5hbWU6MDEyMzQ1Njc4OQ==  # this will log you in
VXNlcm5hbWU6MDEyMzQ  #  so will this
VXNlcm5hbWU6MDEyMz  #  this fails to authorize

However if your base64 hash ends up being shorter than 18 characters it will check them all... I hope! I haven't tested it since I can't change my username to be shorter.

DJDevon3 commented 3 months ago

Also used this commit to delete the twitter api example and use ruff with pre-commit as an experiment.

DJDevon3 commented 3 months ago

This one is ready to go. Please merge this one first, then PR 172

FoamyGuy commented 3 months ago

Figured out why I'm still able to login even with \n appended. It only checks the first 18 characters of the hash, ignores anything after

Yikes! :fearful:

DJDevon3 commented 3 months ago

I'm using pylint ignore module import due to this.

C:\Users\Devon\Documents\GitHub\Adafruit_CircuitPython_Requests\examples\wifi\expanded>black requests_wifi_api_openskynetwork_private.py
All done! ✨ 🍰 ✨
1 file left unchanged.

C:\Users\Devon\Documents\GitHub\Adafruit_CircuitPython_Requests\examples\wifi\expanded>pylint requests_wifi_api_openskynetwork_private.py
************* Module requests_wifi_api_openskynetwork_private
requests_wifi_api_openskynetwork_private.py:10:0: E0401: Unable to import 'adafruit_connection_manager' (import-error)
requests_wifi_api_openskynetwork_private.py:11:0: E0401: Unable to import 'wifi' (import-error)

-------------------------------------------------------------------
Your code has been rated at 9.12/10 (previous run: 10.00/10, -0.88)

This is on my local pylint. It has helped me clean up a lot of errors before submitting but some I cannot get passed.

Will try to submit even with the error and see what happens.

DJDevon3 commented 3 months ago

Apparently that passes here but not on my local pylint. Makes me wonder how many other linting errors I've been fixing unnecessarily.

adafruit / Adafruit_CircuitPython_Requests