New channel: OpenSky master data file

[TwinFan]

OpenSky provides a downloadable database: https://opensky-network.org/datasets/metadata/aircraftDatabase.csv (Already downloaded)

• [ ] Clarify licensing: Are we allowed to use the file?
• [x] Design first:
  • [ ] The file doesn't seem to be sorted in any way. How should we speed up access? Read all ICAOs in memory with their file location so we can seek there?
  • [x] How to update the file? Leave it to the user? Make a function to do so? (maybe separate issue)
• [x] Add a channel, which uses this file as a database
• [x] For each query the file is to be accessed (it's 80MB and too big to be cached in-memory).

[TwinFan]

In the end used a slightly different file (aircraft-database-complete-YYYY-MM.csv from the same metadata folder) with less fields, but those we need, so it is smaller (around 55MB). And it comes sorted. Still, as it is a text file it is difficult to randomly access specific lines. So implementation is

• Verify if the current month's file is available, if not download it. (Walk back up to 3 months to find one.)
• Open the file and once read all lines, saving the file position for every 250th line together with the aircraft id that line represents.
• When looking up data for a given aircraft id, find the nearest stored position and then read forward until the record is found (or the aircraft id in the file becomes larger that what we searched for, then the record we search for isn't in the file)

[TwinFan]

Published in v3.5.0