New VATSIM datafeed format coming

[FlyingFoxVATSIM]

https://forums.vatsim.net/blogs/entry/1-vatsim-tech-blog-q4-2020/

[dneuge]

Not coming, it's already there. ;)

Important excerpts from that announcement:

Moving to production on the 3rd December 0001Z!

The structure for this new format can be viewed at https://api.vatsim.dev/#operation/DataServerV3 The file is retrievable at https://data.vatsim.net/v3/vatsim-data.json

(first link doesn't work though)

Retirement Notice - Txt Data File and JSON v1 These versions of the data will be retired on, or closely following January 1st of 2021, subject to communication with other solutions. We highly encourage all users of the old text file to migrate to the new JSON feed which can be found in more detail above.

Looks like migration to the new JSON format is needed ASAP and a release supporting it should be made until end of the year if that date holds up. Given that the new format was unofficial until April and not well communicated even after that (more like a "hidden feature" for a long time) it feels a bit over-ambitious to announce the shutdown with just a little less than a month remaining to get support implemented everywhere. I'm wondering how all other projects affected by that change will react as the news spreads, I would be a little surprised if VATSIM really sticks to that over-ambitious schedule...

[twSimulations]

Probably you follow the development already. In case if you don't, here is the latest announcement regarding datafeed URLs:

https://forums.vatsim.net/topic/29914-datafeed-urls/

[dneuge]

Interesting question, I would have assumed that file was supposed to be deprecated as well but maybe it is still needed for pilot clients (url1 entries) to find FSD servers.

By the way, the transition period was extended to 1 March 2021 (see original announcement).

[droesen]

It's vital we get a JSON feed parser into a released QuteScoop version, really. :-S

[dneuge]

The format itself does not hold too many surprises:

• old clients section has been split into controllers (regular online ATC but also lists OBS pilots), atis (contains only ATIS stations) and pilots (online), prefilings are still separate as prefiles
• those sections only provide fields that are considered relevant (e.g. controllers no longer have coordinates and pilots no longer have a visibility range, controller message field or frequency)
• timestamps are encoded as ISO 8601
• pilot ratings have been added to pilots
• Enums (facility type, controller rating, pilot rating) are described in separate sections in JSON v3. Instead of hard-coding the IDs (as was done before) it would be better to resolve the IDs and use the short and long names as provided by those enums. For example if a controllers object refers to a rating of ID 3 the enum object from (controller) ratings with id 3 should be looked up which currently gives a short name of S2 and a long name of Tower Controller. If at any point VATSIM would decide to rename those no changes would be required in live clients such as QuteScoop.
• Some fields have gone but I don't think those should be needed by QuteScoop anyway (airport coordinates, flight plan revision, "actual" departure time, client protocol version, controller message update timestamp).
• "last updated" timestamp is somewhat weird, for prefiles it appears to be the time of last edit but every other object (atis, controllers, pilots) seems to be more like a timestamp of when the data server created the snapshot? However, I don't think that information is needed by QuteScoop anyway.

I think it's rather easy (but still quite time-consuming) to adapt QuteScoop to the JSON v3 format. The Qt5 JSON parser also looks very easy to use and does not appear to require any schema.

What I don't know is if support for the old format needs to be kept: QuteScoop can (and was?) used for other networks than VATSIM. Do those networks still use the old format and are currently compatible with QuteScoop? If yes, then instead of working directly with the new VATSIM-specific JSON files (which would make QuteScoop usable only for VATSIM) it would probably be better to translate from the new format to the old data structures?

[dneuge]

Probably you follow the development already. In case if you don't, here is the latest announcement regarding datafeed URLs:

https://forums.vatsim.net/topic/29914-datafeed-urls/

There has been an update today:

As with the old datafiles, we request that you do NOT harcode the datafile URLs, and instead download the file available at https://status.vatsim.net/status.json once when your application starts and then select one of the URLs available randomly each time you make a request.

Unless I misunderstand that, the old "index" file should be avoided and instead URLs should be read from the new status.json when making the switch.

[jonaseberle]

Thank you for that information! I don't think that's too hard to do.

But first of all we should have working builds and releases at all.

[scrvl]

I take it there's not been an update for the new data format yet then? Asking because my current version has now stopped receiving updates from the old .txt url.

[diibify]

There is a json parser for that - you can find the project https://github.com/dneuge/legacy-status-proxy-vatsim May be someone is going to implement it to QuteScoop itself one day.

[dneuge]

Before native support can be added to QuteScoop some decision will probably need to be made on how to continue with this project. Quoting myself from above:

What I don't know is if support for the old format needs to be kept: QuteScoop can (and was?) used for other networks than VATSIM. Do those networks still use the old format and are currently compatible with QuteScoop? If yes, then instead of working directly with the new VATSIM-specific JSON files (which would make QuteScoop usable only for VATSIM) it would probably be better to translate from the new format to the old data structures?

IVAO apparently can still be accessed. I see two options:

1. remove Whazzup support to rework everything towards the new data format (meaning no more compatibility with IVAO or whatever other networks QuteScoop might have been used for) or
2. find a good way to continue support for the old Whazzup formats and the new VATSIM-only JSON format

Either way I would recommend to keep the service URL configurable. The proxy server was written with compatibility between future JSON formats in mind, so that if for example a version 4 or 5 becomes available it can be translated back to, for example, the current JSON format version 3. The proxy is not limited to just JSON-Whazzup conversion (as is needed right now).

Whoever currently has the role of project lead will need to make that decision. I don't see how anyone could start to reasonably work on the migration and come up with a good maintainable solution before a decision has been made.

Personally, I don't feel comfortable starting to work on such a migration myself as I lack experience in C++ programming. While I can of course write basic C++ code it is far from modern standards and I make very, very slow progress as I rarely work in C++. Plus, I already have other unfinished projects at hand that need my attention, so I would not like to start working on yet another open end.

[jonaseberle]

Everything is fine for me. Just go ahead ;) We did only get feedback from IVAO users once or twice in the last 10 years. If it makes things easier, just drop support for that network.