support data file format version 9

[dneuge]

Check for changes since data format 8.

Noticed so far on data files:

• preamble was reduced to just list !CLIENTS field syntax
  • notice about minimum interval of 2 minutes between retrievals has gone
• !GENERAL section:
  • new UNIQUE USERS count
• !PREFILE section:
  • includes VATSIM ID
  • retains flight plans after users logged in
  • may contain duplicates
• !VOICE SERVERS section has been removed (obsolete since AFV)

Noticed on network status file:

• servers.live and servers.dev were added
• no notice about fetch intervals

[dneuge]

Monitor prefiling data over multiple days, especially on days with lower load (i.e. not on weekends). Not sure if prefilings being kept indefinitely and in duplicates is normal behaviour. This could lead to severe problems on graph import as it is right now.

[dneuge]

Additional to points already noticed above:

• character set appears to be a little messed up (used to be plain undefined 8-bit, now looks like double encoded UTF-8 sometimes)
• !CLIENTS section:
  • structure:
  • pilots information appears to have remained unchanged
  • ATC information has new line-break sequence in ATIS message
  • ATC no longer lists a voice server
  • data (constants for ratings etc.) still need to be checked
• !SERVERS section unchanged
• !PREFILE section:
  • no client type (OK, same as before)
  • zeros for lat/lon, GS and altitude, protocol version, rating, transponder, facility type, visual range, heading and QNH (both inHg and hPa)
  • heading being 0 breaks client type guessing
  • rating being 0 breaks prefiling plausability check
  • revision counts up from 0 (may be higher if revised before logon)
  • duplicates are caused if revised
  • actual departure time = planned departure time
  • time of last ATIS and time of logon are garbage
  • flight plans are now really retained for an indefinite time after logon; prefiled flight plan revision stops incrementing after logon

[dneuge]

Although VERSION indicates 8 again, it is still format version 9 for compatibility with too restrictive parser: https://forums.vatsim.net/topic/26710-invalid-vatsim-data-file-version/

It was stated that the format version bump was only due to UNIQUE USERS having been introduced. Data may have been wonky for other reasons; together with #5 it seems that maybe data server has just been switched to a complete reimplementation or old data file format is now generated from JSON or whatever...

[dneuge]

Character set detection has been split to #6.

[dneuge]

Status file complete. Graph import is not part of parser; check and open separate issue if needed. Character set currently breaks line split and increases the amount of broken free-text; not fixed this time, see #6.

Remaining todo:

• check parser for network information