Open vanderaj opened 3 years ago
During discussions, there's a good argument for eliminating taxi, multicrew, and realistically Odyssey vs Horizons is of limited value too. EliteBGS cares primarily about the missing system and faction information, and would appreciate any improvements to the quality of data supplied, such as marketID, stationFaction, etc.
Multicrew
and Taxi
attributes are only present in the journal when you are in physical multicrew or in a taxi . These should probably be removed, as they indicate what mode of transport the player took.
horizons
and odyssey
are added to events by some tools (copied from the LoadGame
event) to indicate which game client generated the data.
PowerPlayState
and Powers
are only present when a player is pledged to a power and the system is controlled or exploited by a power.
Docked - body, bodyType. These are not an omission relative to the schema. Rather, EDMC is sending extra data not currently called for by the schema. https://github.com/EDCD/EDDN/blob/master/schemas/journal-v1.0.json
Location (populated or otherwise) - Many of the noted fields are not always written to the player journal, depending on the specific location of the commander. Station data for example is only written when docked. DistFromStarLS
may not be written if "close to the main star" per the journal manual. This may be less a problem with consistency between clients and more a problem with inconsistent source data.
@klightspeed is correct about powers and state flags
stationAllegiance
is sometimes missing from source events in the journal itself - it's a "known" if rare bug
My observations are that re-engineering any of the current schema to an updated schema is high cost. There are multiple listeners out there.
I think you need to accept that the responsibility is on the listeners to be robust. Accept that clients will be in a variety of states (there is no way to mandate updates either by the programmers or the users).
Odyssey/Horizons must stay, we need to know what client is sending data for tools like EDDB.
interfaces.zip During a recent EliteBGS update to strongly type Journal data from EDDN to better allow data validation of incoming data, as well as incorporate local and improve tick detection, it was necessary to look at the state of data accepted by EDDN to understand what data we might see.
Methodology. I investigated ross.eddb.io looking for how different tools uploaded various types of journal schema messages. The systems used are: Deciat (populated, active, pending, and recovering states, many factions), HIP 36601 (anarchy with no factions but heaps of carriers), Maia (retreat pending, many active states), and HR 1172 (anarchy, 1 megaship, zero factions or pop). If replicating this research, please make sure you use systems with many different states as possible, but within a single tick. Note: I limited my research to tools that had results within the last tick in interesting systems, so the lesser-used tools have no representation even in busy systems like Deciat or an unpopulated system like HIP 36601.
As a result, I have the following recommendations for EDDN and tool data providers:
journal/1schema improvements:
Simplify the message.faction to have a single class Factions?: FactionElement[]
Some tools provide a Faction[] array, which is a subset of FactionElement[]
EDDI Journal FSD Jump subschema, populated system
EDDI - Journal Location subschema, populated system
This would simplify faction storage for EDDN and consumers, and improve faction data validation and processing by tools like EliteBGS, EDSM, or Inara. Where game journal data doesn't exist, by making the data optional within a common FactionElement will help simplify things.
Simply the faction state used by Station and System Factions
Most tools: Docked, FSD Jump, Location
stationFaction?: StationFaction systemFaction?: SystemFaction
Some tools, like GameGlass and EliteLogAgent
stationFaction?: StationFactionClass systemFaction?: StationFactionClass
NB: Note that StationFactionClass is different to most tools' definition of it, but it's the same as the other tools' SystemFaction definition.
Recommendation: Make a single FactionClass element that has the following definition:
Please with tool authors to submit a common definition for system and station faction updates, regardless of location, FSD Jump, populated or not, or docked.
Missing fields by certain tools
EliteBGS requires faction updates, and we care about little else, other tools and sites require a lot more detail than EliteBGS. Many tool data providers do not seem to supply data that might be possible to supply from the journal. I would encourage all tools to review the included .zip file to see if they could add additional missing fields in their updates to EDDN.
Noting that the game's journal may be missing data depending on gameplay or indeed client version (Odyssey versus Horizons), I have tried to account for this by using a single tick with different tools in a busy system used by many CMDRs (in this instance Deciat) using a wide variety of tools. Unpopulated or faction-less systems shouldn't have such variation, because gameplay journal entries should be very similar regardless of tick. I have tried to use the most recent journal entries to account for differences in EDH / EDO clients over time reporting variations as bugs are fixed or game loops change. Additionally, I only took into account one record from each tool, which could be a mistake. A better study might sample all records and then work out a maximal set for each tool in a single tick. Where a tool does not appear doesn't mean it doesn't have missing fields, it simply means there was insufficient data during the tick.
Some of the differences are explained by the presence of a populated system or not, or if the jump occurs from deep space or near a station (so gameplay may have affected these results!). This is perfectly acceptable, as tools can't supply data that is not in the game's journal.
Recommendation: Toolmakers should review their code compared to entries from EDDN from many, many players, their own tools' records and other tools' records to see if they are providing all the relevant fields.
Docked
FSD Jump - unpopulated
FSD Jump - populated
Location - unpopulated
Location - populated
These differences might be explainable by where Cmdrs landed or popped up. More research is required, but by honing in on missing fields, it might be a simple fix to see if the tool actually uploads this data at all, or it was just missing from the data I inspected.