EDCD / EDMarketConnector

Downloads commodity market and other station data from the game Elite: Dangerous for use with all popular online and offline trading tools.
GNU General Public License v2.0
991 stars 155 forks source link

EDDN: Some `codexentry` messages are invalid #1393

Closed Athanasius closed 2 years ago

Athanasius commented 2 years ago

The new EDDN logging has brought to light things like:

2022-01-07 14:45:20,076 - ERROR - Gateway:233: Failed Validation "[<ValidationError: "'' is too short">]" (729, "E:D Market Connector [Windows]", "5.2.4", "https://eddn.edcd.io/schemas/codexentry/1", "-")

Presumably we're allowing through some key where the value is an empty string, whereas the schema thinks there should always be a value of at least one character. This might mean relaxing the EDDN schema, or it might mean we need to do some checks and pre-processing.

Athanasius commented 2 years ago

Checking the schema could be any of:

  1. System (name) - unlikely.
  2. Name - also unlikely I'd hope.
  3. Region - probably not?
  4. Category - please tell me Frontier set this on them all.
  5. SubCategory - perhaps not always applicable?
  6. Traits - but I'd hope it would just not populate the array if so.

Looking at my own examples of codexentry events all of them have everything but Traits, of which I have zero examples. For all I know it somtimes looks like:

Traits: [
 ''
]

or

Traits: [
  'foo',
  ''
]
Athanasius commented 2 years ago

It looks like this might be due to a game bug, sometimes writing '' as the System (name) value. CometBorne supplied a file of all their CodexEntry events, and in it I found over 8000 such instances, the latest being:

<removed>

That system is Pheia Aewsy LV-Y d11 (EDSM).

We probably want to start doing some validation of CodexEntry data in the EDDN plugin.

Ignore all of that. I goofed a jq filter, and set those fields to "", rather than testing if they already were.

Athanasius commented 2 years ago

Closing in favour of #1431 , under the assumption this was some older messages being replayed that were recorded under an older, buggier version of EDMC and/or the game.

NB: We added codexentry schema support in 5.2.0 - 2021-10-25.