search

EDCD / EDDN

EDDN - Elite: Dangerous Data Network
https://eddn.edcd.io/
BSD 3-Clause "New" or "Revised" License
298 stars 59 forks source link

Including FSSSignalDiscovered event to list of allowed events to send over EDDN #152

Closed norohind closed 2 years ago

norohind commented 3 years ago

It will allow EDDN listeners to:

  1. Track FCs movements
  2. Track combat zones
  3. Track Resources extraction points (not quite useful since body info isn't included)
  4. Track NonHuman signals
  5. Track high grade emissions signals
  6. Track installations
  7. Match FCs IDs and names

In order to decide which fields have to be removed I analyzed all keys in all my own journals record for FSSSignalDiscovered event, here is the list with some comments:

SignalName
TimeRemaining - on USS, I don't think it can be useful for anybody but I also don't think it is private information
timestamp
SpawningFaction_Localised - candidate to removing
SignalName_Localised - candidate to removing
IsStation
USSType
USSType_Localised - candidate to removing
SpawningState
SpawningFaction
event
SpawningState_Localised - candidate to removing
ThreatLevel
SystemAddress

There is also an event type that can be considered as personal information, it is missions related signals

{ "timestamp":"2021-10-27T20:33:30Z", "event":"FSSSignalDiscovered", "SystemAddress":3107509342922, "SignalName":"$USS; ", "USSType":"$USS_Type_MissionTarget;", "USSType_Localised":"Mission target", "SpawningState":"$FactionState_None;", "SpawningState_Localised":"None", "SpawningFaction":"$faction_none;", "SpawningFaction_Localised":"None", "ThreatLevel":4 }

Also please note that every jump for single player can generate more than 100 FSSSignalDiscovered events.

Additional values: horizons odyssey StarSystem StarPos With appropriate values as in journal-v1.0.json schema.

Tkael commented 3 years ago

Localized names are already typically scrubbed from the EDDN messages, I see nothing controversial there.

I see limited value in reporting USS with TimeRemaining, since the time wherein each signal exists is quite limited and the most useful of these USS signals (HGE) can already be readily predicted from faction states and allegiances.

I would also agree with removing USS like the $USS_Type_MissionTarget; USS type event identified by the OP as personal information.

Combat zones (except perhaps for nonhuman combat zones) follow naturally from faction states as well - there is limited value to be gained from reporting individual combat zones to EDDN.

Without body information, is resource extraction data still useful?

Athanasius commented 3 years ago

There is an issue with some FSSSignalDiscovered events being written to the Journal before a jump has actually been made/completed, certainly before the matching FSDJump or CarrierJump event (maybe even before Location on login?).

Now, we could just leave this event as only having SystemAddress from the actual journal event. The point of those augmentations is for listeners to get 'full' data on a system they've not seen before without them needing to cross-check against data they hold. But this is not 100% necessary if it's going to be problematic.

If we insist on SystemName and StarPos then senders would need to cross-check that SystemAddress with a 'location' event before adding those augmentations. If it doesn't match then they can either:

  1. Just not send this event (but this loses data).
  2. Not include the augmentations in this mesage when sent.
Athanasius commented 3 years ago

Localized names are already typically scrubbed from the EDDN messages, I see nothing controversial there.

Agreed.

I see limited value in reporting USS with TimeRemaining, since the time wherein each signal exists is quite limited and the most useful of these USS signals (HGE) can already be readily predicted from faction states and allegiances.

Yes, no point to including this.

I would also agree with removing USS like the $USS_Type_MissionTarget; USS type event identified by the OP as personal information.

Yup, personal, and not really useful to anyone else.

Combat zones (except perhaps for nonhuman combat zones) follow naturally from faction states as well - there is limited value to be gained from reporting individual combat zones to EDDN.

Knowing how many CZ there are, and if the data is there, of what type/size, can be useful, even if you don't yet know exactly where they are. I'd leave this up to listeners to decide on the usefulness.

Without body information, is resource extraction data still useful?

Again, it might be useful, so I see no reason to not include it.

Someone has mentioned that perhaps some sort of batching of these events into a single message might be useful. Timing that post-jump might be tricky, but should be considered.

clonedArtie commented 3 years ago

Yes, the FSSSignalDiscovered can happen even before Location event at the startup, so to avoid complications (and possible incorrect values) with stuff like star system coords and name I suggest to have these properties as SHOULD rather than MUST.

The batching is a MUST, to not compromise the server (I can clearly imagine that for highly populated system congested with fleet carriers, with just a few senders at the same moment the server will need to handle hundreds of HTTP requests at once instead of just a couple of requests. I guess it will be better even for the users as their machines won't be making 100+ HTTP requests at the same time.

It can follow an easy mechanics in journal parsing:

  1. If the event is FSSSignalDiscovered, store it to the temporal list and proceed to next event.
  2. If the next event is also FSSSignalDiscovered, repeat 1.
  3. If the next event is any other or there is no other event for more than 10 seconds, send the temporal list in a single message to EDDN.
  4. ???
  5. Profit! ;)
chennin commented 3 years ago

CZs/RES

The difficulty is useful information -- I've seen requests for this information. Even without body information it's better than nothing

Athanasius commented 3 years ago

Something just mentioned on Discord is that currently Canonn gain the names (rather than just call signs) of Fleet Carriers from FSSSignalDiscovered events. Currently EDDB etc only get data with the call sign, giving no way to map them to the names.

The format is a little awkward though, as you get both the name and the call sign in the same field:

{ "timestamp":"2021-10-18T08:42:23Z", "event":"FSSSignalDiscovered", "SystemAddress":3240309557611, "SignalName":"WARD'S OLOGIES X3F-N5Z", "IsStation":true }

clonedArtie commented 3 years ago

It's easy to parse though. I certainly recommend for the specs that senders MUST NOT alter the SignalName any way.

norohind commented 3 years ago

I also think messages should be transmitted as is with minimum changes. Receivers might want to implement some special logic to parse it.

chennin commented 3 years ago

Confirming those 14 fields are all I have in my journals

Possible formats:

USS - the following may or may not appear:

Station/FC

{ "timestamp": "2021-10-30T17:43:06Z", "event": "FSSSignalDiscovered", "SystemAddress": 84456968626, "SignalName": "BUYING PAINITE 230K H8N-TXQ", "IsStation": true }

{ "timestamp": "2021-10-30T12:40:33Z", "event": "FSSSignalDiscovered", "SystemAddress": 5068732310921, "SignalName": "Al-Jazari Orbital", "IsStation": true }

CZ/RES/etc

{ "timestamp": "2021-10-29T11:56:53Z", "event": "FSSSignalDiscovered", "SystemAddress": 1733119906514, "SignalName": "$Warzone_PointRace_High:#index=1;", "SignalName_Localised": "Conflict Zone [High Intensity]" }


Full list of `$` signal names I've encountered. I've noted the indexes I've seen, but there *could* be larger - includes USS (above)

$Aftermath_Large:#index=1; # Goes from 1-2 $AttackAftermath; $FIXED_EVENT_CAPSHIP; $FIXED_EVENT_CHECKPOINT; $FIXED_EVENT_CONVOY; $FIXED_EVENT_DEBRIS; $FIXED_EVENT_DISTRIBUTIONCENTRE; $FIXED_EVENT_HIGHTHREATSCENARIO_T5; $FIXED_EVENT_HIGHTHREATSCENARIO_T6; $FIXED_EVENT_HIGHTHREATSCENARIO_T7; $Fixed_Event_Life_Cloud; $Fixed_Event_Life_Ring; $Gro_controlScenarioTitle; $ListeningPost:#index=1; $MULTIPLAYER_SCENARIO14_TITLE; $MULTIPLAYER_SCENARIO42_TITLE; $MULTIPLAYER_SCENARIO77_TITLE; $MULTIPLAYER_SCENARIO78_TITLE; $MULTIPLAYER_SCENARIO79_TITLE; $MULTIPLAYER_SCENARIO80_TITLE; $MULTIPLAYER_SCENARIO81_TITLE; $NumberStation:#index=1; $USS; # This one has a trailing space! $USS_CeremonialComms; $USS_ConvoyDispersalPattern; $USS_DegradedEmissions; $USS_DistressCall; $USS_HighGradeEmissions; $USS_NonHumanSignalSource; $USS_SalvageHaulageWreckage; $USS_Type_Aftermath; $USS_Type_Convoy; $USS_Type_DistressSignal; $USS_Type_NonHuman; $USS_ThreatLevel:#threatLevel=4; # Goes from 4-9. Yes this is one entry $USS_Type_WeaponsFire; $USS_WeaponsFire; $Warzone_PointRace_High:#index=1; # Goes from 1-3 $Warzone_PointRace_Low:#index=1; # Goes from 1-10 $Warzone_PointRace_Med:#index=1; # Goes from 1-4

Installations/Beacons/Outposts(!)

{ "timestamp": "2021-10-26T21:13:26Z", "event": "FSSSignalDiscovered", "SystemAddress": 121569805492, "SignalName": "Enlightened Mathematic Analysis" } { "timestamp": "2021-10-29T12:02:34Z", "event": "FSSSignalDiscovered", "SystemAddress": 5068732310921, "SignalName": "SVA-046 Dionysus-class Cropper" } { "timestamp": "2021-10-26T20:25:20Z", "event": "FSSSignalDiscovered", "SystemAddress": 24859942069665, "SignalName": "What is Interpol?" } { "timestamp": "2021-10-26T21:35:04Z", "event": "FSSSignalDiscovered", "SystemAddress": 4305444669811, "SignalName": "Sekelj Port" }



For my reference as I don't see a list on Google, EN localized names for the non-obvious ones:

$FIXED_EVENT_HIGHTHREATSCENARIO_T5; $FIXED_EVENT_HIGHTHREATSCENARIO_T6; $FIXED_EVENT_HIGHTHREATSCENARIO_T7;: Pirate Activity Detected
$MULTIPLAYER_SCENARIO14_TITLE;: "Resource Extraction Site"
$MULTIPLAYER_SCENARIO42_TITLE;: "Nav Beacon"
$MULTIPLAYER_SCENARIO77_TITLE;: "Resource Extraction Site [Low]"
$MULTIPLAYER_SCENARIO78_TITLE;: "Resource Extraction Site [High]"
$MULTIPLAYER_SCENARIO79_TITLE;: "Resource Extraction Site [Hazardous]"
$MULTIPLAYER_SCENARIO80_TITLE;: "Compromised Navigation Beacon"
$MULTIPLAYER_SCENARIO81_TITLE;: "Salvageable Wreckage"
norohind commented 3 years ago

Out of curiousity, have you got any with IsStation = False?

chennin commented 3 years ago

Nope, none

kenjij commented 2 years ago

One valuable info that can be obtained from this schema would be Nav Beacon status.

{ "timestamp":"2021-11-12T05:16:07Z", "event":"FSSSignalDiscovered", "SystemAddress":147949848947, "SignalName":"$MULTIPLAYER_SCENARIO42_TITLE;", "SignalName_Localised":"Nav Beacon" }

{ "timestamp":"2021-11-12T05:15:10Z", "event":"FSSSignalDiscovered", "SystemAddress":3996240546163, "SignalName":"$MULTIPLAYER_SCENARIO80_TITLE;", "SignalName_Localised":"Compromised Navigation Beacon" }
Athanasius commented 2 years ago

For anyone following this and wondering if it's been forgotten, it hasn't.

However I do have some concerns about the possible flood of data adding this would cause, so there are some caveats.

  1. We need a sender implementation - if anyone wants to beat me to a PR against ED Market Connector, please do. The purpose of this is to actually test it in action to ensure that we identify any issues with things like batching the events for arrival in a new system into just a single message. During this testing phase it should only attempt sending to beta.eddn.edcd.io (which isn't always up and working).
  2. That will match up with a beta implementation of the schema running on beta.eddn.edcd.io.
  3. We then need to at least get an idea of the message and data rate when e.g. a commander flies around the populated (more signals from things like FCs) bubble for an hour. We'd then combine this with some guesstimate of how many Cmdrs might do that over the course of a week to get an idea of any extra load it would cause on both the EDDN Gateway and listeners.
  4. Some listeners may well not want this data at all. Ideally they would use ZeroMQ features to filter it out. However the version of zeromq that is supported by the legacy version we're currently using for all of EDDN doesn't perform server-side subscription filtering, i.e. the data would get sent to listeners anyway, even if they then subscription-filter it on their end. Thus:
    1. We would need to complete the migration to Python3 and verify that server-side subscription filtering works with the version that uses.
    2. Verify that we are definitely setting ZeroMQ topic correctly per message so that this actually functions.

So, apart from anything else this is now dependent on first completing the migration to Python3. If no-one has the ability and time to assist me in verifying that https://github.com/EDCD/EDDN/pull/155 doesn't break any functionality or introduce regressions, then this is further dependent on implementing as many automated unit and functional tests as possible. That's a whole project in itself.

chennin commented 2 years ago

I parsed all my journals. Of hours that have at least one FSSSignalDiscovered:

Top 10:

Hour FSSSignalDiscoveredCount 2022-03-26T18 3963 2022-03-25T18 3019 2021-09-20T12 2780 2022-03-25T17 2376 2022-06-11T23 2360 2022-06-11T21 1834 2022-03-25T19 1729 2021-09-20T14 1672 2022-03-25T00 1613 2022-01-08T14 1534

Median: 64
Average: 132

If anyone else wants to measure, get your journals on a Linux-like system, then here's one way to do it:

for i in $(ls Journal*); do grep FSSSignalDiscovered $i | jq -r '.timestamp'  | awk -F: '{ count[$1]++ } END { for (i in count) print i, count[i] }' ; done | tee output
#top: 
cat output | sort -k2 -nr | head
#median: 
cat output | sort -k2 -nr | awk '{print $2}' | awk '{a[NR]=$0}END{print(NR%2==1)?a[int(NR/2)+1]:(a[NR/2]+a[NR/2+1])/2}'
#average: 
cat output | sort -k2 -nr | awk '{print $2}' | awk '{x+=$0}END{print x/NR}'
Athanasius commented 2 years ago

The new schema has been live for many weeks now without anyone bringing up an issue, so closing this.