EDSM-NET / FrontEnd

Issues tracker for EDSM
https://www.edsm.net/
37 stars 0 forks source link

Adding data via various JSON files? #503

Open ParzivalWolfram opened 1 year ago

ParzivalWolfram commented 1 year ago

I see that the market data for most locations are incredibly stale, however, the game does add the data to a "Market.json" file next to the journals. I'd suggest adding a parser for this, as this would likely solve the issue. See Market.json.tar.gz for an example.

There are various other files as well, which may be of some use as well, such as station shipyard and coordinate data. These may also be worth adding support for?

klightspeed commented 1 year ago

A lot of the markets are stale because nobody that submits that data (via e.g. EDMC, EDDiscovery, EDDI, etc) actually visits those stations and opens the Commodity Market / Outfitting / Shipyard to get that data.

ParzivalWolfram commented 1 year ago

A lot of the markets are stale because nobody that submits that data (via e.g. EDMC, EDDiscovery, EDDI, etc) actually visits those stations and opens the Commodity Market / Outfitting / Shipyard to get that data.

That is correct, however, in the attached file are multiple markets currently on the list of oldest stations. I'm unsure how to upload this data without external tools like EDDiscovery, and i'm currently having massive issues getting these tools to work under Linux. Having an option to just upload the JSON file on the frontend itself would help with this, as there would be far fewer hurdles to uploading the data and the interface would be platform-agnostic.

Tkael commented 1 year ago

@klightspeed A fair number of the tools you've listed actually use the Frontier API to pull in and submit market data equivalent to that found in Market.json. Market.json is written when the commander opens the Commodities screen but the same data can be accessed via the Frontier API without opening the Commodities screen (with either method the commander does need to have docked at the station).

In the case of EDDI (which is the tool that I maintain) we support sending data from either the Frontier API or Market.json and EDSM receives a copy of the data via a data sharing service that the community has developed called EDDN.

klightspeed commented 1 year ago

@klightspeed A fair number of the tools you've listed actually use the Frontier API to pull in and submit market data equivalent to that found in Market.json.

EDMC does also process those files (refer https://github.com/EDCD/EDMarketConnector/blob/34ccb20f1fe46dd8cdddf367d6ccf08671380aa3/plugins/eddn.py#L2468-L2497). EDDiscovery also processes those files (refer https://github.com/EDDiscovery/EliteDangerousCore/blob/73ba55fb03bbc4fefaf6949b45b4b96072c59e2b/EliteDangerous/JournalEvents/JournalMarket.cs#L58, https://github.com/EDDiscovery/EliteDangerousCore/blob/73ba55fb03bbc4fefaf6949b45b4b96072c59e2b/EliteDangerous/JournalEvents/JournalShipyard.cs#L51, and https://github.com/EDDiscovery/EliteDangerousCore/blob/73ba55fb03bbc4fefaf6949b45b4b96072c59e2b/EliteDangerous/JournalEvents/JournalOutfitting.cs#L51

I'm guessing what the OP is asking is if it would be possible to have a web service where they could upload such ancillary files and have the data in them sent to EDDN - though whoever is using them would need to remember to upload them every time they would have been rewritten - e.g. before undocking (for Market, Outfitting, and Shipyard), or after plotting a route (for NavRoute)

klightspeed commented 1 year ago

That is correct, however, in the attached file are multiple markets currently on the list of oldest stations.

The Market.json file only holds the data for your most recently opened commodities market (in this case Laflamme Hydroponics Habitat in Deorii, which last had anyone reporting to EDDN as having docked to it in May 2021)

I'm unsure how to upload this data without external tools like EDDiscovery, and i'm currently having massive issues getting these tools to work under Linux.

The tar.gz EDMC release file should work extracted under Linux and run using python - refer https://github.com/EDCD/EDMarketConnector/wiki/Running-from-source - you will need to set its journal path in the Configuration tab to something like ~/.steam/steam/steamapps/compatdata/359320/pfx/drive_c/users/steamuser/Saved Games/Frontier Developments/Elite Dangerous

Artoria2e5 commented 1 year ago

NavRoute is a pretty great thing to have, considering it lets you get coords and star type for almost anything within 20k ly.

The bad news is EDSM doesn't even process that from EDDN, and you'd expect json import to come after that.