Automate Deep Dive Data Extrapolation

[MoritzHayden]

Problem

Currently, the information in deepdives.json needs to be manually extracted from the game using a mod like Sandbox Utilities and uploaded to this repo. Obviously, this introduces a human element and is extremely error-prone, not to mention inconvenient. This process should be automated, if possible.

In an ideal world, the seeds from the official GSG DRG endpoint could be fetched as soon as they are live every Thursday at 11am UTC (cron: 0 11 * * 4) and used to extrapolate the deep dive stage information. However, this logic is only contained in-game, and can change from patch to patch.

Ideas

• One could theoretically decompile the game, extract this logic, and rewrite it within this API. However, this logic could change and lead to rework.
• The game could be installed on a server and, once the seeds are live, a script runs to boot the game and scrape the deep dive information from the game files in a similar way to Sandbox Utilities. However, this isn't a very elegant solution as it would require a full installation of the game to be maintained on a remote server.

[MoritzHayden]

If you are reading this, please share your ideas! I'm open to anything and everything.

[ClusterConsultant]

After a brief survey of community tooling, I think the best bet would be a slight modification of your server model.

Instead of running it on a server, have it run on clients. Similar to how https://github.com/EDCD/EDMarketConnector is used to feed their community projects. If enough people grab it, it could even have the coverage to start updating regular missions in real time as well.

Otherwise, I think the only real option would be to get reliably good at extracting the logic. While not ideal, I wonder how often that logic changes.

The absolute best solution would be official support but I doubt that is anywhere near the table currently.

[rolfosian]

sent both of you guys a repo invite regarding this. im new to programming and pretty clueless about github so i dont want to mess around with forks or anything just yet.

[MoritzHayden]

Proposal

We may be able to create a batch job (GitHub Action) that runs on a weekly basis shortly after the refresh (for example, every Thursday at 11:15am UTC (cron: 15 11 4). This job could aggregate data from the official GSG endpoint (seeds, expiration) and a regularly updated Google Docs spreadsheet (dive metadata, stage data) into the JSON schema which this DRG API uses before pushing it up to the repo. This would be a completely automated solution assuming the Google Docs spreadsheet continues to be maintained (which, historically, it has been).

API Endpoints:

• Deep Dive Metadata (JSON): GET https://drg.ghostship.dk/events/deepdive
• Deep Dive Information (CSV): GET https://docs.google.com/feeds/download/spreadsheets/Export?key=1cvRHDfmBKK8Ggmoga6A3PrlyBaunZpbQhZuy987gGKk&exportFormat=csv&gid=672452639

[rolfosian]

I don't really know what endpoints are and I'm afraid I have no idea about using professional API modules or anything like that but I've automated it (providing my power doesn't go out at a bad time) with 59 10 * * 4 cron wakeonlan command from my home server to a tertiary junker machine, Task Scheduler, and a wait until 11am before launching the game and uploading the scraped data python script.

There may be a few hiccups in the coming weeks as the DurationLimit and ComplexityLimit configurations can interact differently with the MissionDNA than the standard missions resulting in indefinite complexity values in some cases and I simply don't have the data to cross reference them yet, but they will be fixed and the scraper updated to handle them as they come. If GSG changes the seed on the dot at 11AM every week (as they did this week) then there should be no issues otherwise.

Should be able to get accurate DD metadata from here, and if you want (what is currently) standard mission Metadata up until half way through September, it's here. There are links to current missions metadata and the upcoming missions metadata on the website also. Sorry if the keys/values arent industry standard or whatever, a lot of whitespaces there I probably shouldn't have used and integers that didnt need to be strings. Oh well, you can find all the code for this at the repo.

[rolfosian]

I might add that it is theoretically possible to automate running the deep dives themselves by executing functions and processing events using the https://github.com/UE4SS-RE/RE-UE4SS/ API, and scrape much more in depth data such as whether a BET-C is present, what dreadnought types are in the cocoons, etc, but it would require a shit ton more work to find the functions, and add logic to process them

[MoritzHayden]

Woah, this looks awesome @rolfosian!! Thank you so much.

[rolfosian]

You're welcome. If you're wondering why the codenames names are missing: https://github.com/UE4SS-RE/RE-UE4SS/issues/129

[MoritzHayden]

Ah good to know - thanks for the link