XmlGenerator: Auto-Download SQLite SDE

[doomlord289]

In peterhaneve/evemon#294, @wvdvegt provided code for the XmlGenerator utility that downloads the SQLite SDE Conversion from fuzzwork.co.uk. I would be nice to have this functionality in the main branch.

[mgoeppner]

The long term plan is to incorporate this into the CI process, so the new datafiles are generated and uploaded automatically when there is a new SDE available.

It looks like the new SDE was uploaded last week as well, so I will look into incorporating the CI changes as part of the work to update the datafiles.

[doomlord289]

I saw a tool linked on the Eve 3rd party discord a bit ago and figured it might be something you'd want to incorporate when you eventually get around to this. EveShipFit developed a tool to download the static data from the client files rather than the actual SDE because the SDE is updated less often than the actual client. If you could set up a pipeline with that, then pipe it into the XmlGenerator, that'd keep things up to date.

[wvdvegt]

Hi,

I had a look into the code but I have some reservations besides some hurdles to take. But the link you mailed is vertainly an interesting piece of code to read.

1) It seems a rather undocumented method (and CCP can/might change it without any notifications. 2) It relies on compiled python scripts to decode the fsdbinaries downloaded. This is hard to integrate with the C# code and probably needs some kind version of Python to be installed in order to work at all. 3) I'm not sure if it hits the EVE EULA by using game files to get data from. It's very close to reverse engineering.

In my latest private (EveMon replacement) tool I can download (and update) most of the info using EveEsi (https://esi.evetech.net/ui/) which is documented, supported and uptodate. It is the official method for 3rd party tools that want to integrate Eve Data. Missing on EveEsi are only some minor things as the stations of research agents.

[mgoeppner]

Hi,

EVEMon will continue to use the existing SDE conversion process in the near to mid term, as my focus is on making things work correctly with dotnet8 and evemon 5.x.x, which opens doors for some other projects that I want to experiement with.

I still have an eventual goal of getting this to work in CI, but there are a few blocker issues regarding missing data that I've had to massage in over the years -- most of this is ensuring the SDE complement files from hoboleaks get integrated into the dump that the XML files are generated from.

In the long term, I would very much like to move away from the xml zip blobs. This could be one of the following options

• Leverage the SQLite SDE conversion directly, and distribute it, instead of reconverting it to XML data.
• Leverage ESI's data directly. This comes with a number of other pitfalls and is missing certain data types.
• Leverage a third party api conversion like EVERef's reference data api or jita.space's open api.

EVEMon's XML data is a bit of legacy leftover from the CREST days when everything was in XML -- "it isn't broken yet and saves a relatively large refactor" is the reason this hasn't been prioritized higher.