ED-NEWP / galos

0 stars 0 forks source link

Collect EDDN FSD Jump Events #2

Closed nixpulvis closed 3 weeks ago

nixpulvis commented 3 years ago

Collect FSDJump events from EDDN for systems and factions. Additionally, we can save System structures from the Location event as well. This can be done with galos sync eddn while the database is running. More configuration and utility will be added later.

Still need to save the following information:

I assume we don't care about the power play information yet. @M1K3SD1 can you confirm this?

nixpulvis commented 3 years ago

Currently, I'm planning on saving everything (until we have actual issues with storage size, or performance), and filtering queries as needed. However, the default configuration in RSB was our systems and ~20LY~ 40Ly cubes around each one.

After about 3 days of running galos sync eddn, new systems have slowed to a trickle and we're resting at 227,118 rows taking up 43 MB of space. More notably, ~3 ticks worth of influence tracking is taking up ~6 MB, with all the factions taking up ~8 MB. See ED-NEWP/galos_db#1 for the first planned space optimizations.

An example query for the 40 Ly cube is saved in https://github.com/ED-NEWP/galos_db/blob/master/sql/cube.sql, but might require some indexing to make faster if we want to use this in API responses as is. Otherwise, we can save the addresses/ids of systems/factions we care about up front somewhere for more detailed queries.

nixpulvis commented 3 years ago

I'm starting to think that galos sync should actually be it's own binary named galos-sync. This might seem like a really minor thing, but I'm hoping that the interface for galos itself can mirror (or be used for) the Discord Bot #7. I was also envisioning our web-server and workers to be named accordingly, i.e. galos-server and galos-worker, so this fits nicely.

EDIT: fixed by #10

nixpulvis commented 3 years ago

Possibly implementing individual FSDJump event tracking in https://github.com/ED-NEWP/galos/compare/jumps-routes. Stay tuned.