EDDB will soon cease operations

[bgol]

In case you didn't notice: https://forums.frontier.co.uk/threads/eddb-a-site-about-systems-stations-commodities-and-trade-routes-in-elite-dangerous.97059/page-37#post-10114765

[Tromador]

Does that make me the Tardis, as I'm hosting?

On Mon, 22 Apr 2024 at 07:56, Oliver Smith @.***> wrote:

It's like the four doctors in here :) 👋 @bgol https://github.com/bgol :)

— Reply to this email directly, view it on GitHub https://github.com/eyeonus/Trade-Dangerous/issues/110#issuecomment-2068624618, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJJGYLGWTO4Z33X35ZCYGYLY6SYCDAVCNFSM6AAAAAAWQEPTGGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRYGYZDINRRHA . You are receiving this because you were mentioned.Message ID: @.***>

-- Omnia dicta fortiora si dicta Latina!

[kfsone]

Is there a reason to prefer the human-readable/strptime datetimes? Having them as utc timestamps either int or float would make parsing and import much, much faster.

[eyeonus]

Historical inertia (that's how it was setup when I took over TD maintenance), and potentially backwards compatibility.

Regarding the former, I've no problems with changing it.

Regarding the latter, as long as it doesn't break anything, I've no problems changing it.

[kfsone]

"Your fault, you daft old fart" is perfectly fine by me :) ^- the you being me, not you, if you see what I mean, oh this is making it worse isn't it? 🤯

[kfsone]

@eyeonus seeing this at the moment with latest, this my fault?

Win|PS> ./trade.py import -P eddblink -O clean,solo
...
NOTE: Rebuilding cache file: this may take a few moments.
NOTE: Missing "C:\Users\oliver.smith\source\github.com\kfsone\Trade-Dangerous\tradedangerous\data\TradeDangerous.prices" file - no price data.
NOTE: Import completed.

that seems like something that shouldn't be missing at the end of an import?

[kfsone]

Oh, that's not the same as regenerating prices file. Duh. I used to figure that eventually the cost of generating the .prices file and stability of TD would mean we didn't have to keep generating the thing, is that all this is?

[eyeonus]

Nope, it's fine. It's a warning by tdb.reloadCache(), and is expected since it's a clean run.

[eyeonus]

Also since you did solo, it didn't download or import the listings, there's nothing to export to a prices file, and so you won't have a prices file after the command finishes, either.

[eyeonus]

Oh, that's not the same as regenerating prices file. Duh. I used to figure that eventually the cost of generating the .prices file and stability of TD would mean we didn't have to keep generating the thing, is that all this is?

No, it will regenerate prices IFF listings are imported, but not otherwise.

[kfsone]

While I'm refinding my feet, I've made a number of qol changes - at least, if you're using an IDE like PyCharm/VSCode.

I tend to configure tox so that my IDEs pick up the settings from it and I get in-ide guidance and refactoring options.

I've also introduced a little bit of flair to make watching it do its import thing a little less tedious, but I'm trying to stagger how I do it so that there's always an easy way to dump the new presentation. This is what happens when I've been watching Sebastian Lague videos lately https://www.youtube.com/watch?v=SO83KQuuZvg ... but it's probably also going to be nice for end-users too.

https://github.com/eyeonus/Trade-Dangerous/assets/323009/9ae278af-9304-4b5b-aad9-97ecf05f5802

These are currently in my kfsone/cleanup-pass branch.

[kfsone]

prices.py -> dumpPrices: "oliver waz 'ere" write large... took me a while to realize that somehow if you don't capture them, the cursor has the rows ready for you to iterate on. nasty. BAD SMITH.

[kfsone]

I'm doing some tuning of the tox config, it doesn't seem like we were actually running a "just flake8" run, or we weren't really using it? I got it enabled in my test branch. It should be fast (demo from pycharm)

https://github.com/eyeonus/Trade-Dangerous/assets/323009/40a8c23c-ade8-4c2b-9243-29a6da4f75ab

[eyeonus]

Oh, that's not the same as regenerating prices file. Duh. I used to figure that eventually the cost of generating the .prices file and stability of TD would mean we didn't have to keep generating the thing, is that all this is?

No, it will regenerate prices IFF listings are imported, but not otherwise.

Also, both the eddblink and spansh plugins use the existence of that file to determine if the database needs to be built. (i.e., if it doesn't exist, assume starting from scratch)

[kfsone]

I was checking a few of the 1MR posts/videos about how they tackle it in Python. We don't have 1 billion but it's not that dissimilar to what we do. Discovering that just using "rb" and doing byte-related operations was a bit of a stunner, but it's annoying trying to switch large tracts of code from utf8-to-bytes. However, it can provide a 4-8x speed up.

For instance, we count the number of lines in some of our files so we can do a progress bar, right? If the file is 86mb that takes ~250ms.

Just using "rb" gets that down to 50ms and a little use of fixed-size buffering gets it down to 41ms.

https://gist.github.com/kfsone/dcb0d7811570e40e73136a14c23bf128

[eyeonus]

Faster is good. I like faster.

[aadler]

I was checking a few of the 1MR posts/videos about how they tackle it in Python. We don't have 1 billion but it's not that dissimilar to what we do. Discovering that just using "rb" and doing byte-related operations was a bit of a stunner, but it's annoying trying to switch large tracts of code from utf8-to-bytes. However, it can provide a 4-8x speed up.

For instance, we count the number of lines in some of our files so we can do a progress bar, right? If the file is 86mb that takes ~250ms.

Just using "rb" gets that down to 50ms and a little use of fixed-size buffering gets it down to 41ms.

https://gist.github.com/kfsone/dcb0d7811570e40e73136a14c23bf128

See https://stackoverflow.com/a/27518377/2726543

[kfsone]

I was checking a few of the 1MR posts/videos about how they tackle it in Python. We don't have 1 billion but it's not that dissimilar to what we do. Discovering that just using "rb" and doing byte-related operations was a bit of a stunner, but it's annoying trying to switch large tracts of code from utf8-to-bytes. However, it can provide a 4-8x speed up. For instance, we count the number of lines in some of our files so we can do a progress bar, right? If the file is 86mb that takes ~250ms. Just using "rb" gets that down to 50ms and a little use of fixed-size buffering gets it down to 41ms. https://gist.github.com/kfsone/dcb0d7811570e40e73136a14c23bf128

See https://stackoverflow.com/a/27518377/2726543

ooorrrrr.... https://github.com/KingFisherSoftware/traderusty/ :)

I'm thinking I should have called it "tradedangersy" since rusty projects like to end with "rs" and python with "y" :)

[kfsone]

Don't read too much into that - an excuse to try a rust-python extension in anger (see https://github.com/kfsone/rumao3) and see how much pain setting up pypi and everything was (it wasn't). And I'm not sure eyeonus is like to want a second language adding to the problem :)

[kfsone]

@Tromador is listings.csv guaranteed to be in station,item order? I think I can optimize by doing a lock-step walk thru the database and listings (you create two generators, one with database entries the other with listing entries, and you keep advancing the one that is "behind"; if the listings one runs out, you stop; if the database one runs out you just don't need to compare)

[eyeonus]

@Tromador is listings.csv guaranteed to be in station,item order?

Yes, both the listings.csv and listings-livs.csv are guaranteed to be sorted by station_id, item_id

[eyeonus]

https://github.com/eyeonus/TradeDangerous-listener/blob/e5899ccae44f832aec088e4c113e82b95da29d3b/tradedangerous_listener.py#L934

https://github.com/eyeonus/TradeDangerous-listener/blob/e5899ccae44f832aec088e4c113e82b95da29d3b/tradedangerous_listener.py#L1024