search

eyeonus / Trade-Dangerous

Mozilla Public License 2.0
96 stars 31 forks source link

feat: add no-download option to eddblink plugin #140

Closed kfsone closed 2 months ago

eyeonus commented 2 months ago

That's what the force option is for

kfsone commented 2 months ago

force seems to force it to download the files?

What I wanted was:

$ trade import -P eddblink -O force  # ensure it gets latest files, does the import
$ time trade import -P eddblink -O force,no-download  # re-use the files even if stale
$ make-changes
$ time trade import -P eddblink -O force,no-download  # re-use the existing data for apples-to-apples
eyeonus commented 2 months ago

No, force doesn't make it download the files., it makes it run even if the downloadFile returns false (which it will if the source is not more recent than the local file).

example:

        if self.getOption("listings"):
            if self.downloadFile(self.listingsPath) or self.getOption("force"):
                self.importListings(self.listingsPath)
            if self.downloadFile(self.liveListingsPath) or self.getOption("force"):
                self.importListings(self.liveListingsPath)
kfsone commented 2 months ago

right - but I was aiming for is this: (god I can't type on a mac)

if self.getOption("listings"):
  new_data = self.getOption("dont-download-stuff") or self.downloadFile(self.listingsPath)
  if new_data or self.getOption("force"):
    self.importListings(self.listingsPath)

so that I can download the files once and then repeatedly run import on the same data without it going away and downloading the latest update.

Without that, this happens:

$ benchmark trade.py import -P eddblink
NOTE: Checking...
NOTE: Downloading listings.csv
NOTE: Importing listings.csv 2024-04-26 11:30:00
$ make changes
$ mv data/Tradedangerous.old data/Tradedangerous.sql  # <-- restore db state
$ benchmark trade.py import -P eddblink
NOTE: Checking...
NOTE: Downloading listings.csv 2024-04-26 17:59:00  # <-- different data!
NOTE: Importing listings.csv 2024-04-26 17:59:00  # <-- different data!

What I want is:

$ benchmark trade.py import -P eddblink
NOTE: Checking...
NOTE: Downloading listings.csv
NOTE: Importing listings.csv 2024-04-26 11:30:00
$ make changes
$ mv data/Tradedangerous.old data/Tradedangerous.sql
$ benchmark trade.py import -P eddblink -O no-download
NOTE: Skipping checking...
NOTE: Skipping download
NOTE: Importing listings.csv 2024-04-26 11:30:00  # <-- same data!
eyeonus commented 2 months ago

I can understand why you would want that for testing purposes, but I can't see any reason to include that functionality in release.

eyeonus commented 2 months ago

If you set your copy of the listings file to be 'modiifed' at some distant point in the future (such as by using os.utime({file}, ({accesstime}, {modifiedtime})), you will get your desired functionality.