Failed version check if not a git repository

Aerilius commented 1 month ago

After installing wahoomc from Pypi, I ran python -m wahoomc cli -co germany,france,switzerland,austria -con and > 20GB of *.osm.pbf files were downloaded. When running the same command again (within 30min), the already downloaded files are deleted. Then they are downloaded again. I did not use force download -fd.

From this comment I understand that besides the force-download option, downloaded maps are always redownloaded if out-of-date. I don't know how frequently they change on the server.

Expected Behavior

Cached files should be reused whenever possible. Redownload should be opt-in. Maybe issue a warning if existing files are from last year.

Current Behavior

After executing the command again, the ~/wahooMapsCreatorData/_download/maps/ folder was cleared.

Context

OS/Platform: Ubuntu

Aerilius commented 1 month ago

It was due to a failed version check:

INFO:Last run was with version None, deleting files of /home/andreas/wahooMapsCreatorData/_tiles directory due to breaking changes.
INFO:Last run was with version None, deleting files of /home/andreas/wahooMapsCreatorData/_download/maps directory due to breaking changes.
INFO:Last run was with version None, deleting files of /home/andreas/wahooMapsCreatorData/_tiles directory due to breaking changes.

I did pip install wahoomc, no editable install from the git repository.

When I clone from GitHub and do pip install editable ./wahooMapsCreator, the warning about version "None" does not appear and downloaded files are reused.

Aerilius commented 1 month ago

The reason was that the previous runs failed at a later stage (after download), but before writing "version_last_run" to ~/wahooMapsCreatorData/.config.json.

I don't know what is the best solution. It seems not like a bug, although it is unfortunate to make the cache invalidation depend not on success of the last download, but success of the whole pipeline.

I should have run Malta first to confirm correct installation, before adapting it to my usecase.
Maybe the wahoomc version or date of the cached files could be saved immediately after the download in the cache folder (like _download/metadata.json). Possibly the issue can be closed.

treee111 commented 1 month ago

Hi @Aerilius, thanks for your issue.

The reason was that the previous runs failed at a later stage (after download), but before writing "version_last_run" to ~/wahooMapsCreatorData/.config.json.

you are correct, this is the reason. I write the file only in the end where I save the config to not redownload, even reprocess etc. if certain things have not changed.

I don't know what is the best solution. It seems not like a bug, although it is unfortunate to make the cache invalidation depend not on success of the last download, but success of the whole pipeline.

correct. The logic for saving the config and checking when to redo certain things and when to not (and e.g. not _re_download correctly downloaded but not processed large map files)

* I should have run Malta first to confirm correct installation, before adapting it to my usecase.

yep. this would be my suggestion as I pasted multiple times in the documentation.

* Maybe the wahoomc version or date of the cached files could be saved immediately after the download in the cache folder (like `_download/metadata.json`).
  Possibly the issue can be closed.

Again - thanks for your comments. I'll have a look if find a possibility to somehow remember if the download was successful due to the download step can be a time consuming step depending on the internet speed.

FYI: The default day to redownload maps is 14 days, there is a argument to overwrite this: -md in the CLI version.

I think your second comment is not valid. If you don't agree, please open another issue.

treee111 / wahooMapsCreator