Split config

[IzzySoft]

Thanks for the great script! I was using the one by n76 until now, but decided to switch over :)

3 separate commits here (feel free to cherry-pick):

• 45e1507: Moved user-config out of the main script (but left defaults there). This should make updates more easy.
• 05fb7a8: README.md was updated to reflect that. While on it, I also applied a little formatting.
• 19c85be: as OpenCellID only permits 1 download per day, it might come in handy to have an option to keep the CSV. To make this option really useful would require:
  • renaming the files instead of simply keeping them (so the new "failing download" doesn't truncate them)
  • capturing return codes from the download and, if it failed, use the old file(s) instead
  • removing the "old copy" as soon as there's a new good one

I didn't implement the latter (yet), but just wanted to be prepared in-case-of :)

[IzzySoft]

I didn't implement the latter (yet)

Now I did with 9f036d0: With KEEP_FILES=1, the script keeps exactly 1 backup of each file (.bak of CSV and DB). So on a repeated run, if d/l from OCI fails, the CSV from the previous run can/will be used (e.g. multiple runs on the same day, one won't succeed with the same API key). If processing fails completely (and user ends up with a broken/empty DB), there'd still be the DB from the last run (but only once ;).

KEEP_FILES=0 (default) will stick to the previous behavior, i.e. no backups kept.

That's it so far, hope you like it and merge it :)

[infinity0]

It would be better to name TMP_DIR as TMPDIR since that is more standard and would also affect sqlite3, see here.

It's a bit easier to set an envvar than to resize your partitions.

[sobrus]

Hi IzzySoft and Infinity. Sorry for such a delayed response. I will see your pull requests and suggestions in my spare time :) Just please remember that I want this script to be as simple and small as possible (without any additional features apart from generating database and even minimal error handling), so anyone can understand it quickly and modify it for his/her purposes (I also use highly modified version)

[IzzySoft]

Hi @sobrus Thanks for merging! And yes, KISS is great – one of the reasons I separated the config from the main script (ease of use, especially with updates).

And thanks again also for spending some of your spare time on this project – very much appreciated!

[sobrus]

@IzzySoft Hi there! Thanks for sharing your ideas and contribution :) Your backup idea is very nice too. I've updated the script again, mainly due to OpenCellID's recent unavailability. Now you can disable data source and use radiocells.org database too.

@infinity0 Hi, this is also a nice idea, especially since it also affects sqlite. I think I will change this variable on next update, then.

[IzzySoft]

@sobrus Just did a fetch+merge and saw you reorganized that backup – first thought it was gone, missing my backup block at the end LOL Good job!

And great idea to have those toggles! flg is currently just running with MOZ+RCO (the latter as temporary replacement for the "stalled" OCI). Indeed, RCO takes quite a while (an uncompressed SQLite file with several hundreds of Megabytes, wow).

Thanks for all your efforts!

PS: Just a thought on those backups. Shouldn't they better be compressed? After all, it's several Gigabyte we're talking about here.

[sobrus]

@IzzySoft @infinity0 I've added compression and changed TMP_DIR to TMPDIR. Compression is very simple (again to reduce script complexity). Please note that on modern hardware something like xz -1kT0 instead of gzip will probably give you better performance AND smaller file sizes.