Open missinglink opened 1 year ago
This is ready for testing, there are some features I would have liked to add but will have to wait for a subsequent PR:
.gz
and do the decompression on import
(saves disk space)us/{ca,ny}/*
to the config file and have the API figure it outI still need to clarify a couple things with the OA team:
openaddr-collected-global.zip
& openaddr-collected-global-sa.zip
)?LGTM
:+1: for decompression on import
and source pattern matching for a next PR.
Maybe you can add a logger.error
for all missing files even if errorsFatal
is false. This allows you to know all the errors at once.
Yeah good idea about the logging, I'll go through and make sure everything is logged, even if it's at the debug log level, I also want to log the output of these child process shell commands if anything goes wrong.
I have a chat going with the OA team at the moment to clarify some of the final questions:
collection-global.zip
which contains everything, addresses, parcels etc. I'd prefer to have a new collection with only the addresses, and in .tar.gz
rather than .zip
but we will see what's possible.Hi @Joxit, @missinglink,
Any update on this? Is that planned to make this available in the pelias/docker repo?
Thanks!
this
draftPR updates the OA downloader to use thebatch.openaddresses.io
endpoint.a few things worth mentioning:
gzip
compressed rather thanzip
I haven't done any testing of the import process, only the download process so far