Offer index backups for download?

powerbilayeredmap commented 5 years ago

Why doesn't pelias offer full elasticsearch index backups for download? Or do they? I'm tempted to post mine somewhere, once its done building.

orangejulius commented 5 years ago

Hi @powerbilayeredmap,

Good question. Several reasons, ultimately coming down to sustainability.

Pelias comes with lots of configuration options, so no one build can meet everyones needs. Not everyone needs the full planet either.
It's quite time consuming and costly (in human and server time) to do a full build
It's even more costly to do a full build regularly and ensure the quality of the build is high enough to be a net benefit (builds can fail for many reason and publishing broken builds would hurt the community more than help)
A full build comes in at around 350GB, so it's a rather large download for people, not to mention expensive to host. For example, if we host them on S3, then each download would cost the project about $30 at $0.09/GB.

powerbilayeredmap commented 5 years ago

I really think you should have a Full US, Full North America, and Full North America + Europe dataset available for purchase. I'd buy it. Snail mail it. These full imports are a nightmare that don't seem to behave the same from run to run. Awesome product though. Good work.

orangejulius commented 5 years ago

Thanks, it's something we're looking at providing through geocode.earth in the future. It's definitely a challenge to run a full build, but to a certain extent that's unavoidable when working with so much data from so many different sources.

pelias / pelias

Offer index backups for download? #770