Planet pelias json config openaddresses

[PhanSon95]

Hi team I want to import pelias planet. I read file pelias.json in planet project and see some confuse about openaddresses config:

    "openaddresses": {
      "datapath": "/data/openaddresses",
      "files": [
      ]
    },

I see don't have any files in this config . It doesn't auto download openaddresses when I run command

pelias download all

I also check pelias command, it doesn't command download openaddresses

Maybe I need download manual data and unzip it batch.openaddresses.io

But I want to know, do I need add specific files in here?

    "openaddresses": {
      "datapath": "/data/openaddresses",
      "files": [
      ]
    },

Or Just run pelias import all ? Because of import too spend time and I not sure import all openaddress data success or not So I create a question

Describe the bug Confuse file config in pelias.json

Steps to Reproduce Go to pelias.json in project planet Check config openaddresses

Expected behavior Add readme.md file more clearly about pelias config

Environment (please complete the following information): OS: Ubuntu 22.04 LTS Docker versions: Docker version 23.0.0, build e92dd87

Pastebin/Screenshots Doesn't have openaddress when run command pelias download all and pelias import all

[PhanSon95]

I've download data planet from batch.openaddresses.io and unzip it Do I need change any pelias.json before run command pelias prepare all ?

[PhanSon95]

I've check code openaddress in github, you seem use files to load data to stream and import to elastic.

I've check folder /data/openaddress There are a lot of folder and data I can't write data in files here manual. This too much

   "openaddresses": {
      "datapath": "/data/openaddresses",
      "files": [
      ]
    }, 

[missinglink]

I'm not able to reproduce this error, when I enter the planet project:

/code/p/docker/p/planet master = *1 !3 ❯ pelias download oa

...
info: [openaddresses-download] Attempting to download all data

The download takes some time to complete so you'll need to be patient, I believe the OpenAddresses team rate-limit downloads.

While the download is ongoing you'll see the container is still active:

docker ps
CONTAINER ID   IMAGE                         COMMAND                  CREATED         STATUS                      PORTS                        NAMES
66db4ac55e48   pelias/openaddresses:master   "./bin/download"         2 minutes ago   Up 2 minutes                                             pelias-openaddresses-run-c6622a6d4959

I would not recommend downloading data from batch.openaddresses.io since it's in geojson format, not in csv format as expected by Pelias.

[PhanSon95]

This is docker container running when I run command pelias download all

[PhanSon95]

I got error the file is too large.

I already download file planet-latest-valhalla.polylines.0sv.gz and unzip here

But I got error when run pelias prepare all

[PhanSon95]

I try to fix import polyline with large file .pbf

devops@pelias:~/docker/projects/planet$ pelias prepare all                                                                                                                                                 
DATA_DIR is already set to './data' - this may cause the DATA_DIR specified in the .env to be ignored                                                                                                      
Creating extract at /data/placeholder/wof.extract                                                                                                                                                          
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!                                                                                                                          
/data/openstreetmap/planet-latest.osm.pbf is very large.                                                                                                                                                   
We strongly recommend using Valhalla to produce extracts for large PBF extracts.                                                                                                                           
You can also download pre-processed polyline extracts from Geocode Earth.                                                                                                                                  
see: https://github.com/pelias/polylines#download-data                                                                                                                                                     
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!                                                                                                                          
Generating polylines from /data/openstreetmap/planet-latest.osm.pbf failed! The file is too large.   

I download planet polyline file and put here But I got error when import polylines plantet. It not using my download data This is my pelias json config :

Error when import polylines :

[

[PhanSon95]

hi @missinglink Please let me how to fix pelias polylines. It not using my downloaded data in step pelias prepare all. File extract.0sv is generate during run pelias prepare all

[PhanSon95]

I got this error too

[PhanSon95]

hi @missinglink I've imported planet . But I see missing some openaddress data Example in montreal city. I check and see I'm missing almost addresses in street Rue John-mc. I read file ca/qc/montreal-addresses-city.geojson and see it contain many address for Rue John-Mc street (house number from 2094 - 2195)

devops@pelias:~/docker/projects/planet/data/openaddresses/ca/qc$ cat montreal-addresses-city.geojson | grep John-Mc
{"type":"Feature","properties":{"hash":"b84fbb8f64757b3b","number":"2094","street":"rue  John-McCurdy","unit":"","city":"","district":"","region":"","postcode":"","id":""},"geometry":{"type":"Point","coordinates":[-73.7024218,45.5176417]}}
{"type":"Feature","properties":{"hash":"493816958f919c2e","number":"2096","street":"rue  John-McCurdy","unit":"","city":"","district":"","region":"","postcode":"","id":""},"geometry":{"type":"Point","coordinates":[-73.7023696,45.5176071]}}
{"type":"Feature","properties":{"hash":"2f5989360df3e123","number":"2098","street":"rue  John-McCurdy","unit":"","city":"","district":"","region":"","postcode":"","id":""},"geometry":{"type":"Point","coordinates":[-73.7023158,45.5175695]}}
{"type":"Feature","properties":{"hash":"d383fb2d3d066b4a","number":"2100","street":"rue  John-McCurdy","unit":"","city":"","district":"","region":"","postcode":"","id":""},"geometry":{"type":"Point","coordinates":[-73.7022634,45.5175307]}}
{"type":"Feature","properties":{"hash":"1b77dd16bb841c6e","number":"2102","street":"rue  John-McCurdy","unit":"","city":"","district":"","region":"","postcode":"","id":""},"geometry":{"type":"Point","coordinates":[-73.702197,45.5175085]}}
{"type":"Feature","properties":{"hash":"f5b9d4e9bffc911b","number":"2104","street":"rue  John-McCurdy","unit":"","city":"","district":"","region":"","postcode":"","id":""},"geometry":{"type":"Point","coordinates":[-73.7021447,45.5174771]}}

...
{"type":"Feature","properties":{"hash":"a3218c0c7a2be83f","number":"2194","street":"rue  John-McCurdy","unit":"","city":"","district":"","region":"","postcode":"","id":""},"geometry":{"type":"Point","coordinates":[-73.7041732,45.5162649]}}
{"type":"Feature","properties":{"hash":"7f3cd6ed0bdfa417","number":"2195","street":"rue  John-McCurdy","unit":"","city":"","district":"","region":"","postcode":"","id":""},"geometry":{"type":"Point","coordinates":[-73.7039813,45.5164989]}}
devops@pelias:~/docker/projects/planet/data/openaddresses/ca/qc$ 

I don't know why only 2094 rue john addresses can be found. I don't see any panic error when run pelias prepare all and pelias import all