pelias / interpolation

global street address interpolation service (beta)
https://interpolation.demo.geocode.earth
MIT License
58 stars 29 forks source link

delete npm unzipper, replace with GDAL virtual filesystem #239

Closed missinglink closed 4 years ago

missinglink commented 4 years ago

I'm getting tired of seeing npm unzip issues, so 👋 bye bye module 👋

Many years ago when I wrote this code I didn't know about GDAL virtual filesystems and assumed that zip files are easy, and streamable, which it turns out they aren't 🤷‍♂

This PR greatly simplifies the code by downloading the .zip files and then... wait for it... not decompressing them 😱 The only process reading these files is ogr2ogr and it'll just as happily read ZIP files as it would the SHP files.

Good news is we can delete the npm module and also the shapefiles directory in the process.

missinglink commented 4 years ago

Works for me too, pity its in bash and isn't easy to test...

I ended up just using a Dockerfile to test it ;)

# base image
FROM pelias/interpolation:latest

ENV TIGERPATH=/tmp/tiger
RUN mkdir -p /tmp/tiger

RUN echo '{\n\
  "logger": { "level": "silly" },\n\
  "imports": {\n\
    "interpolation": {\n\
      "download": {\n\
        "tiger": {\n\
          "datapath": "/tmp/tiger",\n\
          "states": [\n\
            { "state_code": 36, "county_code": 85 },\n\
            { "state_code": 36, "county_code": 47 },\n\
            { "state_code": 36, "county_code": 5 }\n\
          ]\n\
        }\n\
      }\n\
    }\n\
  }\n\
}' > /code/pelias.json
RUN cat /code/pelias.json

COPY . /code/pelias/interpolation/

RUN node script/js/update_tiger.js
RUN ls -lah /tmp/tiger/downloads

RUN mkdir -p /tmp/polylines
RUN curl -s https://s3.amazonaws.com/pelias-data.nextzen.org/polylines/new_york.gz | gunzip > /tmp/polylines/new_york.0sv
ENV POLYLINE_FILE=/tmp/polylines/new_york.0sv

RUN mkdir /tmp/build
ENV BUILDDIR=/tmp/build

# debug
# ENV PROC_STDOUT=/dev/stdout
# ENV PROC_STDERR=/dev/stderr

# create street.db
RUN bash script/import.sh

# cconflate tiger
RUN bash script/conflate_tiger.sh

:shipit: