search

jeancochrane / mellow-bike-map

🚲 Web app and routing API for finding safe, chill bike routes. Like Google Maps if it took you through cozy side streets.
https://mellowbikemap.com
MIT License
13 stars 5 forks source link

Data import bug #17

Closed beamalsky closed 4 years ago

beamalsky commented 4 years ago

I cloned this repo, ran docker-compose build, and then hit this error on the next step in the README: 

Beas-MBP:mellow-bike-map beamalsky$ docker-compose -f docker-compose.yml -f db/docker-compose.yml run --rm make
Creating network "mellow-bike-map_default" with the default driver
Creating volume "mellow-bike-map_mellow-bike-map-db-data" with default driver
Creating volume "mellow-bike-map_mellow-bike-map-node-modules" with default driver
Creating mellow-bike-map-postgres ... done
Building make
Step 1/4 : FROM mdillon/postgis:11
 ---> b2a8fefe8a63
Step 2/4 : RUN apt-get update && apt-get install -y --no-install-recommends     build-essential gcc make cmake gdal-bin postgresql-server-dev-11     expat libexpat1-dev libboost-dev libboost-graph-dev libboost-program-options-dev libpqxx-dev     wget ca-certificates
 ---> Using cache
 ---> ce2f9c425429
Step 3/4 : RUN wget -O pgrouting-3.1.0.tar.gz https://github.com/pgRouting/pgrouting/archive/v3.1.0.tar.gz &&     tar xvfz pgrouting-3.1.0.tar.gz &&     cd pgrouting-3.1.0 &&     mkdir build &&     cd build &&     cmake .. &&     make &&     make install &&     cd / && rm -Rf pgrouting-3.1.0*
 ---> Using cache
 ---> 8700ee10c6bc
Step 4/4 : RUN wget -O osm2pgrouting-2.3.6.tar.gz https://github.com/pgRouting/osm2pgrouting/archive/v2.3.6.tar.gz &&     tar xvfz osm2pgrouting-2.3.6.tar.gz &&     cd osm2pgrouting-2.3.6 &&     cmake -H. -Bbuild &&     cd build &&     make &&     make install &&     cd / && rm -Rf osm2pgrouting-2.3.6*
 ---> Using cache
 ---> e708efbda884

Successfully built e708efbda884
Successfully tagged mellow-bike-map-make:latest
WARNING: Image for service make was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
python manage.py loaddata app/mbm/fixtures/mellowroute.json && touch db/import/mellowroute.table
python: can't open file 'manage.py': [Errno 2] No such file or directory
Makefile:5: recipe for target 'db/import/mellowroute.table' failed
make: *** [db/import/mellowroute.table] Error 2
Beas-MBP:mellow-bike-map beamalsky$ docker-compose -f docker-compose.yml -f db/docker-compose.yml run --rm make
Starting mellow-bike-map-postgres ... done
python manage.py loaddata app/mbm/fixtures/mellowroute.json && touch db/import/mellowroute.table
python: can't open file 'manage.py': [Errno 2] No such file or directory
Makefile:5: recipe for target 'db/import/mellowroute.table' failed
make: *** [db/import/mellowroute.table] Error 2

Do you know what's up here @jeancochrane? Looks like it's having trouble with the Django management command.

jeancochrane commented 4 years ago

There's a couple problems with the db/import/mellowroute.table target:

  1. It requires the dependencies in app/Dockerfile (notably Django), but the Makefile uses db/Dockerfile
  2. We need to run management commands from the app/ directory, not from the root directory

I think fixing both these problems will require refactoring the data import pipeline to use one shared Dockerfile that has both dependencies and adjusting the target to run management commands from the right directory. I can do this but probably not till later tonight.

In the meantime, one work around would be to adjust the command that's listed in the README to skip the db/import/mellowroute.table target:

docker-compose -f docker-compose.yml -f db/docker-compose.yml run --rm make db/import/chicago.table

Then import the MellowRoute fixture manually using the app container, instead of using the broken target:

docker-compose run --rm app ./manage.py loaddata mellowroute
beamalsky commented 4 years ago

I'll do this workaround, thank you! If you didn't want to do the refactor, FWIW I think it'd be fine to just have an extra step in the README.

beamalsky commented 4 years ago

Unfortunately I'm now getting this error: 

Beas-MBP:mellow-bike-map beamalsky$ docker-compose -f docker-compose.yml -f db/docker-compose.yml run --rm make db/import/chicago.table
Starting mellow-bike-map-postgres ... done
osm2pgrouting -f db/raw/chicago.osm -c /usr/local/share/osm2pgrouting/mapconfig_for_bicycles.xml --prefix chicago_ --addnodes --tags --clean \
              -d mbm -U postgres -h postgres -W postgres &&
/bin/sh: 2: Syntax error: end of file unexpected
Makefile:8: recipe for target 'db/import/chicago.table' failed
make: *** [db/import/chicago.table] Error 2
jeancochrane commented 4 years ago

@beamalsky Good catch! I just pushed up 1d4539c91000be49ad4ea8e3b388a697e9ef0642 which should fix that syntax error.

beamalsky commented 4 years ago

I'm now seeing this: 

Beas-MBP:mellow-bike-map beamalskydocker-compose -f docker-compose.yml -f db/docker-compose.yml run --rm make db/import/chicago.table
Starting mellow-bike-map-postgres ... done
wget --no-use-server-timestamps -O db/raw/chicago.osm https://overpass-api.de/api/map?bbox=-87.7488,41.7170,-87.5157,42.0003
--2020-09-07 02:51:43--  https://overpass-api.de/api/map?bbox=-87.7488,41.7170,-87.5157,42.0003
Resolving overpass-api.de (overpass-api.de)... 178.63.11.215, 178.63.48.217, 2a01:4f8:120:6464::2, ...
Connecting to overpass-api.de (overpass-api.de)|178.63.11.215|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/osm3s+xml]
Saving to: ‘db/raw/chicago.osm’

db/raw/chicago.osm      [             <=>    ] 927.82M  5.03MB/s    in 3m 24s

2020-09-07 02:55:14 (4.56 MB/s) - ‘db/raw/chicago.osm’ saved [972891961]

osm2pgrouting -f db/raw/chicago.osm -c /usr/local/share/osm2pgrouting/mapconfig_for_bicycles.xml --prefix chicago_ --addnodes --tags --clean \
              -d mbm -U postgres -h postgres -W postgres && \
PGPASSWORD=postgres psql -U postgres -h postgres -d mbm -c " \
    UPDATE chicago_ways SET one_way = 2, oneway = 'NO', reverse_cost = cost \
    FROM osm_ways \
    WHERE osm_ways.osm_id = chicago_ways.osm_id \
    AND osm_ways.tags @> 'oneway:bicycle => no'" && \
touch db/import/chicago.table
Execution starts at: Mon Sep  7 02:55:14 2020

***************************************************
           COMMAND LINE CONFIGURATION             *
***************************************************
Filename = db/raw/chicago.osm
Configuration file = /usr/local/share/osm2pgrouting/mapconfig_for_bicycles.xml
host = postgres
port = 5432
dbname = mbm
username = postgres
schema=
prefix = chicago_
suffix =
Drop tables
Don't create indexes
Add OSM nodes
***************************************************
Testing database connection: mbm
database connection successful: mbm
Connecting to the database
connection success

Dropping tables...
TABLE: chicago_ways dropped ... OK.
TABLE: chicago_ways_vertices_pgr dropped ... OK.
TABLE: chicago_pointsofinterest dropped ... OK.
TABLE: configuration dropped ... OK.
TABLE: osm_nodes dropped ... OK.
TABLE: osm_ways dropped ... OK.
TABLE: osm_relations dropped ... OK.

Creating tables...
TABLE: chicago_ways_vertices_pgr created ... OK.
TABLE: chicago_ways created ... OK.
TABLE: chicago_pointsofinterest created ... OK.
TABLE: configuration created ... OK.
TABLE: osm_nodes created ... OK.
TABLE: osm_ways created ... OK.
TABLE: osm_relations created ... OK.
Opening configuration file: /usr/local/share/osm2pgrouting/mapconfig_for_bicycles.xml
    Parsing configuration

Exporting configuration ...
  - Done
Counting lines ...
  - Done
Opening data file: db/raw/chicago.osm   total lines: 13185664
    Parsing data

Current osm_nodes:Killed760000
Makefile:8: recipe for target 'db/import/chicago.table' failed
make: *** [db/import/chicago.table] Error 137
beamalsky commented 4 years ago

Oh looks like this is a memory error—let me try upping my docker system resources!

jeancochrane commented 4 years ago

What’s your memory limit in your Docker preferences? That happened to me and I’m pretty sure it’s an OutOfMemoryError, I was able to I get past it by bumping my memory limit.

Le dim. 6 sept. 2020 à 10:09 PM, Bea Malsky notifications@github.com a écrit :

I'm now seeing this:

Beas-MBP:mellow-bike-map beamalskydocker-compose -f docker-compose.yml -f db/docker-compose.yml run --rm make db/import/chicago.table

Starting mellow-bike-map-postgres ... done

wget --no-use-server-timestamps -O db/raw/chicago.osm https://overpass-api.de/api/map?bbox=-87.7488,41.7170,-87.5157,42.0003

--2020-09-07 02:51:43-- https://overpass-api.de/api/map?bbox=-87.7488,41.7170,-87.5157,42.0003

Resolving overpass-api.de (overpass-api.de)... 178.63.11.215, 178.63.48.217, 2a01:4f8:120:6464::2, ...

Connecting to overpass-api.de (overpass-api.de)|178.63.11.215|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: unspecified [application/osm3s+xml]

Saving to: ‘db/raw/chicago.osm’

db/raw/chicago.osm [ <=> ] 927.82M 5.03MB/s in 3m 24s

2020-09-07 02:55:14 (4.56 MB/s) - ‘db/raw/chicago.osm’ saved [972891961]

osm2pgrouting -f db/raw/chicago.osm -c /usr/local/share/osm2pgrouting/mapconfig_forbicycles.xml --prefix chicago --addnodes --tags --clean \

          -d mbm -U postgres -h postgres -W postgres && \

PGPASSWORD=postgres psql -U postgres -h postgres -d mbm -c " \

UPDATE chicago_ways SET one_way = 2, oneway = 'NO', reverse_cost = cost \

FROM osm_ways \

WHERE osm_ways.osm_id = chicago_ways.osm_id \

AND osm_ways.tags @> 'oneway:bicycle => no'" && \

touch db/import/chicago.table

Execution starts at: Mon Sep 7 02:55:14 2020

       COMMAND LINE CONFIGURATION             *

Filename = db/raw/chicago.osm

Configuration file = /usr/local/share/osm2pgrouting/mapconfig_for_bicycles.xml

host = postgres

port = 5432

dbname = mbm

username = postgres

schema=

prefix = chicago_

suffix =

Drop tables

Don't create indexes

Add OSM nodes

Testing database connection: mbm

database connection successful: mbm

Connecting to the database

connection success

Dropping tables...

TABLE: chicago_ways dropped ... OK.

TABLE: chicago_ways_vertices_pgr dropped ... OK.

TABLE: chicago_pointsofinterest dropped ... OK.

TABLE: configuration dropped ... OK.

TABLE: osm_nodes dropped ... OK.

TABLE: osm_ways dropped ... OK.

TABLE: osm_relations dropped ... OK.

Creating tables...

TABLE: chicago_ways_vertices_pgr created ... OK.

TABLE: chicago_ways created ... OK.

TABLE: chicago_pointsofinterest created ... OK.

TABLE: configuration created ... OK.

TABLE: osm_nodes created ... OK.

TABLE: osm_ways created ... OK.

TABLE: osm_relations created ... OK.

Opening configuration file: /usr/local/share/osm2pgrouting/mapconfig_for_bicycles.xml

Parsing configuration

Exporting configuration ...

  • Done

Counting lines ...

  • Done

Opening data file: db/raw/chicago.osm total lines: 13185664

Parsing data

Current osm_nodes:Killed760000

Makefile:8: recipe for target 'db/import/chicago.table' failed

make: *** [db/import/chicago.table] Error 137

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jeancochrane/mellow-bike-map/issues/17#issuecomment-687999623, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADMDUGRBKN5E5SRQ24PDXM3SERFIFANCNFSM4Q46ZZPQ .

beamalsky commented 4 years ago

I upped my memory limit from 2 to 6 GiB, so far so good