search

pelias-deprecated / dockerfiles

Containerized Local Installation Package for the Pelias geocoder
https://pelias.io
MIT License
32 stars 25 forks source link

Libpostal errors #9

Closed 3vivekb closed 6 years ago

3vivekb commented 7 years ago

I've been trying a deployment on Ubuntu 16 (using the default config) and the build process around libpostal shows a lot of errors. The whole container does build successfully but when you try to run it error messages surround how libpostal is missing.

3vivekb commented 7 years ago

Errors show up in the build process.

Docker version 17.06.0-ce, build 02c1d87 docker-compose version 1.14.0, build c7bdf9e

Distributor ID: Ubuntu Description: Ubuntu 16.04.2 LTS Release: 16.04 Codename: xenial

First set of no:

checking for sysroot... no
checking for a working dd... /bin/dd
checking how to truncate binary pipes... /bin/dd bs=4096 count=1
checking for mt... no
checking if : is a manifest tool... no

checking for memmove... yes
checking for memset... yes
checking for regcomp... yes
checking for setlocale... yes
checking for sqrt... yes
checking for strdup... yes
checking for strndup... yes
checking for shuf... yes
checking for gshuf... no
checking cblas.h usability... no
checking cblas.h presence... no
checking for cblas.h... no
configure: extra cflags for scanner.c:
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating libpostal.pc
config.status: creating src/Makefile
config.status: creating test/Makefile
config.status: creating config.h
config.status: executing depfiles commands
config.status: executing libtool commands

First set of errors:


In file included from sparse_matrix.h:46:0,
                 from averaged_perceptron.h:27,
                 from address_parser.h:52,
                 from libpostal.c:10:
matrix.h:19:2: warning: #warning "No CBLAS" [-Wcpp]
 #warning "No CBLAS"
  ^
libpostal.c: In function ‘add_string_alternatives’:
libpostal.c:326:46: warning: unused variable ‘start_index’ [-Wunused-variable]
                                     uint32_t start_index = cstring_array_start_token(tree->strings);
                                              ^
libpostal.c:344:38: warning: unused variable ‘start_index’ [-Wunused-variable]
                             uint32_t start_index = cstring_array_start_token(tree->strings);
                                      ^
libpostal.c:367:26: warning: unused variable ‘start_index’ [-Wunused-variable]
                 uint32_t start_index = cstring_array_start_token(tree->strings);
                          ^
libpostal.c: In function ‘add_affix_expansions’:
libpostal.c:548:72: warning: unused parameter ‘lang’ [-Wunused-parameter]
 static bool add_affix_expansions(string_tree_t *tree, char *str, char *lang, token_t token, phrase_t prefix, phrase_t suffix, libpostal_normalize_options_t options) {
                                                                        ^
libpostal.c: In function ‘expand_alternative’:
libpostal.c:859:11: warning: unused variable ‘lang’ [-Wunused-variable]
     char *lang;
           ^
libpostal.c: In function ‘expand_alternative.isra.13’:
libpostal.c:602:45: warning: ‘*((void *)&suffix_expansion+24)’ may be used uninitialized in this function [-Wmaybe-uninitialized]
                 add_space = suffix_expansion.separable;
                                             ^
libpostal.c:562:25: note: ‘*((void *)&suffix_expansion+24)’ was declared here
     address_expansion_t suffix_expansion;
                         ^
libtool: compile:  gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include -Wall -Wextra -Wno-unused-function -Wformat -Werror=format-security -Winit-self -Wno-sign-compare -DLIBPOSTAL_DATA_DIR=\"/usr/share/libpostal/libpostal\" -g -mfpmath=sse -msse2 -DUSE_SSE -g -O2 -O2 -MT libpostal_la-libpostal.lo -MD -MP -MF .deps/libpostal_la-libpostal.Tpo -c libpostal.c -o libpostal_la-libpostal.o >/dev/null 2>&1
mv -f .deps/libpostal_la-libpostal.Tpo .deps/libpostal_la-libpostal.Plo
/bin/bash ../libtool  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include    -Wall -Wextra -Wno-unused-function -Wformat -Werror=format-security -Winit-self -Wno-sign-compare -DLIBPOSTAL_DATA_DIR='"/usr/share/libpostal/libpostal"' -g -mfpmath=sse -msse2 -DUSE_SSE -g -O2 -O2  -MT libpostal_la-address_dictionary.lo -MD -MP -MF .deps/libpostal_la-address_dictionary.Tpo -c -o libpostal_la-address_dictionary.lo `test -f 'address_dictionary.c' || echo './'`address_dictionary.c
libtool: compile:  gcc -DHAVE```
dianashk commented 7 years ago

These errors are very helpful. Is there a log of errors coming from the API container which uses libpostal?

3vivekb commented 7 years ago

Logs from docker-compose up: output.txt

missinglink commented 7 years ago

I usually see a bunch of colorful warnings when compiling libpostal locally on my desktop, it's nothing to worry about, for me libpostal still works fine.

$ cat /etc/issue
Ubuntu 16.10 \n \l

looking at your attached log file there seem to be other errors which need to be resolved such as:

Error: Cannot find module '/data/placeholder/graph.json'

@dianashk could you explain the process for downloading the placeholder data, I suspect that this is the real issue.

dianashk commented 7 years ago

Looking through the log file, it seems there are 2 issues:

  1. WOF data doesn't seem to exist in the /data/whosonfirst mounted directory
  2. Libpostal is not properly installed so API and interpolation can't start up

I'm not sure what could cause such issues other than intermittent connectivity issues or lack of space on the mounted directory. When you started up the build script, what was DATA_DIR environment variable set to? If you list all the things in that directory after the build was finished what do you see? The data directory should look like this:

$ ls -al /pelias/data/
drwxr-xr-x  3 systemd-resolve systemd-timesync 4096 Jul 28 19:53 elasticsearch
drwxr-xr-x  2 root            root             4096 Jul 28 20:23 interpolation
drwxr-xr-x  3 root            root             4096 Jul 28 19:54 openaddresses
drwxr-xr-x  2 root            root             4096 Jul 28 19:54 openstreetmap
drwxr-xr-x  2 root            root             4096 Jul 28 19:55 placeholder
drwxr-xr-x  2 root            root             4096 Jul 28 19:55 polylines
drwxr-xr-x  4 root            root             4096 Jul 28 19:54 tiger
drwxr-xr-x  4 root            root             4096 Jul 28 19:54 whosonfirst

And looking at the whosonfirst data subdirectory you should see this:

$ ls -al /pelias/data/whosonfirst/data/
drwxr-xr-x  3 root root 4096 Jul 28 19:54 101
drwxr-xr-x  4 root root 4096 Jul 28 19:54 102
drwxr-xr-x  4 root root 4096 Jul 28 19:54 110
drwxr-xr-x  4 root root 4096 Jul 28 19:54 420
drwxr-xr-x  3 root root 4096 Jul 28 19:54 538
drwxr-xr-x  5 root root 4096 Jul 28 19:54 554
drwxr-xr-x  4 root root 4096 Jul 28 19:54 856
drwxr-xr-x 30 root root 4096 Jul 28 19:54 858

If you don't see the data in your data directory as mentioned above then something went wrong during data download steps.

Last week I provisioned a new EC2 machine and tested the build script start to finish and did not run into any issues. The build script ran end-to-end and once it was done I was able to query the Pelias API and all the services were up and running.

The machine was setup with the following:

3vivekb commented 7 years ago

I'm rebuilding and seeing something different this time. My original deployment didn't have all the data you show (maybe it got deleted since I used tmp? or my environment variable got unset?). I'll update when I know more.

Not sure why my instance got suppperrr sloww.

May I suggest you store the DATA_DIR in a different way? Perhaps inputted from a .env file or put it into the json config? And then maybe not suggest using /tmp/data/ and instead /pelias/data/ .

dianashk commented 7 years ago

@3vivekb, how are your setup efforts going? We've made some updates to the docker images, so be sure to pull down the latest from dockerhub before attempting another build.

orangejulius commented 6 years ago

Hi all, with the recently merged PR #42, we've now switched completely over to using a pre-built service that provides libpostal. Thus I believe any issues like this are no longer possible. If there are questions or issues, don't hesitate to comment or re-open this issue.