Open koly1983 opened 6 years ago
Hi Kolitha, thanks for the detailed report! We just moved our releases from a CloudFront distribution on Mapzen's AWS infrastructure to Github, so that might make the downloads take longer depending on the connection, as I don't think Github uses a CDN for release assets.
The download script keeps track of the latest update it has for each file (so it will only download from the server if there are new versions of the files available). If you've downloaded the parser using an external program, go to $DATADIR/libpostal
where $DATADIR
is whatever was specified during the configure
stage, extract the tarball there, and then run date -u > last_updated_parser
and try make
again. It should not attempt to download from Github the next time.
In the longer term, the parser is still a pretty large download, and bandwidth costs are highly variable around the world, so I would like to try to reduce the size of the model and make it more accessible.
Thanks you very much for your quick response, it worked. I wish this log is OK
I got the following log after running the build, please let me know if it's a successful build or a failure.
Libraries have been installed in: /usr/local/lib
If you ever happen to want to link against installed libraries in a given directory, LIBDIR, you must either use libtool, and specify the full pathname of the library, or use the '-LLIBDIR' flag during linking and do at least one of the following:
/bin/mkdir -p '/usr/local/include/libpostal'
/usr/bin/install -c -m 644 libpostal.h '/usr/local/include/libpostal'
make[2]: Leaving directory '/devsystem/ML/NLP/libpostal/python/libpostal/src'
make[1]: Leaving directory '/devsystem/ML/NLP/libpostal/python/libpostal/src'
Making install in test
make[1]: Entering directory '/devsystem/ML/NLP/libpostal/python/libpostal/test'
gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -MT test_libpostal-test.o -MD -MP -MF .deps/test_libpostal-test.Tpo -c -o test_libpostal-test.o test -f 'test.c' || echo './'
test.c
mv -f .deps/test_libpostal-test.Tpo .deps/test_libpostal-test.Po
gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -MT test_libpostal-test_expand.o -MD -MP -MF .deps/test_libpostal-test_expand.Tpo -c -o test_libpostal-test_expand.o test -f 'test_expand.c' || echo './'
test_expand.c
mv -f .deps/test_libpostal-test_expand.Tpo .deps/test_libpostal-test_expand.Po
gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -MT test_libpostal-test_parser.o -MD -MP -MF .deps/test_libpostal-test_parser.Tpo -c -o test_libpostal-test_parser.o test -f 'test_parser.c' || echo './'
test_parser.c
In file included from ../src/sparse_matrix.h:46:0,
from ../src/averaged_perceptron.h:27,
from ../src/address_parser.h:52,
from test_parser.c:8:
../src/matrix.h:19:2: warning: #warning "No CBLAS" [-Wcpp]
^
mv -f .deps/test_libpostal-test_parser.Tpo .deps/test_libpostal-test_parser.Po
gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -MT test_libpostal-test_transliterate.o -MD -MP -MF .deps/test_libpostal-test_transliterate.Tpo -c -o test_libpostal-test_transliterate.o test -f 'test_transliterate.c' || echo './'
test_transliterate.c
mv -f .deps/test_libpostal-test_transliterate.Tpo .deps/test_libpostal-test_transliterate.Po
gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -MT test_libpostal-test_numex.o -MD -MP -MF .deps/test_libpostal-test_numex.Tpo -c -o test_libpostal-test_numex.o test -f 'test_numex.c' || echo './'
test_numex.c
mv -f .deps/test_libpostal-test_numex.Tpo .deps/test_libpostal-test_numex.Po
gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -MT test_libpostal-test_trie.o -MD -MP -MF .deps/test_libpostal-test_trie.Tpo -c -o test_libpostal-test_trie.o test -f 'test_trie.c' || echo './'
test_trie.c
mv -f .deps/test_libpostal-test_trie.Tpo .deps/test_libpostal-test_trie.Po
gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -MT test_libpostal-test_string_utils.o -MD -MP -MF .deps/test_libpostal-test_string_utils.Tpo -c -o test_libpostal-test_string_utils.o test -f 'test_string_utils.c' || echo './'
test_string_utils.c
mv -f .deps/test_libpostal-test_string_utils.Tpo .deps/test_libpostal-test_string_utils.Po
gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -MT test_libpostal-test_crf_context.o -MD -MP -MF .deps/test_libpostal-test_crf_context.Tpo -c -o test_libpostal-test_crf_context.o test -f 'test_crf_context.c' || echo './'
test_crf_context.c
In file included from ../src/crf_context.h:20:0,
from ../src/crf_context.c:1,
from test_crf_context.c:8:
../src/matrix.h:19:2: warning: #warning "No CBLAS" [-Wcpp]
^
In file included from test_crf_context.c:8:0:
../src/crf_context.c:295:12: warning: ‘double_matrix_get_row’ is static but used in inline function ‘exp_state_trans_score_all’ which is not static
return double_matrix_get_row(self->exp_state_trans, t);
^
mv -f .deps/test_libpostal-test_crf_context.Tpo .deps/test_libpostal-test_crf_context.Po
gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -MT ../src/test_libpostal-strndup.o -MD -MP -MF ../src/.deps/test_libpostal-strndup.Tpo -c -o ../src/test_libpostal-strndup.o test -f '../src/strndup.c' || echo './'
../src/strndup.c
mv -f ../src/.deps/test_libpostal-strndup.Tpo ../src/.deps/test_libpostal-strndup.Po
gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -MT ../src/test_libpostal-file_utils.o -MD -MP -MF ../src/.deps/test_libpostal-file_utils.Tpo -c -o ../src/test_libpostal-file_utils.o test -f '../src/file_utils.c' || echo './'
../src/file_utils.c
mv -f ../src/.deps/test_libpostal-file_utils.Tpo ../src/.deps/test_libpostal-file_utils.Po
gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -MT ../src/test_libpostal-string_utils.o -MD -MP -MF ../src/.deps/test_libpostal-string_utils.Tpo -c -o ../src/test_libpostal-string_utils.o test -f '../src/string_utils.c' || echo './'
../src/string_utils.c
mv -f ../src/.deps/test_libpostal-string_utils.Tpo ../src/.deps/test_libpostal-string_utils.Po
gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -MT ../src/utf8proc/test_libpostal-utf8proc.o -MD -MP -MF ../src/utf8proc/.deps/test_libpostal-utf8proc.Tpo -c -o ../src/utf8proc/test_libpostal-utf8proc.o test -f '../src/utf8proc/utf8proc.c' || echo './'
../src/utf8proc/utf8proc.c
mv -f ../src/utf8proc/.deps/test_libpostal-utf8proc.Tpo ../src/utf8proc/.deps/test_libpostal-utf8proc.Po
gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -MT ../src/test_libpostal-trie.o -MD -MP -MF ../src/.deps/test_libpostal-trie.Tpo -c -o ../src/test_libpostal-trie.o test -f '../src/trie.c' || echo './'
../src/trie.c
mv -f ../src/.deps/test_libpostal-trie.Tpo ../src/.deps/test_libpostal-trie.Po
gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -MT ../src/test_libpostal-trie_search.o -MD -MP -MF ../src/.deps/test_libpostal-trie_search.Tpo -c -o ../src/test_libpostal-trie_search.o test -f '../src/trie_search.c' || echo './'
../src/trie_search.c
In file included from ../src/collections.h:8:0,
from ../src/trie.h:31,
from ../src/trie_search.h:10,
from ../src/trie_search.c:1:
../src/trie_search.c: In function ‘trie_search_prefixes_from_index’:
../src/log/log.h:52:55: warning: format ‘%zu’ expects argument of type ‘size_t’, but argument 3 has type ‘uint32_t {aka unsigned int}’ [-Wformat=]
^
../src/trie_search.c:670:21: note: in expansion of macro ‘log_debug’
log_debug("No space transition, phrase_len=%zu\n", phrase_len);
^
mv -f ../src/.deps/test_libpostal-trie_search.Tpo ../src/.deps/test_libpostal-trie_search.Po
gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -MT ../src/test_libpostal-transliterate.o -MD -MP -MF ../src/.deps/test_libpostal-transliterate.Tpo -c -o ../src/test_libpostal-transliterate.o test -f '../src/transliterate.c' || echo './'
../src/transliterate.c
mv -f ../src/.deps/test_libpostal-transliterate.Tpo ../src/.deps/test_libpostal-transliterate.Po
gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -MT ../src/test_libpostal-numex.o -MD -MP -MF ../src/.deps/test_libpostal-numex.Tpo -c -o ../src/test_libpostal-numex.o test -f '../src/numex.c' || echo './'
../src/numex.c
mv -f ../src/.deps/test_libpostal-numex.Tpo ../src/.deps/test_libpostal-numex.Po
gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -MT ../src/test_libpostal-features.o -MD -MP -MF ../src/.deps/test_libpostal-features.Tpo -c -o ../src/test_libpostal-features.o test -f '../src/features.c' || echo './'
../src/features.c
mv -f ../src/.deps/test_libpostal-features.Tpo ../src/.deps/test_libpostal-features.Po
/bin/bash ../libtool --tag=CC --mode=link gcc -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/devsystem/ml_data/libpostal"' -g -L/usr/local/lib -o test_libpostal test_libpostal-test.o test_libpostal-test_expand.o test_libpostal-test_parser.o test_libpostal-test_transliterate.o test_libpostal-test_numex.o test_libpostal-test_trie.o test_libpostal-test_string_utils.o test_libpostal-test_crf_context.o ../src/test_libpostal-strndup.o ../src/test_libpostal-file_utils.o ../src/test_libpostal-string_utils.o ../src/utf8proc/test_libpostal-utf8proc.o ../src/test_libpostal-trie.o ../src/test_libpostal-trie_search.o ../src/test_libpostal-transliterate.o ../src/test_libpostal-numex.o ../src/test_libpostal-features.o ../src/libpostal.la ../src/libscanner.la -lm
libtool: link: gcc -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR=\"/devsystem/ml_data/libpostal\" -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR=\"/devsystem/ml_data/libpostal\" -g -o .libs/test_libpostal test_libpostal-test.o test_libpostal-test_expand.o test_libpostal-test_parser.o test_libpostal-test_transliterate.o test_libpostal-test_numex.o test_libpostal-test_trie.o test_libpostal-test_string_utils.o test_libpostal-test_crf_context.o ../src/test_libpostal-strndup.o ../src/test_libpostal-file_utils.o ../src/test_libpostal-string_utils.o ../src/utf8proc/test_libpostal-utf8proc.o ../src/test_libpostal-trie.o ../src/test_libpostal-trie_search.o ../src/test_libpostal-transliterate.o ../src/test_libpostal-numex.o ../src/test_libpostal-features.o -L/usr/local/lib ../src/.libs/libpostal.so ../src/.libs/libscanner.a -lm
make[2]: Entering directory '/devsystem/ML/NLP/libpostal/python/libpostal/test'
make[2]: Nothing to be done for 'install-exec-am'.
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/devsystem/ML/NLP/libpostal/python/libpostal/test'
make[1]: Leaving directory '/devsystem/ML/NLP/libpostal/python/libpostal/test'
make[1]: Entering directory '/devsystem/ML/NLP/libpostal/python/libpostal'
make[2]: Entering directory '/devsystem/ML/NLP/libpostal/python/libpostal'
make[2]: Nothing to be done for 'install-exec-am'.
/bin/mkdir -p '/usr/local/lib/pkgconfig'
/usr/bin/install -c -m 644 libpostal.pc '/usr/local/lib/pkgconfig'
make[2]: Leaving directory '/devsystem/ML/NLP/libpostal/python/libpostal'
make[1]: Leaving directory '/devsystem/ML/NLP/libpostal/python/libpostal'
Yes, that looks good!
Hi,
I just downloaded the parser.tar. gz file via external downloader and followed the instruction: "go to $DATADIR/libpostal where $DATADIR is whatever was specified during the configure stage, extract the tarball there, and then run date -u > last_updated_parser and try make again. " However, each time, the script attempts to download the parser file. I am not sure if I can disable the downloading. Thanks,
The filename should be parser_model_file_version
(instead of last_updated_parser
) and its content not the current date, but the version string like "v1.0.0".
see src/libpostal_data
Let me try. According to this, the command should be date -u > parser_model_file_version ?
Let me try. According to this, the command should be date -u > parser_model_file_version ?
No. Look into src/libpostal_data
, where the file contents get set/read and compared with current version string, like "v1.0.0".
Since you downloaded from https://github.com/openvenues/libpostal/releases/download/v1.0.0/parser.tar.gz (look at the version in the path), the version should be "v1.0.0".
Put that into parser_model_file_version
, like so: echo "v1.0.0" > parser_model_file_version
It works. Thank you so much.
Hi, I downloaded the data file from an external downloader, uGET and extracted the parser.tar.gz in $DATADIR/libpostal ($DATADIR as mentioned during configuration). the filegot extracted as parser
. I've then run both the commandsdate -u > last_updated_parser
and echo "v1.0.0" > parser_model_file_version
, double checked the files for the changes. Running the make
command still starts to check for new files and begins downloading them. How do i disable the downloading ?
hi, this is still a pretty big issues.
I'm unable to install this on my computer becasue of the parser failing to extract.
Is there a good workaround that's consistent or something we can change in the make file to solve this problem where the parser stops the installation?
hi, this is still a pretty big issues.
I'm unable to install this on my computer becasue of the parser failing to extract.
Is there a good workaround that's consistent or something we can change in the make file to solve this problem where the parser stops the installation?
fix is very simple, a hack(shortcut) I used after wasting a lot my time. open "/src/libpostal_data.in" and edit replace multipart download with curl single, "download_release_multipart $url $local_path $num_chunks" to "curl -L $url --retry 3 --retry-delay 2 -o $local_path"
You can also download all the data files referenced in there and then change the "LIBPOSTAL_BASE_URL" to serve files from localhost.
Hi!
I was trying to build the source after taking the source code.
My country is
I'm located in india , and the connection seems to be very slow
Here's how I'm using libpostal
I wanted to do experiment about the library for a business Use case came up in my current project
Here's what I did step by step
1) git clone https://github.com/openvenues/libpostal 2) cd libpostal 3) run ./bootstrap.sh 4) set data folder to ./configure --datadir=[...some dir with a few GB of space...] 5) run the build (make) sudo make install
Here is the error I got :
koly@koly-X555UA:/devsystem/ML/NLP/libpostal/python/libpostal$ sudo make install [sudo] password for koly: Making install in src make[1]: Entering directory '/devsystem/ML/NLP/libpostal/python/libpostal/src' ./libpostal_data download all /devsystem/ml_data/libpostal Checking for new libpostal language classifier data file... libpostal language classifier data file up to date Checking for new libpostal data file... libpostal data file up to date Checking for new libpostal parser data file... New libpostal parser data file available Downloading multipart: https://github.com/openvenues/libpostal/releases/download/v1.0.0/parser.tar.gz, size=752483239, num_chunks=11 Downloading part 1: filename=/devsystem/ml_data/libpostal/parser.tar.gz.1, offset=0, max=67108863 Downloading part 2: filename=/devsystem/ml_data/libpostal/parser.tar.gz.2, offset=67108864, max=134217727 Downloading part 3: filename=/devsystem/ml_data/libpostal/parser.tar.gz.3, offset=134217728, max=201326591 Downloading part 4: filename=/devsystem/ml_data/libpostal/parser.tar.gz.4, offset=201326592, max=268435455 Downloading part 5: filename=/devsystem/ml_data/libpostal/parser.tar.gz.5, offset=268435456, max=335544319 Downloading part 6: filename=/devsystem/ml_data/libpostal/parser.tar.gz.6, offset=335544320, max=402653183 Downloading part 7: filename=/devsystem/ml_data/libpostal/parser.tar.gz.7, offset=402653184, max=469762047 Downloading part 8: filename=/devsystem/ml_data/libpostal/parser.tar.gz.8, offset=469762048, max=536870911 Downloading part 9: filename=/devsystem/ml_data/libpostal/parser.tar.gz.9, offset=536870912, max=603979775 Downloading part 10: filename=/devsystem/ml_data/libpostal/parser.tar.gz.10, offset=603979776, max=671088639 Downloading part 11: filename=/devsystem/ml_data/libpostal/parser.tar.gz.11, offset=671088640, max=752483239 Makefile:4118: recipe for target 'all-local' failed make[1]: [all-local] Error 123 make[1]: Leaving directory '/devsystem/ML/NLP/libpostal/python/libpostal/src' Makefile:454: recipe for target 'install-recursive' failed make: [install-recursive] Error 1
The problem ::
Every time I run the build, i'm getting stuck at downloading the parser.tar.gz file so I downloaded the file using uGet and added to the folder, but when i'm trying to build the source its trying again and again downloading the same file and fails
Please help me to identify a way to bypass file downloading and make the build a success.
Thank you very much, Kolitha.