Frankly speaking, speed results are not so different just because we use parallel download and parallel run of several instances of bzip2 ( const simultaneousDownloads in download_data_all.js, for example ). Memory consumption is just 4 times higher for lbzip2 (200MB vs 50MB)
But reason why I start to investigate it is that in PiP there is hardcoded one single sqlite file whosonfirst-data-latest.db which extracted extremely slow by one CPU core when not using lbzip2.
Summary of changes:
lbzip2
installed on system we use it. If not, using legacybzip2
.Dockerfile
to installlbzip2
command-exists
Tested
lbzip2
version on my machine for commandnpm run download
. Results are below:Using lbzip2
Using bzip2
Frankly speaking, speed results are not so different just because we use parallel download and parallel run of several instances of bzip2 (
const simultaneousDownloads
indownload_data_all.js
, for example ). Memory consumption is just 4 times higher for lbzip2 (200MB vs 50MB)But reason why I start to investigate it is that in PiP there is hardcoded one single sqlite file
whosonfirst-data-latest.db
which extracted extremely slow by one CPU core when not using lbzip2.Fixes #480