acalcutt / wdb-map-gen

OpenMapTiles basemap generation for WifiDB.net
Other
5 stars 0 forks source link

Errors from load_planet.sh load_wikidata.sh #7

Closed jas- closed 3 years ago

jas- commented 3 years ago

I am seeing errors with the load_planet.sh script such as the following:

[2021-11-14T23:45:28-05:00] 53:29:30 [fatal] bulk insert into "osm_highway_linestring": SQL Error: write tcp 127.0.0.1:60130->127.0.0.1:5432: write: connection reset by peer in query COPY "import"."osm_highway_linestring" ("osm_id", "geometry", "highway", "construction", "ref", "network", "z_order", "layer", "level", "indoor", "name", "name_en", "name_de", "tags", "short_name", "is_tunnel", "is_bridge", "is_ramp", "is_ford", "is_oneway", "is_area", "service", "usage", "public_transport", "man_made", "bicycle", "foot", "horse", "mtb_scale", "surface") FROM STDIN

And from load_wikidata.sh like the following:

Found 13 tables with the 'tags' hstore fields
  * osm_aerodrome_label_point
  * osm_city_point
  * osm_continent_point
  * osm_country_point
  * osm_island_point
  * osm_island_polygon
  * osm_marine_point
  * osm_peak_point
  * osm_poi_point
  * osm_poi_polygon
  * osm_state_point
  * osm_water_polygon
  * osm_waterway_linestring
Cache file wdb-map-gen/data/cache/wikidata-cache.json does not exist, will create
Searching for Wikidata IDs in 13 tables...
Traceback (most recent call last):
  File "/usr/local/bin/import-wikidata", line 284, in <module>
    asyncio.run(main(docopt(__doc__, version=openmaptiles.__version__)))
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/bin/import-wikidata", line 96, in main
    ids = await find_needed_wd_ids(conn, tables, verbose)
  File "/usr/local/bin/import-wikidata", line 253, in find_needed_wd_ids
    for row in await conn.fetch(query):
  File "/usr/local/lib/python3.8/dist-packages/asyncpg/connection.py", line 601, in fetch
    return await self._execute(
  File "/usr/local/lib/python3.8/dist-packages/asyncpg/connection.py", line 1639, in _execute
    result, _ = await self.__execute(
  File "/usr/local/lib/python3.8/dist-packages/asyncpg/connection.py", line 1664, in __execute
    return await self._do_execute(
  File "/usr/local/lib/python3.8/dist-packages/asyncpg/connection.py", line 1691, in _do_execute
    stmt = await self._get_statement(
  File "/usr/local/lib/python3.8/dist-packages/asyncpg/connection.py", line 393, in _get_statement
    statement = await self._protocol.prepare(
  File "asyncpg/protocol/protocol.pyx", line 168, in prepare
asyncpg.exceptions.UndefinedTableError: relation "osm_aerodrome_label_point" does not exist

Any insight is appreciated.

acalcutt commented 3 years ago

I don't think I've seen the first one. Sounds like the connection to Postgres is failing or getting dropped. How much memory have you given to Postgres? maybe it is running out of memory and droping the connection? or for that matter, when postgres is importing have you run something like 'htop' to see if your system has run out of memory and swap space or 'df' to make sure you have disk space left for the cache?.

The second error just means one of the tables use by the wiki data import script is missing. This is likely because loadplanet.sh did not complete properly and that table did not get created (all the "osm" tables get created by load_planet.sh). I'm going to assume config/env.config ( POSTGRES_HOST, POSTGRES_PORT, POSTGRES_DB, POSTGRES_USER, POSTGRES_PASS) have been configured since it looks from the wikidata import that some of the tables got imported.

Trust me I know how hard it is to get all this together. it took me almost 2 years to get it working myself, going though the various openmaptiles docker images scattered all over the place to see what they were actually doing and lots of trial and error. I've gotten these scripts to the point where I can pretty much get a working server in a few hours, so that is a great imrovement.

acalcutt commented 3 years ago

Also I should note, the article on my website could use a bit of updating. The one on the repo readme is a bit more up to date since most of the steps on my website got moved to tools/install_tools.sh

My current process is this 1.) Install debian 11 2.) git clone the wdb-map-gen repo 3.) Run wdb-map-gen/tools/install_tools.sh and say Y to all install questions 4.) configure config/env.config 5.) configure postgres (like mentioned at the bottom of https://www.techidiots.net/notes/mapbox-gl-tileserver-gl/create-an-open-map-tiles-map-without-docker ) 6.) run the import process. I usually like to run them one at a time so I can see the errors, so this might be why load_all.sh needs an update ./setup_db.sh ./create_border_csv.sh (optional) ./create_centerlines.sh (optional) ./load_osmborder.sh ./load_water.sh ./load_naturalearth.sh ./load_planet.sh ./load_wikidata.sh ./load_sql.sh

EDIT: also to note, It's not required buy I usually run these scripts with 'screen' so if my ssh session is dropped I can recover it with 'screen -r -d'. Since some of this can take a long time to run a lost session sucks. screen also allows you to do multiple console windows and split screens, so when I was troubleshooting it was nice to have htop or another tool up in one console and the import going in the other console.

jas- commented 3 years ago

Thanks again, I thought I had the latest from this repo but now that I see the install_tools.sh I was mistaken.

Not sure about the postgres errors. Here are my system specs: Memory:

MemTotal:       65724280 kB
MemFree:         7965168 kB
MemAvailable:   62274028 kB
Buffers:          576028 kB
Cached:         53975040 kB
SwapCached:        33324 kB
Active:         20116612 kB
Inactive:       35683176 kB
Active(anon):     807992 kB
Inactive(anon):  1257684 kB
Active(file):   19308620 kB
Inactive(file): 34425492 kB
Unevictable:        1096 kB
Mlocked:               0 kB
SwapTotal:      67108860 kB
SwapFree:       66704460 kB
Dirty:             17108 kB
Writeback:             0 kB
AnonPages:       1231272 kB
Mapped:          1348504 kB
Shmem:            816976 kB
KReclaimable:    1307076 kB
Slab:            1523116 kB
SReclaimable:    1307076 kB
SUnreclaim:       216040 kB
KernelStack:        9888 kB
PageTables:        28144 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    99971000 kB
Committed_AS:   40420576 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       90992 kB
VmallocChunk:          0 kB
Percpu:             7488 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:    49089676 kB
DirectMap2M:    17874944 kB
DirectMap1G:     1048576 kB

CPU (Intel I7):

rocessor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 158
model name      : Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
stepping        : 9
microcode       : 0xea
cpu MHz         : 3480.821
cache size      : 6144 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds
bogomips        : 5599.85
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 158
model name      : Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
stepping        : 9
microcode       : 0xea
cpu MHz         : 3793.367
cache size      : 6144 KB
physical id     : 0
siblings        : 8
core id         : 1
cpu cores       : 4
apicid          : 2
initial apicid  : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds
bogomips        : 5599.85
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 158
model name      : Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
stepping        : 9
microcode       : 0xea
cpu MHz         : 3503.296
cache size      : 6144 KB
physical id     : 0
siblings        : 8
core id         : 2
cpu cores       : 4
apicid          : 4
initial apicid  : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds
bogomips        : 5599.85
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 158
model name      : Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
stepping        : 9
microcode       : 0xea
cpu MHz         : 3439.088
cache size      : 6144 KB
physical id     : 0
siblings        : 8
core id         : 3
cpu cores       : 4
apicid          : 6
initial apicid  : 6
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds
bogomips        : 5599.85
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

processor       : 4
vendor_id       : GenuineIntel
cpu family      : 6
model           : 158
model name      : Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
stepping        : 9
microcode       : 0xea
cpu MHz         : 3502.133
cache size      : 6144 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds
bogomips        : 5599.85
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

processor       : 5
vendor_id       : GenuineIntel
cpu family      : 6
model           : 158
model name      : Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
stepping        : 9
microcode       : 0xea
cpu MHz         : 3782.435
cache size      : 6144 KB
physical id     : 0
siblings        : 8
core id         : 1
cpu cores       : 4
apicid          : 3
initial apicid  : 3
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds
bogomips        : 5599.85
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

processor       : 6
vendor_id       : GenuineIntel
cpu family      : 6
model           : 158
model name      : Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
stepping        : 9
microcode       : 0xea
cpu MHz         : 3550.282
cache size      : 6144 KB
physical id     : 0
siblings        : 8
core id         : 2
cpu cores       : 4
apicid          : 5
initial apicid  : 5
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds
bogomips        : 5599.85
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

processor       : 7
vendor_id       : GenuineIntel
cpu family      : 6
model           : 158
model name      : Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
stepping        : 9
microcode       : 0xea
cpu MHz         : 3528.287
cache size      : 6144 KB
physical id     : 0
siblings        : 8
core id         : 3
cpu cores       : 4
apicid          : 7
initial apicid  : 7
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds
bogomips        : 5599.85
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

I am in the process of reinstalling using the tools script. I know for certain I was using Postgres v14 vs. the v13 you have listed here. There may have been a version mismatch with some of the other tools as well since I used the blog post and where I felt applicable used the latest versions.

If I trim out some of the data in the config/wdbmap.yaml; layers and maybe languages on the first pass will it update that on a second or third pass?

Here is the config/env.config file as well:

POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_DB=osm
POSTGRES_USER=<user>
POSTGRES_PASS=<pass>

IMPORT_LAYER_FILE=openmaptiles.yaml
EXPORT_LAYER_FILE=wdbmap.yaml

RENDER_SCHEME=pyramid
MIN_ZOOM=0
MAX_ZOOM=14
BBOX=-180,-85.0511,180,85.0511
COPY_CONCURRENCY=96
UV_THREADPOOL_SIZE=96
MAX_PARALLEL_PSQL=60
TILE_TIMEOUT=1800000
MBTILES_NAME=tiles.mbtiles

Just trying to get it to work without having to wait days for it to fail etc.

acalcutt commented 3 years ago

To get this working initially, I would modify load_planet.sh to load a smaller area than the full planet. That will allow you to test if the process works much faster since there is not as much data to load in.

For example, in that file if you change download-osm planet -o data/$PBF to something like a state download-osm massachusetts -o data/$PBF

For your question about modifying 'config/wdbmap.yaml', if you notice load_planet.sh actually uses the IMPORT_LAYER_FILE in the config file, which is the original openmaptiles.yaml. while the EXPORT_LAYER_FILE, wdbmap.yaml, is used as part of export_mbtiles.sh. The reason for this is because when I tried to import with a stripped down yaml, sometimes I ran into issues later in the wikidata or sql scripts. I decided to keep the import using the original openmaptiles.yaml so I had all the tables the later processes might need...because it was easier that dealing with missing tables. So in short, you can change wdbmap.yaml but it only affects the output file...however i don't recommend changing the openmaptiles.yaml to save yourself some hassle.

For you memory it seems like you have 64GB, which should be plenty...however it only shows 7GB free which seems low (it might just be postgres using it all). like i mentioned, I would run 'htop' (install it with apt) and watch them memory and swap use while load_planet.sh is running and see if both are getting filled when the import crashes. if it does, that could be your issue and you could add more swap space.

Edit: also, I'm not sure about Posgresql 14. I wouldn't think it wouldn't be an issue but i haven't tried it yet. I can say I have had an issue building the extensions when Posgresql wasn't the latest version, so if you see an issue with the osml10n or pgsql it could be that posgresql dev tools don't match the posgresql version, which i usually fixed by upgrading Posgresql

acalcutt commented 3 years ago

Just wanted to note I marked these two scripts as optional above

./create_border_csv.sh (optional)
./create_centerlines.sh (optional)

Without running these, the other scripts just download the last version of the border and centerlines files I created, which saves a lot of time for testing (create_centerlines.sh took FOREVER)

jas- commented 3 years ago

Thanks for the tips, I was able to get everything done. I did notice an issue with the gdal requiring some dependencies that I had to roll back due to the ubuntu proj versioning forcing ogr2ogr to link to two separate versions of the libproj shared objects.