Running natural_earth.sh results in error "tolerance condition error"

[adamakhtar]

When I run natural_earth.sh on master on my Arm M1 macbook pro I get the error described in the title.

For instance, if I comment out all of the ne source urls in the script except for the last one and run the script I get this:

./natural_earth.sh
NOTICE:  extension "postgis" already exists, skipping
CREATE EXTENSION
fetching https://naciscdn.org/naturalearth/10m/physical/ne_10m_ocean.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 3114k  100 3114k    0     0  1936k      0  0:00:01  0:00:01 --:--:-- 1941k
Archive:  0.zip
  inflating: 0/ne_10m_ocean.README.html
 extracting: 0/ne_10m_ocean.VERSION.txt
 extracting: 0/ne_10m_ocean.cpg
  inflating: 0/ne_10m_ocean.dbf
  inflating: 0/ne_10m_ocean.prj
  inflating: 0/ne_10m_ocean.shp
  inflating: 0/ne_10m_ocean.shx
ERROR 1: tolerance condition error
ERROR 1: tolerance condition error
ERROR 1: tolerance condition error
ERROR 1: tolerance condition error
ERROR 1: tolerance condition error
ERROR 1: tolerance condition error
ERROR 1: tolerance condition error
ERROR 1: tolerance condition error
ERROR 1: tolerance condition error
ERROR 1: tolerance condition error
ERROR 1: tolerance condition error
ERROR 1: tolerance condition error
ERROR 1: tolerance condition error
ERROR 1: tolerance condition error
ERROR 1: tolerance condition error
ERROR 1: tolerance condition error
ERROR 1: tolerance condition error
ERROR 1: tolerance condition error
ERROR 1: tolerance condition error
ERROR 1: Reprojection failed, err = -20, further errors will be suppressed on the transform object.

This is the GDALversion I am using:

ogr2ogr --version
GDAL 3.4.1, released 2021/12/27

I have also been able to reproduce this on an intel s3 instance so I do not think this is ARM related.

I posted a question on the GDAL mailing list re: the above and I received this answer:

This dataset includes latitude up to 90 degree north, which projects to infinity in EPSG:3857. You should use the -clipsrc option of ogr2ogr to for example cut at 85 degree

I'm not exactly sure how to use this option correctly. Looking at some examples online I found this -clipsrc -180.1 -85.0511 180.1 85.0511 and tried it like so:

OGR_ENABLE_PARTIAL_REPROJECTION=true ogr2ogr -unsetFieldWidth -t_srs EPSG:3857 -clipsrc -180.1 -85.0511 180.1 85.0511 -nlt PROMOTE_TO_MULTI -f PostgreSQL PG:"dbname='ne' host='localhost' port='5432' user='myuser' password=''redacted" ne_10m_ocean.shp

The command no longer fails and I find 3 rows in the created table but just because there were no errors doesn't mean its the correct fix for this problem.

Could you confirm this is indeed a bug and the above is the correct fix?

[adamakhtar]

As per my last comment I tried updating natural_earth.sh to

OGR_ENABLE_PARTIAL_REPROJECTION=true ogr2ogr -unsetFieldWidth -t_srs EPSG:3857 -clipsrc -180.1 -85.0511 180.1 85.0511 -nlt PROMOTE_TO_MULTI -f PostgreSQL PG:"dbname='$DB_NAME' host='$DB_HOST' port='$DB_PORT' user='$DB_USER' password='$DB_PW'"

i.e. adding -clipsrc -180.1 -85.0511 180.1 85.0511.

and then ran it as per normal. Nearly all the natural earth files ran fine but but one or two still produce a few of the above errors but far less than before.

Is no one else getting the above errors? I'm trying to work out if its an environment issue or I wonder if its anything to do with the datas' source urls being updated recently. Perhaps the raw natural earth data has changed?

(cc @ARolek 🙇🏻)

[adamakhtar]

Sorry I forgot to add in my last comment the error I am still seeing. Here it is

Archive:  61.zip
  inflating: 61/ne_10m_lakes_north_america.README.html
 extracting: 61/ne_10m_lakes_north_america.VERSION.txt
 extracting: 61/ne_10m_lakes_north_america.cpg
  inflating: 61/ne_10m_lakes_north_america.dbf
  inflating: 61/ne_10m_lakes_north_america.prj
  inflating: 61/ne_10m_lakes_north_america.shp
  inflating: 61/ne_10m_lakes_north_america.shx
Input geom 0 is INVALID: Self-intersection at or near point -93.810065734214476 50.475055438868417 (-93.810065734214475697 50.475055438868416502)
<A>
POLYGON ((-93.8179440637089925 50.4308355812790978, -93.8499163576565536 50.4208094165111191, -93.8967956691880374 50.4166883870498594, -93.9280410214826560 50.4276889160359048, -93.8921053046715883 50.4453518722649932, -93.8087717138336359 50.4519046170063419, -93.8132948526465356 50.4662333068101603, -93.8047481692926794 50.4895833234770492, -93.7875066521550735 50.5108989376523567, -93.7705259140755487 50.5210039017706691, -93.7308350412120319 50.5192439160760927, -93.7095650821384538 50.5116950072548505, -93.7322536045641499 50.5097336130517576, -93.7615301006236734 50.5113228920604271, -93.7726259035747489 50.4974251259773865, -93.7905495188592937 50.4839842391629503, -93.8102983765965064 50.4761144003827482, -93.8072212463625306 50.4621076555067987, -93.7644947641007604 50.4549225835861961, -93.7094546981023768 50.4529433803538438, -93.6758725440474223 50.4453499740687974, -93.6125031029548325 50.4601394845921334, -93.5948419728205891 50.4471434799242147, -93.6092826873922945 50.4406438041435905, -93.6440307038893565 50.4364653674272674, -93.6894898636331135 50.4364525552598053, -93.7311121568103971 50.4436828817032392, -93.7754336800916519 50.4418257547950617, -93.8179440637089925 50.4308355812790978))
</A>
ERROR 1: TopologyException: Input geom 0 is invalid: Self-intersection at -93.810065734214476 50.475055438868417

[ARolek]

@adamakhtar hey there! apologies for the slow response to these comments.

Could you confirm this is indeed a bug and the above is the correct fix?

I need to give this a test myself. I'm not sure if this is an appropriate fix.

Is no one else getting the above errors? I'm trying to work out if its an environment issue or I wonder if its anything to do with the datas' source urls being updated recently. Perhaps the raw natural earth data has changed?

There was a PR not too long ago submitted which changed the data. I ran the import scripts after the change to confirm things were working prior to accepting the PR. The data might have changed.

ERROR 1: TopologyException: Input geom 0 is invalid: Self-intersection at -93.810065734214476 50.475055438868417

This error is not ideal, but does happen sometimes. I believe the data is still imported though and can potentially be fixed by running ST_MakeValid(geom) on the geometry. This error is different than the tolerance errors you're receiving.

[adamakhtar]

Thanks @ARolek for the help. I'll look into ST_MakeValid(geom).