Segmentation Fault On "Unexpected Geometry"

[apjoseph]

Getting a segmentation fault when performing a basic select on a shapefile foreign table:

2017-12-06 02:22:41.449 UTC [1] LOG:  server process (PID 2044) was terminated by signal 11: Segmentation fault
2017-12-06 02:22:41.449 UTC [1] LOG:  terminating any other active server processes
2017-12-06 02:22:41.449 UTC [2103] WARNING:  terminating connection because of crash of another server process
2017-12-06 02:22:41.449 UTC [2103] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2017-12-06 02:22:41.449 UTC [2103] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2017-12-06 02:22:41.450 UTC [2041] WARNING:  terminating connection because of crash of another server process
2017-12-06 02:22:41.450 UTC [2041] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2017-12-06 02:22:41.450 UTC [2041] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2017-12-06 02:22:41.453 UTC [1] LOG:  all server processes terminated; reinitializing
2017-12-06 02:22:41.466 UTC [2140] LOG:  database system was interrupted; last known up at 2017-12-06 00:51:48 UTC
2017-12-06 02:22:41.498 UTC [2140] LOG:  database system was not properly shut down; automatic recovery in progress
2017-12-06 02:22:41.502 UTC [2140] LOG:  redo starts at 1A/8D35CB08
2017-12-06 02:22:41.503 UTC [2140] LOG:  unexpected pageaddr 1A/4C382000 in log segment 000000010000001A0000008D, offset 3678208
2017-12-06 02:22:41.503 UTC [2140] LOG:  redo done at 1A/8D3813F0
2017-12-06 02:22:41.525 UTC [1] LOG:  database system is ready to accept connections

When successfully attempting to import the shapefile with ogr2ogr, I received the following message in debug output:

Warning 1: organizePolygons() received an unexpected geometry. Either a polygon with interior rings, or a polygon with less than 4 points, or a non-Polygon geometry. Return arguments as a collection. Warning 1: Geometry of polygon of fid 607455 cannot be translated to Simple Geometry. All polygons will be contained in a MultiPolygon.

I received the same debug message when attempting to isolate the raw geometry via the GDAL python bindings using the most basic possible script:

from osgeo import ogr
import os

shapefile = "testshp.shp"
driver = ogr.GetDriverByName("ESRI Shapefile")
dataSource = driver.Open(shapefile, 0)
layer = dataSource.GetLayer()

layer.SetAttributeFilter("key = '698777'")

for feature in layer:
    print feature.GetField("key")
    geom = feature.GetGeometryRef()
    print geom.ExportToWkt()

which proceeds without a crash and yields the following WKT (EPSG:2278):

MULTIPOLYGON (((2112445.77193154 13677916.1665116,2112445.89276364 13677915.5667031,2112443.78972197 13677924.6541983,2112445.27247515 13677918.5634415,2112445.5906834 13677917.0666083,2112445.77193154 13677916.1665116)),((2112443.36732164 13677926.1855915,2112443.60642582 13677925.3172389,2112441.07458381 13677933.7414376,2112443.12821744 13677927.0542002,2112443.36732164 13677926.1855915)),((2112440.59048744 13677935.1471347,2112440.91791169 13677934.1937899,2112437.7839572 13677942.6574126,2112440.26280718 13677936.1009914,2112440.59048744 13677935.1471347)),((2112437.25890078 13677943.9207736,2112437.68795712 13677942.8878128,2112433.92603414 13677951.3657716,2112436.82984444 13677944.9537345,2112437.25890078 13677943.9207736)),((2112432.87003329 13677953.5166853,2112433.8817461 13677951.4566516,2112429.64468673 13677959.5995061,2112432.87003329 13677953.5166853)),((2112424.58151472 13677967.9804408,2112429.49287862 13677959.8654903,2112424.58151472 13677967.9804408)),((2112424.58151472 13677967.9804408,2112390.23166342 13677929.0922179,2112381.41834441 13677919.1153779,2112363.85724245 13677899.2402901,2112298.11434221 13677939.1530259,2112337.26701333 13677995.2690385,2112339.28634293 13677997.8904805,2112341.54426472 13678000.3112185,2112344.0233707 13678002.5066762,2112346.69806083 13678004.4573977,2112349.49563104 13678006.1152551,2112352.53896147 13678007.5631922,2112355.59816389 13678008.6650171,2112355.65397194 13678008.6821691,2112358.86011848 13678009.5118658,2112362.12975308 13678010.0300102,2112365.43420371 13678010.2378823,2112368.77859037 13678010.1262663,2112372.02774495 13678009.7146179,2112375.2369635 13678008.9926974,2112375.74691591 13678008.8278332,2112378.40522201 13678007.9584565,2112381.44036043 13678006.6374955,2112383.0480417 13678005.7476388,2112384.33725873 13678005.0349342,2112398.51352599 13677996.2950873,2112399.51883879 13677995.4653906,2112402.26444097 13677993.2445888,2112404.55026679 13677991.2078512,2112405.88633186 13677990.0425383,2112406.72038051 13677989.2310176,2112409.37407863 13677986.6940556,2112412.72153729 13677983.2052849,2112415.92153983 13677979.581346,2112417.04563673 13677978.1979209,2112418.97101425 13677975.828383,2112421.08685593 13677973.0003487,2112421.85306454 13677971.9479319,2112422.6738012 13677970.754715,2112424.58151472 13677967.9804408)))

Unfortunately there appears to be no method to view the raw geometry -looks like GDAL needs to perform the transformation as soon as each feature is created.

If I limit to everything before that row, query works with no issue. Only occurs when row 607455 is reached -so appears to be an issue with ogr_fdw at that specific geometry. Perhaps some sort of memory allocation issue where the MultiPolygon is larger than the original geometry?

Versions: "POSTGIS="2.4.2 r16113" PGSQL="100" GEOS="3.6.2-CAPI-1.10.2 4d2925d6" SFCGAL="1.3.0" PROJ="Rel. 4.9.3, 15 August 2016" GDAL="GDAL 2.3.0dev, released 2017/99/99" LIBXML="2.9.3" LIBJSON="0.11.99" TOPOLOGY RASTER" GDAL Commit: 36222a597c8118b048f1b042660ca3f227f58f31 ogr_fdw Commit: b80242ea89ec3898f1817267a440c87a60ef2b4a

[pramsey]

Any chance you could share the shape file? Without it I'll just be guessing and the debug cycle could take a long time. Since the code that unrolls the shape format is being called by ogr_fdw, my guess is that something in my handling of the warning is bad.

[apjoseph]

Here's a link -unfortunately it's too big to attach. Nothing special needs to be done to trigger the issue, occurs as soon as row 607455 is reached.

[pramsey]

hum, running the latest master version... it works for me

select * from zone where zonekey = 698777;

fid        | 607455
geom       | 0106000020AD100000070000000103000000010000000600000020A7CEE2DE1D404120105485AB166A41401446F2DE1D4041906E2272AB166A41009C15E5DD1D40415831EF94AC166A414077E0A2DE1D404178B607D2AB166A4180839BCBDE1D4041C0A721A2AB166A4120A7CEE2DE1D404120105485AB166A4101030000000100000005000000406504AFDD1D4041B05DF0C5AC166A41805C9FCDDD1D404120D226AAAC166A4160F68B89DC1D404140DBB9B7AD166A41E06D6990DD1D40412802BCE1AC166A41406504AFDD1D4041B05DF0C5AC166A4101030000000100000005000000A017954BDC1D4041C853B5E4AD166A4160217E75DC1D4041E88633C6AD166A41A0B558E4DA1D4041408609D5AE166A4160AAA321DC1D404160523B03AE166A41A017954BDC1D4041C853B5E4AD166A410103000000010000000500000020A923A1DA1D404148FA76FDAE166A41A0FA0ED8DA1D404170F668DCAE166A41604988F6D81D40419066B4EBAF166A41C057386ADA1D404128FE841EAF166A4120A923A1DA1D404148FA76FDAE166A410103000000010000000400000040405D6FD81D404190AF8830B0166A41600EDDF0D81D4041E0E39CEEAF166A41401885D2D61D404170272FF3B0166A4140405D6FD81D404190AF8830B0166A410103000000010000000300000000136F4AD41D404150C55FFFB1166A4180A516BFD61D4041C018B2FBB0166A4100136F4AD41D404150C55FFFB1166A410103000000010000002800000000136F4AD41D404150C55FFFB1166A41A025A71DC31D4041E072F322AD166A41404F8CB5BE1D4041102DB1E3AB166A41E01EBAEDB51D4041F074B067A9166A4100C4A20E951D40417896E564AE166A41207E2DA2A81D404188F69B68B5166A41A0E2A6A4A91D404108D17EBCB5166A416077AAC5AA1D40416880F509B6166A41A0CFFD02AC1D404100B13650B6166A41A00E5A59AD1D40419800A38EB6166A4180D670BFAE1D4041682BB0C3B6166A4180B0FC44B01D4041B0AB05F2B6166A4160A290CCB11D4041F0D14715B7166A41405AB5D3B11D40415054D415B7166A41C05C186EB31D404150346130B7166A41C0BF9B10B51D4041E8D7F540B7166A41C0FC93B7B61D404170BB9C47B7166A4160D9A863B81D4041885F0A44B7166A4180258D03BA1D40416826DE36B7166A41E0D1549EBB1D4041402DC41FB7166A41C0F09ADFBB1D4041189C7D1AB7166A41A050DE33BD1D404108ADABFEB6166A4100BB5DB8BE1D4041F05C66D4B6166A41003B2686BF1D404128A8ECB7B6166A41404B2B2BC01D4041582E1EA1B6166A414038BB41C71D4041D85A7189B5166A41404F69C2C71D4041D87AE46EB5166A41A033D921C91D4041F8ABD327B5166A4160246F46CA1D404198B7A6E6B4166A41805273F1CA1D404148795CC1B4166A41C06D355CCB1D4041207F64A7B4166A4100CFE1AFCC1D404128B43556B4166A4180555B5CCE1D404190B191E6B3166A416004F5F5CF1D4041E0629A72B3166A41A06CD785D01D4041305E5546B3166A41E0314A7CD11D4041101D82FAB2166A4160181E8BD21D404160DB02A0B2166A41003831EDD21D40415075557EB2166A41201E3F56D31D404100A02658B2166A4100136F4AD41D404150C55FFFB1166A41
zonekey    | 698777
caseno     | 
base       | I-1
specdistri | 
speccondit | 
speccond_1 | 
zoning     | I-1
ordinancek | 0
zoningdeta | I-1
basedescri | General Industrial District (Sec. 35-310.13)
specdist_1 | 
entrydate  | 2015-05-21
modifiedda | 2015-05-21
shape_area | 7968.14
shape_len  | 455.526

More about your platform?

[apjoseph]

After further testing, It looks like removing the --enable-debug flag when configuring GDAL works -so definitely seems as though the extra debug messages are to blame. You should be able to replicate by adding that flag during GDAL configuration.

[pramsey]

Aha, there's a very recent commit that fixed some serious awfulness in that code, upgrade to master and see if the problem is gone.

[apjoseph]

Tested again, compiling GDAL with --enable-debug and that commit does indeed fix the issue, thanks! Glad the segfault was not a more serious issue with the geometry itself!