Advanced Digitizing Split Tool zeroes M values in a PolyLineZ shapefile

What is the bug or the crash?

When editing a PolyLineZ shapefile (type 13), and saving it, all M-values are set to zero. The ESRI shapefile documentation does not specify what the M-values represent. In the case of a distance, the edit may invalidate some of the values but they should not be set to zero. There is already a separate tool to do this, if desired. In my case, M represents a timestamp which I need to remain in the file.

Steps to reproduce the issue

1) Load my attached shapefile. 2) Use the Info tool to click on the line and expand Derived. You will see M values which look like YYYYMMDDHHMMSS, a huge integer stored accurately in a double. 3) Edit the layer, cut the line with the Advanced Digitizing Toolbox Split Features tool in the last segment of the line (to keep modified shapefile very similar to original). 4) Discard the small separated line. 5) View feature, as in step 2. You will see that the M value is not even displayed. It has been set to zero.

Note that ESRI defines "not a value" as anything less that -10**38, not zero. The hex dumps, below show that the values have been set to zero. If the current behavior is desirable for some reason, it should be controlled by an option (I cannot fine one), but there is already a tool for zeroing M values.

Below is an hex dump of the original and modified .shp file. Offsets are octal. Notice the end of the modified file is all zero.

D:\Stewart\GATC\Survey Data\Hike Inn\shp>od -h "2022-08-23_094634_T2 - Copy.shp" BEFORE EDIT 0000000 0000 0a27 0000 0000 0000 0000 0000 0000 0000020 0000 0000 0000 0000 0000 1e01 03e8 0000 0000040 000d 0000 397b f3d9 0d52 c055 b0f0 5bd7 0000060 4bcd 4041 c8be baa4 0d4f c055 7127 eaa1 0000100 4bcd 4041 d788 e628 ec3f 408c 007e 7585 0000120 4b02 408d ad00 fae1 6406 42b2 fd00 fae1 0000140 6406 42b2 0000 0100 0000 e800 000d 0000 0000160 397b f3d9 0d52 c055 b0f0 5bd7 4bcd 4041 0000200 c8be baa4 0d4f c055 7127 eaa1 4bcd 4041 0000220 0001 0000 000c 0000 0000 0000 df9a 41e2 0000240 0d50 c055 7127 eaa1 4bcd 4041 c8be baa4 0000260 0d4f c055 16c9 9e5b 4bcd 4041 549b 42f9 0000300 0d50 c055 817b 895a 4bcd 4041 0e91 9061 0000320 0d50 c055 8f37 a37b 4bcd 4041 8ab7 e021 0000340 0d50 c055 819f ccd4 4bcd 4041 99c2 3978 0000360 0d51 c055 2c57 d3ff 4bcd 4041 5443 8890 0000400 0d51 c055 3efb d888 4bcd 4041 920e d460 0000420 0d51 c055 e303 bf24 4bcd 4041 b63d 23a2 0000440 0d52 c055 93d7 bf9e 4bcd 4041 184f 6d81 0000460 0d52 c055 3c62 ada5 4bcd 4041 aebb bdd4 0000500 0d52 c055 5200 6c86 4bcd 4041 397b f3d9 0000520 0d52 c055 b0f0 5bd7 4bcd 4041 d788 e628 0000540 ec3f 408c 007e 7585 4b02 408d 007e 7585 0000560 4b02 408d d788 e628 ec3f 408c 3be2 c564 0000600 ef8f 408c b6c4 7378 00d7 408d 06a3 0b06 0000620 0d46 408d 2974 04e1 10c5 408d 7f78 31ef 0000640 1477 408d ec03 2dd6 1621 408d b6c4 7378 0000660 15d7 408d e9f7 a6ab 150a 408d 6afd ea41 0000700 1495 408d 753a 8e18 1606 408d ad00 fae1 0000720 6406 42b2 fd00 fae1 6406 42b2 ad00 fae1 0000740 6406 42b2 b500 fae1 6406 42b2 b800 fae1 0000760 6406 42b2 ba00 fae1 6406 42b2 bc00 fae1 0001000 6406 42b2 be00 fae1 6406 42b2 c000 fae1 0001020 6406 42b2 c200 fae1 6406 42b2 ec00 fae1 0001040 6406 42b2 ee00 fae1 6406 42b2 f100 fae1 0001060 6406 42b2 fd00 fae1 6406 42b2 0001074

D:\Stewart\GATC\Survey Data\Hike Inn\shp>od -h 2022-08-23_094634_T2.shp AFTER EDIT 0000000 0000 0a27 0000 0000 0000 0000 0000 0000 0000020 0000 0000 0000 0000 0000 1e01 03e8 0000 0000040 000d 0000 397b f3d9 0d52 c055 b0f0 5bd7 Word 32, 40 octal is shapefile type = d = 13 0000060 4bcd 4041 c8be baa4 0d4f c055 3efb d888 0000100 4bcd 4041 d788 e628 ec3f 408c ec03 2dd6 0000120 1621 408d 0000 0000 0000 0000 0000 0000 0000140 0000 0000 0000 0100 0000 e800 000d 0000 0000160 397b f3d9 0d52 c055 b0f0 5bd7 4bcd 4041 0000200 c8be baa4 0d4f c055 3efb d888 4bcd 4041 0000220 0001 0000 000c 0000 0000 0000 e75f f5d5 0000240 0d4f c055 3a30 bfbd 4bcd 4041 c8be baa4 0000260 0d4f c055 16c9 9e5b 4bcd 4041 549b 42f9 0000300 0d50 c055 817b 895a 4bcd 4041 0e91 9061 0000320 0d50 c055 8f37 a37b 4bcd 4041 8ab7 e021 0000340 0d50 c055 819f ccd4 4bcd 4041 99c2 3978 0000360 0d51 c055 2c57 d3ff 4bcd 4041 5443 8890 0000400 0d51 c055 3efb d888 4bcd 4041 920e d460 0000420 0d51 c055 e303 bf24 4bcd 4041 b63d 23a2 0000440 0d52 c055 93d7 bf9e 4bcd 4041 184f 6d81 0000460 0d52 c055 3c62 ada5 4bcd 4041 aebb bdd4 0000500 0d52 c055 5200 6c86 4bcd 4041 397b f3d9 0000520 0d52 c055 b0f0 5bd7 4bcd 4041 d788 e628 0000540 ec3f 408c ec03 2dd6 1621 408d d7f1 3122 0000560 15b9 408d d788 e628 ec3f 408c 3be2 c564 0000600 ef8f 408c b6c4 7378 00d7 408d 06a3 0b06 0000620 0d46 408d 2974 04e1 10c5 408d 7f78 31ef 0000640 1477 408d ec03 2dd6 1621 408d b6c4 7378 0000660 15d7 408d e9f7 a6ab 150a 408d 6afd ea41 0000700 1495 408d 753a 8e18 1606 408d 0000 0000 0000720 0000 0000 0000 0000 0000 0000 0000 0000 * 0001060 0000 0000 0000 0000 0000 0000 0001074

shp.zip

Versions

QGIS version 3.26.2-Buenos Aires QGIS code revision feec3d3b12f Qt version 5.15.3 Python version 3.9.5 GDAL/OGR version 3.5.1 PROJ version 9.0.1 EPSG Registry database version v10.064 (2022-05-19) GEOS version 3.10.3-CAPI-1.16.1 SQLite version 3.38.1 PDAL version 2.4.2 PostgreSQL client version unknown SpatiaLite version 5.0.1 QWT version 6.1.6 QScintilla2 version 2.13.1 OS version Windows 10 Version 2009

Active Python plugins db_manager 0.1.20 grassprovider 2.12.99 MetaSearch 0.3.6 processing 2.12.99 sagaprovider 2.12.99

Supported QGIS version

[X] I'm running a supported QGIS version according to the roadmap.

New profile

[X] I tried with a new QGIS profile

Additional context

This was first noticed in 3.26.1 and reproduced with 3.26.2 in a new project with a new profile.

Hi @stewartbholt, the issue is not related to the particular values of the M coordinate in your layer. The issue occurs also with any other value (except for the 0 value, in which case obviously you will not notice the issue).

It seems to me you are experiencing two different issues: 1) the "Split feature" digitizing tool incorrectly replaces any M value stored in all the vertices of the feature with the nan value without any warning: this happens also for non ESRI Shapefile layers (e.g. Memory or GeoPackage) 2) the nan value in the M coordinate is stored as 0 instead of nan (i.e. a value < -10**38 as per Shapefile technical description) if the layer is an ESRI Shapefile layer or if the layer is saved as an ESRI Shapefile layer. This doesn't happen if the layer is not an ESRI Shapefile layer (e.g. GeoPackage).

I had never tried to split a line which contained M values. To make sure I understand, are you agreeing that the split should not alter the M values? In that case the second issue doesn't come up. There are tools to remove the M values and I have not checked them but they should be putting an ESRI NaN value when used on a shapefile.

I was able to accomplish my immediate task of cleaning up the two ends of a line to meet other lines with the vertex tool which does not alter M values.

Stewart

On Sat, Aug 27, 2022 at 12:18 PM Andrea Giudiceandrea < @.***> wrote:

Hi @stewartbholt https://github.com/stewartbholt, the issue is not related to the particular values of the M coordinate in your layer. The issue occurs also with any other value (except for the 0 value, in which case obviously you will not notice the issue).

It seems to me you are experiencing two different issues:

the "Split feature" digitizing tool incorrectly replaces any M value stored in all the vertices of the feature with the nan value: this happens also for non ESRI Shapefile layers (e.g. Memory or GeoPackage)

the nan value in the M coordinate is stored as 0 instead of nan (i.e. a value < -10**38 as per Shapefile technical description) if the layer is an ESRI Shapefile layer or if the layer is saved as an ESRI Shapefile layer. This doesn't happen if the layer is not an ESRI Shapefile layer (e.g. GeoPackage).

— Reply to this email directly, view it on GitHub https://github.com/qgis/QGIS/issues/49971#issuecomment-1229221323, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7BMXGPAE27QKA7MN6OCF3V3I5VBANCNFSM57XJYLGA . You are receiving this because you were mentioned.Message ID: @.***>

are you agreeing that the split should not alter the M values?

@stewartbholt, yes, of course. The first issue is your actual issue. Anyway the second issue is independent from the first: the nan value is a valid value for M coordinate in Shapefile and should not be stored as 0.

I think the title of this issue report doesn't match with the actual issue. In fact I think it is not true that simply "saving PolyLineZ shapefile zeroes M values".

Could you please change the title in order to let it reflect the actual issue, i.e. that the Split feature tool alters/removes the M value of all the vertices of the split features?

Done. I was filling out the form as I prepared info and didn't realize the title still reflected my earlier incorrect assumptions about the problem.

Stewart

On Sat, Aug 27, 2022 at 8:17 PM Andrea Giudiceandrea < @.***> wrote:

are you agreeing that the split should not alter the M values?

@stewartbholt https://github.com/stewartbholt, yes, of course. The first issue is your actual issue. Anyway the second issue is independent from the first: the nan value is a valid value for M coordinate in Shapefile and should not be stored as 0.

I think the title of this issue report doesn't match with the actual issue. In fact I think it is not true that simply "saving PolyLineZ shapefile zeroes M values".

Could you please change the title in order to let it reflect the actual issue, i.e. that the Split feature tool alters/removes the M value of all the vertices of the split features?

— Reply to this email directly, view it on GitHub https://github.com/qgis/QGIS/issues/49971#issuecomment-1229344401, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7BMXGQK3SV26SAUQ5FMHTV3KVZFANCNFSM57XJYLGA . You are receiving this because you were mentioned.Message ID: @.***>

Using split tool means using GEOS which does not support M geometry, hence returns NaN. However, you could perform an action to join M value from initial geometry. It's not ideal, but the only workaround for now.

One solution could be to add a post-action, when it's possible, after using GEOS algorithms to ensure M validity, but it's applicable everywhere (thinking about buffer).

What do you think @agiudiceandrea @Djedouas ?

@Djedouas It seems to me you've already done that somewhere?

qgis / QGIS