Import symbol information from LIDAR-generated DXF files

[dg0yt]

dg0yt reported on Sourceforge [tickets:#261]:

DXF files generated by Terje's script for lidar data processing (http://tmsw.no/mapping/basemap_generation.html) contain layer and symbol information (such as "103_form_line"). This information shall be applied to the objects during import.

(The information may be imported as object tags and later be interpreted by the same function that would assing object types from OSM tags.)

See #260 for an example file.

[dg0yt]

Somebody posted on Sourceforge:

Support for layers in DXF would be great. A first step could perhaps be to let the user supply a file that for each DXF layer name specifies the OOMapper symbol to map it to?

[dg0yt]

• labels: --> File formats, Import, DXF
• Milestone: 0.6.0 --> 0.6.1

[TerjeWiigMathisen]

First of all, Thank You! to everyone who is working on providing an open source mapping program.

The test file I provided in https://github.com/OpenOrienteering/mapper/issues/828 contains vegetation areas, it seems like this is a separate issue from the basic 'Batch conversion of imported tags to ISOM symbols'?

I.e. we need a way for OOM to import lakes with islands and other complicated areas, not just a singly connected boundary line which must then be filled afterwards.

PS. I had to work for a while with the OCAD programmers before they got this right...

[dg0yt]

The custom DXF importer in the stable version is to be replaced by a GDAL/OGR based importer (unfinished, already in the unstable snapshots). GDAL/OGR should be able to handle things like that. But I didn't see any imported object from the test file from 828 with any of the importers so I can't tell you very much at the moment

[TerjeWiigMathisen]

The GDAL/OGR usage is good news, thanks!

It is strange that they didn't find any objects to import though, I'll enclose the result of importing that DXF file in OCAD and then loading the OCAD file in OOM which results in a file with 4698 area objects.

terje_vegetation_omap.zip

[TerjeWiigMathisen]

I hope I am getting closer now: I have downloaded the gdal library code and traced through the hatch reader function: From that it is obvious that the order of attributes is critical and in particular the following structure must be maintained:

91 -> Number of polylines describing the surface

must be immediately followed by this number of polylines each of which look like this

92 -> Number of vertices in the current line (Optional features like 73 -> Closed polygon) 10 -> x, 20 -> y repeated the number of times specified in the 92 field

My current code still imports cleanly in OCAD but croaks in OOM tile_603000_6657500.zip

It would be great to know where it breaks down this time!

[dg0yt]

With GDAL/OGR enabled for DXF in the settings dialog, it does now import some objects and properties. Unfortunately, due to the unfinished implementation in Mapper, it is still mirrored upside down. This is the major blocker for releasing the GDAL/OGR features:

The file now also opens in LibreOffice Draw.

[TerjeWiigMathisen]

This is wonderful progress and a big step towards making OOM suitable for all stages of base map creation. :-)

PS. I did notice that at least one DXF viewer also showed my vegetation objects upside down when I went via Shape -> DXF.

[dg0yt]

There are some graphics contexts where the y axis increases upwards, and some where it increases downwards. To add more confusion, Mapper displays a different sign than what is stored internally...

[TerjeWiigMathisen]

I just tried it after merging all my individual DXF tiles into a single file, you are correct that everything ends up upside down.

Just as important, the real world coordinates seems to be dropped and instead I got paper (mm) measurements. I'm pretty sure these two things are strongly related so when you fix one you'll also get the other. :-)

[dg0yt]

The latest unstable build should handle coordinates and spatial references much better.

[TerjeWiigMathisen]

The coordinates still seems to be messed up when importing DXF data.

Steps to reproduce:

1) Start a new blank map 2) Import a background (template) image with world file, set UTM zone (32 N in the case of Oslo), configure lower right corner status line to display UTM coordinates. 3) Import "contours.dxf" with contour line data for the exact same area, using the GDAL importer

At this point the reference location jumps away from the correct spot and the template is no longer visible. rotnes.zip

[dg0yt]

If you import into a map with reasonable georefencing (i.e. reasonable project coordinates for paper coordinate origin (0, 0), everything is okay.

But how do you load the data: Do you just open the DXF file? Or import into an empty file? Or open it as a template? The point is, there is no georeferencing in a DXF file. I even can't tell if the coordinates are meters or millimeters or something else. So it will take a lot of heuristics or dialogs with these files, unless the map is already properly georeferenced.

[TerjeWiigMathisen]

I appreciate that a DXF file just has coordinates with no actual real world reference, it is just a common default to say that "real world coordinates == UTM", at which point you just need the UTM Zone in order to turn it into a fixed real world location, right?

This is why I started by taking a blank map and then imported a template with world file and immediately added the UTM Zone: At this point this (template-only) map should have a proper real world reference.

When next a DXF import happens and the initial extents of that file happens to correlate very closely (in fact exactly) with the template world file reference, then it would seem that a very simple heuristic would suffice?

Alternatively, or in addition, you use the same dialog box as in the old importer which just ask if the import is in local or real world coordinates.

My final suggestion as a programmer would be to look at the DXF extents, and if they have a big offset from zero but reasonable total area, then it is quite likely that they are in fact UTM.

Anyway, I repeated my experiment but this time I actually started drawing on top of the template before I tried the DXF import, but I still got the huge reference shift. I.e. the map was completely georef'ed before the import happened.

This was all using the sample files in rotnes.zip, an area just north of Oslo, Norway.

[dg0yt]

I find it quite unusual to have DXF coordinates in ground units.

Please think again about matching UTM coordinates against the paper coordinates. We have to force the paper coordinates into a reasonable range. This is what is achieved most easily by setting reasonable projected coordinates for the paper coordinate origin (0, 0). There is no automation for this so far.

And please keep in my mind that so far I touched the generic GDAL/OGR implementation only. There are no special hooks for DXF yet.

Where do I find rotnes.zip?

[TerjeWiigMathisen]

DXF (and all image world files) as UTM meters have been the default for all georef'ed OCAD maps since georeferencing first became possible, so at least in the orienteering world this is not unusual. :-)

The rotnes.zip file was attached to my first message today, the one you replied to half an hour later or so.

[dg0yt]

I could reproduce the issue with rotnes.zip.

[TerjeWiigMathisen]

Good to know that you could reproduce the problem!

This is my 40th year as a programmer, and non-reproducible bugs are no fun at all.

The only thing that's worse is an intermittent bug... :-)