MGCP TRD 4.5.1 FUN (Condition of Facility) Attribute - "Unknown" values changed to "Fully Functional"

[tdvorakcaci]

Describe the bug Field values attributed as FUN="Unknown" at data import with Hootenanny are either dropped at import or are translated to FUN="Fully Functional" when data is exported. This occurs for multiple feature classes, possibly all.

To Reproduce Steps to reproduce the behavior:

1. See screenshots below
2. Import MGCP TRD v4.5.1 data with Hootenanny
3. Export data with MGCP schema switcher with Hootenanny

Expected behavior While some features can be incorrectly attributed as FUN="Unknown", it is not desirable to lose this attribution when FUN may in fact be unknown. Features may be extracted from a hardcopy source, be underground, unverifiable due to tree canopy, etc. Analysts should determing FUN and attribution should not be automatically translated.

Screenshots FerryL and StorageP examples showing original data used with Hootenanny import (TOP) and Hootenanny exported data (BOTTOM)

Desktop (please complete the following information): N/A

Smartphone (please complete the following information): N/A

Additional context Add any other context about the problem here.

[mattjdnv]

Hi @tdvorakcaci,
I'm not sure what the best way to solve this is. A long time ago, the MGCP translator was changed to make FUN=6 (Fully Functional) the default for all features that have a FUN attribute. This was done to support an automated map rendering pipeline that has "issues" when the condition of the feature is unknown.

OSM (generally) does not store tags that are of no use - name = unknown, speed = unknown etc If the tag has a value then it is stored. If you don't have a value then you don't have a tag. This is different to the schema that are generally stored as FGDB or shapefiles where each attribute of a feature must have a value. This is why the default is to drop all of the UNK,N_A,'-32767` values when importing TDS,MGCP,GGDM etc

Some of the discussion is in this issue: #4959. An outcome from this is #5098 where we now have the option to preserve all of the no use attributes when translating to OSM.

Using a BD120 (Reef) polygon as an example. NOTE: This is debug output from the translation script that shows what goes in and what comes out. I.e. Tags in, Attributes Out & Attributes In, Tags Out OSM -> MGCP:

LayerName:   Geometry: Area  Element Type: Way
In tags: error:circular: :15:
In tags: natural: :reef:
TableName 0: ABD120  FCode: undefined  Geom: Area
LayerName:   Geometry: Area  Element Type: Way
Out attrs: FCODE: :BD120:
Out attrs: UID: :aafe35ea-acfa-4aea-abf6-2e7e6936d6f9:

Now import the ABD120.shp file. NOTE: It is full of default values

14:40:52.049 STATUS ...cpp/hoot/core/io/OgrReader.cpp(1195) Reading: ABD120...
LayerName: ABD120  Geometry: Area  Element Type:
In Attrs: ACC: :1:
In Attrs: CCN: :UNK:
In Attrs: COD: :1000:
In Attrs: FCODE: :BD120:
In Attrs: NAM: :UNK:
In Attrs: NFI: :N_A:
In Attrs: NFN: :N_A:
In Attrs: SRT: :0:
In Attrs: TXT: :N_A:
In Attrs: UID: :976e608c-1f8a-436c-81e3-1b805eeb0b3a:
In Attrs: WLE: :0:
LayerName: ABD120  Geometry: Area  Element Type:
Out tags: natural: :reef:
Out tags: source: :mgcp:abd120:
Out tags: source:accuracy:horizontal:category: :accurate:
Out tags: uuid: :{976e608c-1f8a-436c-81e3-1b805eeb0b3a}:

Import the ABD120 shape file with -D reader.drop.defaults=false to keep all of the default values.

14:43:26.224 STATUS ...cpp/hoot/core/io/OgrReader.cpp(1195) Reading: ABD120...
LayerName: ABD120  Geometry: Area  Element Type:
In Attrs: ACC: :1:
In Attrs: CCN: :UNK:
In Attrs: COD: :1000:
In Attrs: FCODE: :BD120:
In Attrs: NAM: :UNK:
In Attrs: NFI: :N_A:
In Attrs: NFN: :N_A:
In Attrs: SRT: :0:
In Attrs: TXT: :N_A:
In Attrs: UID: :8cc221bc-8558-4a97-9314-c47b56b41fc1:
In Attrs: WLE: :0:
LayerName: ABD120  Geometry: Area  Element Type:
Out tags: delineation: :unknown:
Out tags: gndb_id: :N_A:
Out tags: gndb_id:2: :N_A:
Out tags: hydrographic_vertical_position: :unknown:
Out tags: name: :UNK:
Out tags: natural: :reef:
Out tags: note: :N_A:
Out tags: source: :mgcp:abd120:
Out tags: source:accuracy:horizontal:category: :accurate:
Out tags: source:copyright: :UNK:
Out tags: source:name: :unknown:
Out tags: uuid: :{8cc221bc-8558-4a97-9314-c47b56b41fc1}:

Going back to the issue with the FUN attribute. You can explicitly tag features that have an unknown condition with condition=unknown. This will ensure that they export with FUN=0. On import, these will still be dropped - unless the reader.drop.defaults is unset - as there is no way to know that the attribute value is really unknown or is just a default value.

[brianhatchl]

I think their Hoot instance can be customized to preserve the "unknown" values on import by adding "reader.drop.defaults": "false" to core/conf/Import.conf

cc: @hummeltech

[MikeTho16]

@mattjdnv Although you are correct in that "unknown" values are not normally explicitly stored in OSM, there is no technical reason why they could not be. Of course if the output was going into the OSM database we would run the risk of someone coming along and removing any tag whose value is "unknown", but in our instance we would not face that concern.

[tdvorakcaci]

@mattjdnv @MikeTho16 I also think this is an issue, "A long time ago, the MGCP translator was changed to make FUN=6 (Fully Functional) the default for all features that have a FUN attribute." I don't think that is the case for Under Constrution? Under Construction and Unknown can be allowable values and desirable and important information to users of the data.

[brianhatchl]

There is also a config option now which can be overridden by clients such as JOSM with the advanced option key hoot.drop.defaults or by unchecking the "Drop defaults" in the preferences UI below

So that Unknown, UNK, N_A, No Information, 0, -32767, etc. values won't be stripped out during translation while interactively editing tags in a translated schema.

[mattjdnv]

@tdvorakcaci Yes, There is no technical reason why you can'1 store these values. It's just the OSM convention to not store them.

The only time a feature would get the default FUN=6 (Fully Functional) on export is when the feature does not have a condition tag or one of the lifecycle tags (destroyed, abandoned etc) on it. On export, the translator fills in the missing attributes with the default values. If you tag a feature with condition=construction you get the appropriate FUN value on export. Using the building=construction tag

LayerName:   Geometry: Area  Element Type: Way
In tags: building: :construction:
In tags: error:circular: :15:
In tags: name: :Building Construction:
TableName 0: AAL015  FCode: undefined  Geom: Area
LayerName:   Geometry: Area  Element Type: Way
Out attrs: FCODE: :AL015:
Out attrs: FUN: :1:
Out attrs: NAM: :Building Construction:
Out attrs: UID: :9f17d99b-a5b8-495e-aacb-d8bd5e214678:

Using the condition tag

LayerName:   Geometry: Area  Element Type: Way
In tags: building: :yes:
In tags: condition: :construction:
In tags: error:circular: :15:
In tags: name: :Different Building:
TableName 0: AAL015  FCode: undefined  Geom: Area
LayerName:   Geometry: Area  Element Type: Way
Out attrs: FCODE: :AL015:
Out attrs: FUN: :1:
Out attrs: NAM: :Different Building:
Out attrs: UID: :83377884-1968-4552-8cbd-5ac28242c10c:

[mattjdnv]

@tdvorakcaci Thinking more about the default FUN value, I agree that there may be a use case for having the condition of the feature be unknown but if the condition really is unknown, I believe that the user needs to explicitly tag the feature with condition=unknown. The current default FUN=6 (Fully Functional) is using this idea - The building/bridge/road etc is fully functional unless I tag it otherwise. If I change this, it will break the map rendering (CWS) that is used by a lot of NOME users.

The MGCP TRD has this to say about it:

2. Within each attribute there are a range of values of which one must be chosen to reflect the characteristics of the captured feature. Every effort should be made to attribute features as accurately as possible using all available sources. The following paragraphs explain some important considerations when attributing features:

   a. Attributes play an important role in symbolisation, and it is essential that due consideration is given to their population. Attributes need to be populated accurately in order to derive a symbol in digital or hardcopy map display, and for use in GIS analysis.

   b. Although a valid attribute, use of the ‘Unknown’ value for attribution should not be used as the ‘easy option’ and therefore, should be avoided. It may only be used when it is impossible to determine an attribute from the available sources.

If you turn off "drop defaults" when importing you get all of the 0 and unknown values and these are preserved in the export. E.g. Importing from shapefile:

LayerName: AAL015  Geometry: Area  Element Type:
In Attrs: ACC: :1:
In Attrs: CCN: :UNK:
In Attrs: FCODE: :AL015:
In Attrs: FFN: :0:
In Attrs: FUN: :0:
In Attrs: HGT: :-32767:
In Attrs: HWT: :0:
In Attrs: NAM: :A building:
In Attrs: NFI: :N_A:
In Attrs: NFN: :N_A:
In Attrs: SDP: :N_A:
In Attrs: SDV: :UNK:
In Attrs: SRT: :0:
In Attrs: TXT: :N_A:
In Attrs: UID: :64c78383-3481-4820-af95-8703f1bd9274:
In Attrs: VOI: :UNK:
LayerName: AAL015  Geometry: Area  Element Type:
Out tags: aeroway:obstruction: :UNK:
Out tags: building: :yes:
Out tags: condition: :unknown:
Out tags: feature_function: :unknown:
Out tags: gndb_id: :N_A:
Out tags: gndb_id:2: :N_A:
Out tags: height: :-32767:
Out tags: house_of_worship: :unknown:
Out tags: name: :A building:
Out tags: note: :N_A:
Out tags: source: :N_A:
Out tags: source:accuracy:horizontal:category: :accurate:
Out tags: source:copyright: :UNK:
Out tags: source:datetime: :UNK:
Out tags: source:name: :unknown:
Out tags: uuid: :{64c78383-3481-4820-af95-8703f1bd9274}:

Exporting the same feature:

LayerName:   Geometry: Area  Element Type: Way
In tags: aeroway:obstruction: :UNK:
In tags: building: :yes:
In tags: condition: :unknown:
In tags: error:circular: :15:
In tags: feature_function: :unknown:
In tags: gndb_id: :N_A:
In tags: gndb_id:2: :N_A:
In tags: height: :-32767:
In tags: house_of_worship: :unknown:
In tags: name: :A building:
In tags: note: :N_A:
In tags: source: :N_A:
In tags: source:accuracy:horizontal:category: :accurate:
In tags: source:copyright: :UNK:
In tags: source:datetime: :UNK:
In tags: source:ingest:datetime: :2022-02-01T17:01:30.220Z:
In tags: source:name: :unknown:
In tags: uuid: :{64c78383-3481-4820-af95-8703f1bd9274}:
TableName 0: AAL015  FCode: undefined  Geom: Area
LayerName:   Geometry: Area  Element Type: Way
Out attrs: ACC: :1:
Out attrs: CCN: :UNK:
Out attrs: FCODE: :AL015:
Out attrs: FFN: :0:
Out attrs: FUN: :0:
Out attrs: HGT: :-32767:
Out attrs: HWT: :998:
Out attrs: NAM: :A building:
Out attrs: NFI: :N_A:
Out attrs: NFN: :N_A:
Out attrs: SDP: :unknown:
Out attrs: SRT: :0:
Out attrs: TXT: :N_A:
Out attrs: UID: :64c78383-3481-4820-af95-8703f1bd9274:
Out attrs: VOI: :UNK:

You can still add the unknown condition tag to a feature and it is kept during export. E.g. exporting a generic building with an unknown condition.

LayerName:   Geometry: Area  Element Type: Way
In tags: building: :yes:
In tags: condition: :unknown:
In tags: error:circular: :15:
In tags: name: :A building:
TableName 0: AAL015  FCode: undefined  Geom: Area
LayerName:   Geometry: Area  Element Type: Way
Out attrs: FCODE: :AL015:
Out attrs: FUN: :0:
Out attrs: NAM: :A building:
Out attrs: UID: :d96c4d16-55c4-4b29-bf36-89dbebed8a8e:

[tdvorakcaci]

I think that works. Every effort should be made to code it, but updating them all to FUN=6 could be problematic. Using that option on import sounds like that will do the trick, and as we edit, we can specifically tag them as Unknown if necessary.