Question regarding reason of error: "Field "+field_key+ " missing in object "+str(feature["_id"]))

[taatuut]

Hi,

I'm importing geojson with polygons created with ogr2ogr from OpenStreetMap pbf into MongoDB. I created the 2dsphere index upfront, so all imported documents have geometries that MongoDB can index properly.

ogr2ogr -explodecollections -skipfailures -simplify .1 -makevalid -lco COORDINATE_PRECISION=4 -f GeoJSONSeq antarctica-latest.osm.json antarctica-latest.osm.pbf multipolygons

mongoimport --uri mongodb://127.0.0.1:27017/test --collection polygons --type json --file antarctica-latest.osm.json

Then in QGIS load the Mongo Connector and connect to the collection. This fails with:

Error: Field place missing in object 615c561502e90dfb3d7f6c00

Comes from def createAttributes(self,feature): in qgsmongolayer.py

So not a bug I guess, but a feature to ensure that (all) fields exist in (all) documents?

Would it make sense to make this behaviour optional, say if a field is not there just ignore and add NULL value for that row?

That way it would be possible to use data sets with Mongo Connector like OpenStreetMap that don't come with same set of fields on all features.

Regards,

Emil

[mk45]

Look here: https://github.com/gospodarka-przestrzenna/MongoConnector/issues/7#issuecomment-465213963 Furthermore, mongo is database, It can handle the data better than any other software including QGIS and our plugin.

We strongly encourage making data preprocess on database side.

[mk45]

Maybe something like this can do the work: https://docs.mongodb.com/manual/reference/operator/aggregation/ifNull/

[taatuut]

Your comment makes sense, although one of the benefits of a document database is supporting documents with varying fields/different schemas.

I found a quick fix if I make the following change in qgsmongolayer.py

    def createAttributes(self,feature):
        """
        Manage attribute of features
        :param feature: feature for witch we obtain geometry
        :return: prepared list of attributes
        """
        #check keys existence
        featureKeys=self.getFeatureKeys(feature)
#        for field_key in self.featuresKeys:
#            if field_key not in featureKeys:
#                raise ValueError("Field "+field_key+
#                                 " missing in object "+str(feature["_id"]))

        attributes=[]
        for key in self.featuresKeys:
            try:

                attrib=self.featuresKeyType[key](self.getFeatureValue(feature,key))
            except UnicodeEncodeError as e:
                # if unicode we simply pass it (it's a string)
                attrib = self.getFeatureValue(feature,key)
            except:
                attrib = "<null>"
#                raise TypeError("Can't cast attribute "+key+
#                                " in object "+str(feature['_id'])+
#                               " to type "+str(self.featuresKeyType[key]))

            attributes.append(attrib)
        return attributes

The data displays in QGIS, see the "null" value (string...) in the screenshot.