Functionality matrix

[VasiliosKalogirou]

I have prepared a functionality matrix for the other two plugins for the purposes of my dissertation project. I will upload tonight a table comparing the functionality of all 3 plugins used for visualising data.

[jeleniste]

Are you able to create some test dataset for testing some bigger data? I have on my git some tool for create mongo layers from czech cadastre data with nested columns... I`m able to prepare some layers. Or you prefer to use your plugin to save some "flat" data for example from osm?

[VasiliosKalogirou]

I have a number of sample datasets, some of them are big as well. I can upload everything in a few hours. However, I suggest we focus on the functionality of the plugin first (name of the layer created, format of the attribute table, appearance of the interface, compatibility between visualising data to QGIS and storing data from QGIS to MongoDB), and then test for bigger data.

[VasiliosKalogirou]

Hi all, As promised, I uploaded 5 files to test the plugins, 3 geojson files downloaded from OSM and the two files mentioned in MongoDB Manual, https://docs.mongodb.com/manual/tutorial/geospatial-tutorial/. I also uploaded a functionality matrix I created in word document. Ignore the first point, it's about storing a geojson feature collection in MongoDB, which will be stored as a single document. This is not important, features should be stored as separate documents in the db. I can send you my dissertation project report if you want to have a look, as I have a much more descriptive summary in it (sorry Jan, I didn't evaluate your plugin in my report:). The other two points, outside the matrix I created, I think should be: how does the interface look like, and how do the plugins respond to the collections stored in MongoDB using my plugin. Jan's plugin works great. Maciej's plugin could not load the data. After these, we can focus on adding more functionality to the merged plugin and taking care of loading large datasets without crushing. Please let me know your thoughts.

[jeleniste]

Problem with importing in mongo memorizer was in ObjectId(..., because i used in my test data ma own primary key and in module was ID casted to int and used as ogc_fid, i corected this few days ago after testing data saved from your module. In recent version it should work. Word document, sorry, but using binaries in git is not good option, i prefer use some text format (rst, md, or, for example html) because of versioning and changes. IMHO markdown is best option because of display ON github.

[mk45]

1. @jeleniste Than You! I was going to say that pushing .docx to github is probably not the best file format for versioning nor *nix community access.
2. You're right! I'm changing to assign exact name for created layer as mongo collection name.
3. Plugin is for Qgis witch supports only single type elements on layer. Thats why I consider neighborhoods dataset broken as it contains Polygons and MultiPoligons. Of course plugin could cast all previous data to Multi* variant. But is plugin responsible for fixing the data? No plugin should not touch it at all. What if there is Point geometry in the LineString set? Cast it ? To what ? That is all the same situations, and showing user an Error with reference is the best we can do.
4. Same with restaurants dataset. One can not simply have geojson geometry without naming sub-object "geometry" as in geojson specification. If "location" contains geojson geometry it should be named "geometry" else user should process data first in order to be sure what is inside. But it is data processing problem that's why we use such powerful engine as mongo. OK data comes from mongo website and they are geospatial, but it doesn't mean it fits into Qgis philosophy and design just like that.
5. Saving into mongo - yes I really need this one. No time for doing it for now.
6. As fas I know latest ogr2ogr stores data in mongo some how. I didn't checked it yet. But finding how it stores geometry (WKT ? geojson? WKB+bson?...) might be crucial for understanding what today is the standard for storing geometry in mongodb and what behavior is widely expected.

[jeleniste]

@mk45 good point - IMHO best option form "mixed geometry types" is create layer for line types, layer for polygons layers etc. I test my plugin only on polygon layers, on mixed layers it would probably crash, in my data i have two geometries with different names (geometry - POLYGON and geometry_p - point).

[VasiliosKalogirou]

Thanks for replying so soon! 1: I am a beginner in Git so apologies for using a .doc for the matrix! 2: Maciej, I have developed a plugin which supports storing data. The purpose of this group is to merge all our work in one complete plugin that will support all the capabilities between QGIS and MongoDB. Please have a look at Paolo's comments on my plugin: https://github.com/VasiliosKalogirou/Save-layer-in-MongoDB/issues/2. Now personally, I like Jan's plugin interface, so I thought it would be nice to create a complete plugin with 4 dialogs: connect to the database, load data from MongoDB, query data, and store data. I can certainly contribute to the last dialog as the code is already ready. We can agree on the interface and the code for the dialogs, and start merging! About the neighborhoods and restaurants json files, I will have a look again. We will certainly provide a template of how geospatial data will be stored in MongoDB, here, on the merged plugin's GitHub repository.

[jeleniste]

My main interest on this is work with non-relational fields in qgis. My interface is created in Qtdesigner, so is possible to open it in this and add another card, or change layout, edit controls etc. About functionality there si a one question about displaying, if we want memoriza a whole collection or only map view (i have option to filter only features in map view, but it dont reload layer on pan).

[mk45]

Ah Yes! The goals. My plugin was helper only It's interface I wish to be as easy as 1,2,3,clicks and added. I work with modeling spatial data and I want to add them quickly from database where they are (let's say) processed to the environment where they are displayed. I didn't want my plugin to filter/trim any data because that was (and I believe should be done) as part of data preparation/creation/analysis process. This process is the reason why I chose mongo over SQL. As when I started project there were no mongo-qgis plugins at all so I figured something that was for me ok. I believe that the best part of my plugin is initialization process which is better than in an example. I'll be glad to contribute to create the best universal qgis-mongo plugin.

[jeleniste]

@mk45 What you mean "initialization process" Btw, best part of my plugin is icon, other things are pymongo and qtdesigner. My goals was probably similar, i was testing potential of nosql databases on spatial data and I needed some tool for visualisation output of some query.

[mk45]

Each menu action/entry is listed that causes no additional troubles during loading/unloading/reloading plugin. Separating action logic among classes is easier (other than in example). more here: https://github.com/mk45/MongoConnector/blob/master/plugin.py and two first lines of init here: https://github.com/mk45/MongoConnector/blob/master/connect_action.py

Yes my icon is rubbish

[VasiliosKalogirou]

Should we start by building the interface with 4 separate dialogs?

[jeleniste]

@VasiliosKalogirou

1. connect (host, port, auth, database)
2. display (collection, "geometry column", srid??, nested columns sanitization, bounding box)
3. save (layer/file, nested fields sanitization, SRID, collection, update/append, primary key/object id)
4. query (query) ?

[jeleniste]

2

[VasiliosKalogirou]

Looks great to me @jeleniste !!