Updated wms filter (lost from contrib)

[wiibaa]

I recently found out that the wms/wmts plugins were not migrated from contrib to own repo. I bundled them together in https://github.com/wiibaa/logstash-filter-wms and updated them to 1.5 for your review. This plugin contains the original commits made to contrib + my corrections + https://github.com/elastic/logstash-contrib/pull/107

[purbon]

@wiibaa I apologies for the confusion, are you willing to have this under logstash-plugins?

[wiibaa]

This would be better IMHO, just a question my repository bundle together two similar filter wms and wmsts, is it acceptable to keep them together as they are very similar or split them ??

[purbon]

My inclination is to split them, what do you think? I mean they are similar, but actually tracking different things, right?

[wiibaa]

I'm not using them myself. Looking at https://en.wikipedia.org/wiki/Web_Map_Tile_Service, they are 2 different implementation, so I agree to split them.

[mfournier]

These plugins allow to extract geographical coordinates and other infos out of URLs adhering to the WMS and WMTS standards. They are indeed different, as the standards are different.

Splitting them in 2 is probably what users would expect: bin/plugin install foo will allow you to add filter { foo { ... }} to the configuration.

[purbon]

yeah! I agree, @wiibaa @mfournier I will create to repos in the logstash-plugins org, and you guys can actually do an initial PR there, ok?

[wiibaa]

@purbon I have cleanup /squashed the code and split in https://github.com/wiibaa/logstash-filter-wms and https://github.com/wiibaa/logstash-filter-wmts, do you think I could simply transfer ownership to logstash-plugins, or this Github feature does not work as I suppose ?

Additionally, doing a fresh bundle install and running spec I got an error due to grok and must add the following in my own spec https://github.com/wiibaa/logstash-filter-wmts/blob/master/spec/filters/wmts_spec.rb#L5-L12 , thoughts?

[jratike80]

I wonder if it will be possible to install this wms plugin with the plugin installer some day. I have tried to build and install the plugin by following the generic instructions in https://github.com/wiibaa/logstash-filter-wms with poor success. Alternatively a step-by-step tutorial for Ruby dummies would be nice. I have managed to get promising messages like "Successfully built RubyGem Name: logstash-filter-wms Version: 0.0.1 File: logstash-filter-wms-0.0.1.gem" and "Installing logstash-filter-wms Installation successful" but despite the success running "plugin list" does not list filter-wms.

UPDATE: After repeating the installation as described in section 2.1 one more time it was now successful. Had to rise the logstash version for v. 2.0 into "logstash-core", '>= 1.4.0', '< 2.1.0' and s.files = git ls-files.split($) did not work for me but I used s.files line from another plugin.

[wiibaa]

@jratike80 You are right the plugin is lagging behind, I will update it and ping you when I manage to put it on rubygem so it could be installed with the plugin manager. Also thanks for trying out this plugin, please report any additional success / failure / question you encounter. Cheers

[jrahkonen]

The plugin does work and is writes WMS parameters as separate fields into the index which is a good start. I can't find Logstash 1.4 documentation anywhere so if this plugin had documentation then it is more or less lost. But perhaps there has never been so much. Initially it came perhaps from camptocamp http://www.camptocamp.com/en/actualite/logstash-devient-spatial/ but I suppose they do not maintain it here.

If I could I would map some fields which are now strings into integers: width, height, i, j, x, y, updatesequence, feature_count. At least width and height, and probably also calculate a new field "size" as [width] x [height]. And then it would be good to add an alternative to store BBOX as geo_shape piolygons which are defined in https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-shape.html

{ "location" : { "type" : "polygon", "coordinates" : [ [ [100.0, 0.0], [101.0, 0.0], [101.0, 1.0], [100.0, 1.0], [100.0, 0.0] ] ] } }

If output_epsg is using the default that is set in "config :output_epsg, :validate => :string, :default => 'epsg:4326'" the the corners of the EPSG:4326 polygon can be captured from

event["[#{@target}][output_bbox]"] = { "minx" => bbox[0], "miny" => bbox[1], "maxx" => bbox[2], "maxy" => bbox[3] }

If my sketching was correc, the result should be { "location" : { "type" : "polygon", "coordinates" : [ [ [bbox[0],bbox[1]], [bbox[2],bbox[1]], [bbox[2],bbox[3]], [bbox[0],bbox[3]], [bbox[0],bbox[1]] ] ] } }

If output_epsg is not 4326 but geo_shape should still be saved then the reprojecting code should be duplicated to handle the original BBOX, reprojected BBOX into output_bbox projection, and reprojected BBOX that is always in EPSG:4326 and that would be used for populating the geo_shape.

The order of coordinates must be longitude, latitude. Hopefully the geoscript library could do it right when it is converting the native BBOX into projected BBOX, taking into account that order of coordinates in the original WMS GetMap request may be different in WMS 1.1.x and WMS 1.3.0. The current reprojection code does not analyze the WMS version and the possible need for flipping the coordinate axis. That would be a nasty thing to implement.

[wiibaa]

@jratike80 the plugin is now on rubygem for Logstash 2.x, so you could install it with less effort @jrahkonen thanks for the interesting suggestion, I will put this on my todo list !

[jrahkonen]

@wiibaa, how about putting https://github.com/wiibaa/logstash-filter-wmts as well into rubygem?