Issue accessing es apis

[zakhtar1998]

Hello,

I have an elasticsearch index which does not require a username or password(http://10.4.2.41:9500/hurricane_dorian_2019_test) and I have koop installed where i have set the relevant config and proxy routes. When i run the koop server it is working fine, and when i access the /es/res/info it is showing fine. However, when i access /es/rest/services/10.4.2.41/local/FeatureServer it gives a proxy error (even though i have the route specified). Can anyone let me know what the problem is here and how I can resolve this?

Koop running

Config File

Sample JSON (trying to retrieve lat and lon from location_info --> resolved_location)

Info showing fine

Proxy Error on Webpage

Koop Error

Mod Proxy File

[rgwozdz]

Looks to me like the EsClient fails to initialize because of an invalid connection string. I assume it is invalid because the userName and password is undefined. If you put a break point in elasticConnectUtil.js you might confirm that.

How common is it that an Elastic instance doesn't require and username / password? That seem strange to me. Wouldn't that mean anybody could connect and do bad things like delete or edit documents?

[zakhtar1998]

So in order to be connected to the ES Index I have to be in the organization network for which we don't need a username and password. Please see the screenshot below showing the ES connecting successfully without any credentials when I use my VPM to connect to the organization. Without the VPN, the ES will not connect.

Does that mean I have to make the ES public with the FQDN for this error to be resolved?

[atangeman]

The ES cluster is internal to their their network and is not accessible outside via NAT or proxy. Do you have a workaround for no authentication? Do they need a hostname for this cluster or does IP suffice?

[rgwozdz]

Ok, well, I guess the thing to try is changing the ES provider code so that the username and password is omitted (or maybe the whole auth property) and see if it builds the connection to ES properly.

[rgwozdz]

See https://github.com/koopjs/koop-provider-elasticsearch/blob/master/src/utils/elasticConnectUtil.js#L31-L34

[zakhtar1998]

FYI commenting out the auth proprty gave the same error, as there are many references that require the username and password in the node_modules e.g. node_modules/@elastic/elasticsearch/index.js, node_modules/@koopjs/provider-elasticsearch/src/model.js. I tried commenting out the references but kept getting new errors as there is a tight connection between alot of files. Do you have any workaround to this?

[dhatcher]

If you just put an empty string in the config for the userName and password it should all pass through fine. That's the way I run it on clusters that don't require authorization.

"userName": "",
"password": "",

[zakhtar1998]

I tried adding it to the config, but still, receive the same error for some reason.

[dhatcher]

I see a few other problems. Your URL looks incorrect, from your config it should be more like

/es/rest/services/local/myService1/FeatureServer

It looks like the field you are pointing to for geometry is also an object, not a geo_point field. So that will most likely fail as well.

[dhatcher]

You should also double check your return fields, if they are nested you need to include the full path to them, so for instance _location_info.resolvedlocation.lat instead of just lat

[zakhtar1998]

Oh I see, yes we were a bit unclear about the URL construction, especially with :host and :id. Thanks for clarifying this! With regards to the geometry, instead of geo_point, we were unsure what to place here.

[zakhtar1998]

In our case do we just remove the geometryType or should we replace it with Point?

[dhatcher]

That depends on what type of field has your location information. If it is actually a geo_point field then it should stay, if it's geo_shape then you can put in Point/Polyline/Polgyon as the type. It MUST be either geo_shape or geo_point though for elastic to allow geospatial queries.

[zakhtar1998]

How does a typical geo_point field look like in ES? Does it combine both the lat and lon together? This is because our index has them as separate, so would that mean we have to adjust our index? Also I believe geo_point is what we are looking for and not a Point because our lat and lon represent georeferenced points that should be visualized onto the map specifically the U.S region to reflect tweets from Hurricane Dorian.

This is because with the current config, the koop does not start on https://flood-insights.qcri.org/es/rest/services/local/myService1/FeatureServer as it provides the same error.

[zakhtar1998]

By adding the userName and password in the config file, it does not resolve the error. Even others tried adding userName and password in this Issue: https://github.com/koopjs/koop-provider-elasticsearch/issues/48# and I am not sure if they were able to do so successfully. @dhatcher @rgwozdz Do you have any other suggestions I can try out to get this working?

[zakhtar1998]

Good news, looks like I have got it to work partially, as I edited the config file to look like this:

And when i access https://flood-insights.qcri.org/es/rest/services/local/myService1/FeatureServer i see the following:

And these are the warnings i see:

When i try to load the feauture in ArcGIS portal (which is empty), i get the following error on the console:

Also i placed the default _id and it mentions this field is not defined in metadata.fields, how can i go about resolving this?

[zakhtar1998]

FYI the mapping is correct as i store the geo_point as a geohash, and I am not sure why Koop is showing an error. @dhatcher @rgwozdz @keithfraley do you have any idea why this is happening?

                    "resolved_location": {
                        "properties": {
                            "address": {
                                "properties": {
                                    "city": {
                                        "type": "text",
                                        "fields": {
                                            "keyword": {
                                                "type": "keyword"
                                            }
                                        }
                                    },
                                    "country": {
                                        "type": "text",
                                        "fields": {
                                            "keyword": {
                                                "type": "keyword",
                                                "ignore_above": 256
                                            }
                                        }
                                    },
                                    "country_code": {
                                        "type": "keyword"
                                    },
                                    "county": {
                                        "type": "text",
                                        "fields": {
                                            "keyword": {
                                                "type": "keyword",
                                                "ignore_above": 256
                                            }
                                        }
                                    },
                                    "neighbourhood": {
                                        "type": "text",
                                        "fields": {
                                            "keyword": {
                                                "type": "keyword",
                                                "ignore_above": 256
                                            }
                                        }
                                    },
                                    "state": {
                                        "type": "text",
                                        "fields": {
                                            "keyword": {
                                                "type": "keyword"
                                            }
                                        }
                                    }
                                }
                            },
                            "geohash": {
                                "type": "geo_point"
                            },
                            "lat": {
                                "type": "double"
                            },
                            "lon": {
                                "type": "double"
                            },
                            "query_term": {
                                "type": "text",
                                "fields": {
                                    "keyword": {
                                        "type": "keyword"
                                    }
                                }
                            },
                            "response": {
                                "type": "text",
                                "index": false
                            }
                        }
                    },

I even tried creating another test index, where I inserted several records, one geo_point as a string, array and geohash and for all of them it gives an error. This means that there is something from the Koop server end that needs to be resolved.

Also when i run koop validate i get the following: