FINRAOS / herd

Herd is a managed data lake for the cloud. The Herd unified data catalog helps separate storage from compute in the cloud. Manage petabytes of data and make it accessible for data processing and analytical purposes by any cloud compute platform.
http://finraos.github.io/herd/
Apache License 2.0
135 stars 41 forks source link

ElasticSearch index activation #373

Open namebrandon opened 6 years ago

namebrandon commented 6 years ago

The following is the response from every Curl call to ES.. It appears that this is an ES issue?

This is output from my local machine (OSX) curling to the Herd instance created by CloudFormation (artifact jar 0.65).


 curl -X POST \
>   --header 'Content-Type: application/json' \
>   --header 'Accept: application/json' \
>   -d '{"searchIndexType": "BUS_OBJCT_DFNTN"}' \
>   http://myip:8080/herd-app/rest/searchIndexes

{"statusCode":400,"statusDescription":"Bad Request","message":"unable to detect content type from source []","messageDetails":[]}

I have tried the same from Postman and see the same response.

DavidBalash commented 6 years ago

I have added some more information to the Elasticsearch setup page on the herd wiki: https://github.com/FINRAOS/herd/wiki/Elasticsearch-setup and the information provided may help in solving your issue. Summary below:

We are currently using Elasticsearch version 5.1.1, please make sure that is the version that you have installed. This version number corresponds to the org.elasticsearch dependency artifact and the JEST client dependency artifact that the herd code uses to connect to Elasticsearch. Elasticsearch version 5.2 and later contains breaking changes https://www.elastic.co/guide/en/elasticsearch/reference/5.5/release-notes-5.2.0.html.

Important: Please install an Elasticsearch software version corresponding to the Elasticsearch library version in the Herd pom.xml file. The version number can be found in the <elasticsearch.version> property in the top level pom.xml (https://github.com/FINRAOS/herd/blob/master/pom.xml) file in the herd repository.

Also make sure that the configuration values elasticsearch.bdef.mappings.json, elasticsearch.bdef.settings.json, elasticsearch.tag.mappings.json, and elasticsearch.tag.settings.json are set as CLOB values in the configuration table (cnfgn), cnfgn_value_cl column.

namebrandon commented 6 years ago

Thanks, David! I'm just using the out of the box CloudFormation template, not sure if there's an ability to control the ES version? I think Nate had mentioned this was being updated, with some more changes to come.

DavidBalash commented 6 years ago

The CloadFormation template does a yum install of ES version 5.x, which ends up installing ES 5.6.8. We will need to update the CF template so that it installs 5.1.1. For now as a work around, you could login to the EC2 instance and re-install ES under /usr/share/elasticsearch/, or modify the CF template and rebuild the stack.

dmoore247 commented 6 years ago

The 0.68 version of the CloudFormation template installs the correct version of the Elastic Search (5.1.1), and it creates the two indexes bdef_###### and tag_##### however it fails to activate those indexes within Herd. The Elastic Search Setup page shows how to activate the indexes. To discover the existing indexes. this command is valuable: curl --header 'Accept: application/json' -X GET http://localhost:8080/herd-app/rest/searchIndexes |jq Once this was done, I was able to navigate the UI rather successfully.