search

logstash-plugins / logstash-output-elastic_app_search

Apache License 2.0
0 stars 9 forks source link

Add Workplace Search output #26

Closed davishmcclurg closed 3 years ago

davishmcclurg commented 3 years ago

This is mostly copied from the app search output with some changes for the API and client differences.

I tried to switch to the shared client but that resulted in a dependency conflict on elasticsearch-transport.

The integration spec is using internal Workplace Search APIs because we don't currently expose enough to make it work with public APIs (can't get access token or content source ID). Hopefully this will change in the near future.

I'm not sure if this should be a separate plugin or be combined with app search like it is here. If they're combined, we probably want to rename the plugin elastic_enterprise_search, but I'm not sure what that would take.

Related: https://github.com/elastic/workplace-search-team/issues/1629

jonasll commented 3 years ago

Currently gathering more info from Logstash team and setting up a conversation to ensure alignment. Stay posted.

jonasll commented 3 years ago

Following up on the above - anything we can help with on the Enterprise Search team to help with the appropriate strategy?

andsel commented 3 years ago

@jonasll I've created the integration plugin https://github.com/logstash-plugins/logstash-integration-elastic_enterprise_search that aligns to Logstash AppSearch output 1.2.0. So this check the first point in the previous list.

For the second point I've this PR https://github.com/logstash-plugins/logstash-integration-elastic_enterprise_search/pull/3 that introduces the dependency on elastic-enterprise-search 7.12 but can be merged only after public release of Logstash 7.14.

In this context if you want could start a new PR, based on the https://github.com/logstash-plugins/logstash-integration-elastic_enterprise_search/pull/3, to introduce the Workplace Search output.

jonasll commented 3 years ago

Sorry for missing this, @andsel - and thank you for wrangling all of this together.

So I'm clear, we'll get the benefits of a combined Logstash plugin comes 7.14, but for the time being, should we want to release a Workplace Search plugin in the current window (i.e. until 7.14 public), it would have to be its own thing for now. Am I summarizing the above accurately?

I'm not familiar with the nuance of plugin types, thus my question.

andsel commented 3 years ago

@jonasll with

it would have to be its own thing for now.

do you mean to release separate plugin for the workplace search something like logstash-output-workplace_search?

andsel commented 3 years ago

Actually we have this new integration plugin logstash-interation-elastic_enterprise_search 2.0.0 that's aligned to logstash-output-elastic_app_search 1.2.0, we could think to move only the code related to the workplace search, from this repository to that repository. In this case the code leverage the old "elastic-workplace-search", '~>0.4.1' which doesn't have any dependency on elasticsearch-transport gen, and release a new version of logstash-interation-elastic_enterprise_search with this new output.

Then when Logstash 7.14 is out, we could update the dependencies and switch to the enterprise-search gem.

davishmcclurg commented 3 years ago

@andsel I copied the workplace search output (without renaming anything) from this PR to the new integration plugin: https://github.com/logstash-plugins/logstash-integration-elastic_enterprise_search/pull/4

If we merge it there, is it possible to get it out before 7.14?

andsel commented 3 years ago

@davishmcclurg thanks for it, it requires some touches but it's ok and is absolutely possible to have it out before 7.14, comments on PR

andsel commented 3 years ago

Close this because the work landed in logstash-integration-elastic_enterprise_search plugin https://github.com/logstash-plugins/logstash-integration-elastic_enterprise_search/pull/4