search

NLPchina / elasticsearch-sql

Use SQL to query Elasticsearch
Apache License 2.0
6.99k stars 1.54k forks source link

transportclient可以切换成resthighlevelclient吗？ #1130

Open kinbod opened 3 years ago

kinbod commented 3 years ago

transportclient可以切换成resthighlevelclient吗？

shi-yuan commented 3 years ago

请问，目前的使用方式是？ 1、未安装插件，直接引用jar包，使用SearchDao 2、未安装插件，使用jdbc 3、已安装插件，使用rest api

cccccccAi commented 3 years ago

@shi-yuan 网络环境限制长连接只能保持4min，现在项目查询等操作都是走es-sql，插入已经修改为JestClient.

（目前es版本5.5.6 ，es-sql 5.5.1 并且不能做升级）

目前可以用什么方案解决长连接断开后，es-sql查询不到ES数据的问题，还请指导一下： 1、修改为 SearchDao 来做查询操作 2、修改成JDBC的方式，保障长连接断开后及时重连

shi-yuan commented 3 years ago

1、如果是httpclient，可以借助keep-alive和HttpRequestRetryHandler

2、如果是TransportClient，定时器默认每隔5秒，调度NodeSampler（根据配置client.transport.sniff确定实例化SniffNodesSampler还是SimpleNodeSampler）的sample方法，方法内部会同node建立连接

zwwlyvv commented 2 years ago

所以能和highlevel一起用吗？

zwwlyvv commented 2 years ago

@shi-yuan 3、已安装插件，使用rest api

shi-yuan commented 2 years ago

这种方式问题不大。 如果是未安装插件，纯粹只在客户端使用rest api，目前是不支持的

@shi-yuan 3、已安装插件，使用rest api

zwwlyvv commented 2 years ago

这种方式有例子吗？

shi-yuan commented 2 years ago 
POST _nlpcn/sql
{
    "sql":"select * from myindex"
}
POST _nlpcn/sql/explain
{
    "sql":"select * from myindex"
}
zwwlyvv commented 2 years ago

@shi-yuan 我指的是resthighlevelclient，通过java来对es进行查询，我看过用了elasticsearch-sql插件的例子里面使用的客户端是transportclient客户端，现在想把transportclient更改成resthighlevel客户端，不知道可不可以这样改。版本6.1.4

shi-yuan commented 2 years ago

目前不支持直接替换的哈

zwwlyvv commented 2 years ago

目前不支持直接替换的哈

@shi-yuan 嗨喽，万分感谢回复，我用这种方法实现了，借助一个空的TransportClient完成SQL的请求构建，然后将请求转换成RestHighLevelClient的请求，最后得到response。

shi-yuan commented 2 years ago

也可以，不过某些转换是不支持的。

其实可以不用构建空的，直接传null，进去就行

SearchDao searchDao = new SearchDao(null);
String dsl = searchDao.explain("SELECT * FROM myindex").explain().explain();
System.out.println(dsl);
LiuFuyuan commented 1 year ago

目前不支持直接替换的哈

@shi-yuan 嗨喽，万分感谢回复，我用这种方法实现了，借助一个空的TransportClient完成SQL的请求构建，然后将请求转换成RestHighLevelClient的请求，最后得到response。

兄弟，我目前也遇到这个问题了，能否发一段你这里的处理方式代码参考一下，感谢

shi-yuan commented 1 year ago

目前不支持直接替换的哈

@shi-yuan 嗨喽，万分感谢回复，我用这种方法实现了，借助一个空的TransportClient完成SQL的请求构建，然后将请求转换成RestHighLevelClient的请求，最后得到response。

兄弟，我目前也遇到这个问题了，能否发一段你这里的处理方式代码参考一下，感谢

SearchDao searchDao = new SearchDao(null);
String dsl = searchDao.explain("SELECT * FROM myindex").explain().explain();
System.out.println(dsl);

SearchSourceBuilder sourceBuilder;
try (XContentParser parser = JsonXContent.jsonXContent.createParser(NamedXContentRegistry.EMPTY, LoggingDeprecationHandler.INSTANCE, dsl)) {
    sourceBuilder = SearchSourceBuilder.fromXContent(parser);
}

SearchRequest request = new SearchRequest();
request.source(sourceBuilder);
shi-yuan commented 1 year ago

最新版8.5.3使用的elasticsearch-java客户端，以http的方式，9200端口

public static void main(String[] args) throws Exception {
    SearchDao searchDao = new SearchDao(createElasticsearchClient());
    ActionResponse response = searchDao.explain("SELECT * FROM myindex").explain().get();
    System.out.println(response);
}

private static Client createElasticsearchClient() throws Exception {
    return new ElasticsearchRestClient(new ElasticsearchClient(getElasticsearchTransport(getRestClient())));
}

private static RestClient getRestClient() throws Exception {
    InetSocketAddress address = getTransportAddress().address();
    String hostPort = String.format("http://%s:%s", address.getHostString(), address.getPort());

    RestClientBuilder builder = RestClient.builder(HttpHost.create(hostPort));
    builder.setHttpClientConfigCallback(clientBuilder -> {
        RequestConfig.Builder requestConfigBuilder = RequestConfig.custom();
        requestConfigBuilder.setConnectTimeout(10 * 1000);

        int socketTimeout = 90 * 1000;
        requestConfigBuilder.setSocketTimeout(socketTimeout);
        requestConfigBuilder.setConnectionRequestTimeout(socketTimeout);
        clientBuilder.setDefaultRequestConfig(requestConfigBuilder.build());

        return clientBuilder;
    });
    return builder.build();
}

private static TransportAddress getTransportAddress() throws Exception {
    String host = System.getenv("ES_TEST_HOST");
    String port = System.getenv("ES_TEST_PORT");

    if (host == null) {
        host = "localhost";
        System.out.println("ES_TEST_HOST enviroment variable does not exist. choose default 'localhost'");
    }

    if (port == null) {
        port = "9200";
        System.out.println("ES_TEST_PORT enviroment variable does not exist. choose default '9200'");
    }

    System.out.println(String.format("Connection details: host: %s. port:%s.", host, port));
    return new TransportAddress(InetAddress.getByName(host), Integer.parseInt(port));
}

private static ElasticsearchTransport getElasticsearchTransport(RestClient restClient) {
    TransportOptions.Builder transportOptionsBuilder = new RestClientOptions(RequestOptions.DEFAULT).toBuilder();

    ContentType jsonContentType = Version.VERSION == null ? ContentType.APPLICATION_JSON
            : ContentType.create("application/vnd.elasticsearch+json",
            new BasicNameValuePair("compatible-with", String.valueOf(Version.VERSION.major())));

    Consumer<String> setHeaderIfNotPresent = header -> {
        if (transportOptionsBuilder.build().headers().stream().noneMatch((h) -> h.getKey().equalsIgnoreCase(header))) {
            transportOptionsBuilder.addHeader(header, jsonContentType.toString());
        }
    };

    setHeaderIfNotPresent.accept("Content-Type");
    setHeaderIfNotPresent.accept("Accept");

    TransportOptions transportOptionsWithHeader = transportOptionsBuilder.build();
    return new RestClientTransport(restClient, new JacksonJsonpMapper(), transportOptionsWithHeader);
}
lizhaowh commented 1 year ago

目前不支持直接替换的哈

@shi-yuan 嗨喽，万分感谢回复，我用这种方法实现了，借助一个空的TransportClient完成SQL的请求构建，然后将请求转换成RestHighLevelClient的请求，最后得到response。

兄弟，我目前也遇到这个问题了，能否发一段你这里的处理方式代码参考一下，感谢

SearchDao searchDao = new SearchDao(null);
String dsl = searchDao.explain("SELECT * FROM myindex").explain().explain();
System.out.println(dsl);

SearchSourceBuilder sourceBuilder;
try (XContentParser parser = JsonXContent.jsonXContent.createParser(NamedXContentRegistry.EMPTY, LoggingDeprecationHandler.INSTANCE, dsl)) {
    sourceBuilder = SearchSourceBuilder.fromXContent(parser);
}

SearchRequest request = new SearchRequest();
request.source(sourceBuilder);

"{\"from\":0,\"size\":10,\"query\":{\"bool\":{\"filter\":[{\"bool\":{\"must\":[{\"nested\":{\"query\":{\"bool\":{\"must\":[{\"wildcard\":{\"productList.productName\":{\"wildcard\":\"香奈*\",\"boost\":1.0}}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"path\":\"productList\",\"ignore_unmapped\":false,\"score_mode\":\"none\",\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}}}"

这段dsl解析错误 请问什么原因

shi-yuan commented 1 year ago

目前不支持直接替换的哈

@shi-yuan 嗨喽，万分感谢回复，我用这种方法实现了，借助一个空的TransportClient完成SQL的请求构建，然后将请求转换成RestHighLevelClient的请求，最后得到response。

兄弟，我目前也遇到这个问题了，能否发一段你这里的处理方式代码参考一下，感谢

SearchDao searchDao = new SearchDao(null);
String dsl = searchDao.explain("SELECT * FROM myindex").explain().explain();
System.out.println(dsl);

SearchSourceBuilder sourceBuilder;
try (XContentParser parser = JsonXContent.jsonXContent.createParser(NamedXContentRegistry.EMPTY, LoggingDeprecationHandler.INSTANCE, dsl)) {
    sourceBuilder = SearchSourceBuilder.fromXContent(parser);
}

SearchRequest request = new SearchRequest();
request.source(sourceBuilder);

"{"from":0,"size":10,"query":{"bool":{"filter":[{"bool":{"must":[{"nested":{"query":{"bool":{"must":[{"wildcard":{"productList.productName":{"wildcard":"香奈*","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"path":"productList","ignore_unmapped":false,"score_mode":"none","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}}"

这段dsl解析错误 请问什么原因

需要注册相关QueryBuilder：

...
SearchSourceBuilder sourceBuilder;
try (XContentParser parser = JsonXContent.jsonXContent.createParser(new NamedXContentRegistry(new SearchModule(Settings.EMPTY, false, Collections.emptyList()).getNamedXContents()), LoggingDeprecationHandler.INSTANCE, dsl)) {
    sourceBuilder = SearchSourceBuilder.fromXContent(parser);
}
...
LiuFuyuan commented 1 year ago

目前不支持直接替换的哈

@shi-yuan 嗨喽，万分感谢回复，我用这种方法实现了，借助一个空的TransportClient完成SQL的请求构建，然后将请求转换成RestHighLevelClient的请求，最后得到response。

兄弟，我目前也遇到这个问题了，能否发一段你这里的处理方式代码参考一下，感谢

SearchDao searchDao = new SearchDao(null);
String dsl = searchDao.explain("SELECT * FROM myindex").explain().explain();
System.out.println(dsl);

SearchSourceBuilder sourceBuilder;
try (XContentParser parser = JsonXContent.jsonXContent.createParser(NamedXContentRegistry.EMPTY, LoggingDeprecationHandler.INSTANCE, dsl)) {
    sourceBuilder = SearchSourceBuilder.fromXContent(parser);
}

SearchRequest request = new SearchRequest();
request.source(sourceBuilder);

非常感谢，现在才看见回复 String dsl = searchDao.explain("SELECT * FROM myindex").explain().explain(); 受你之前这段代码的启发，我这边最终需要得到的是SearchResponse对象，ES是7.7的版本，经过查看本项目源码之后，做了一个强转，改造如下，给其他有需要的同学一个参考： ActionRequest request = search.explain(query).explain().request(); SearchRequest searchRequest = (SearchRequest) request; SearchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);