search

NLPchina / elasticsearch-sql

Use SQL to query Elasticsearch
Apache License 2.0
6.98k stars 1.53k forks source link

how to config the es authentication token in elasticsearch-site #383

Open jaminlai opened 7 years ago

jaminlai commented 7 years ago

Hello , I have run elasticsearch-site and visite http://localhost:8080/web/sql/, but when I click the search button by some query condition, it case some error in logfile such as:

org.elasticsearch.client.transport.TransportClientNodesService-13181 INFO [2017-02-16 17:57:38] failed to get node info for {#transport#-1}{17TpB-YTRRmB0TOxSK56yA}{xx.xx.xx.xx}{xx.xx.xx.xx:9300}, disconnecting... org.elasticsearch.transport.RemoteTransportException: [51.45][xx.xx.xx.xx:9300][cluster:monitor/nodes/liveness] Caused by: org.elasticsearch.ElasticsearchSecurityException: missing authentication token for action [cluster:monitor/nodes/liveness] at org.elasticsearch.xpack.security.support.Exceptions.authenticationError(Exceptions.java:39) ~[?:?] at org.elasticsearch.xpack.security.authc.DefaultAuthenticationFailureHandler.missingToken(DefaultAuthenticationFailureHandler.java:74) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticationService$AuditableTransportRequest.anonymousAccessDenied(AuthenticationService.java:506) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$handleNullToken$13(AuthenticationService.java:300) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.handleNullToken(AuthenticationService.java:305) ~[?:?]

ansjsun commented 7 years ago

setting elasticsearch ip and port open 'elasticsearch-site/jcoder_home/resource/ioc.js'

make suer ip and port can visit . port default transport 9300,

suggest , install it on your es machine

jaminlai commented 7 years ago

@ansjsun I have install es at the same machine, and I have config the es ip port at 'elasticsearch-site/jcoder_home/resource/ioc.js'

However in secury I have settting the authentication for access es. So if visit es by java client, I must settting the 'xpack.security.user ' key value when new PreBuiltXPackTransportClient(ESClient.SETTINGS)

But, I don't know how to config the 'xpack.security.user ' in elasticsearch-site

ansjsun commented 7 years ago

I am so sorry , currently not supported 'xpack.security.user' config, i will add it now .

jaminlai commented 7 years ago

@ansjsun Thank you so much, pls response me when finished

ansjsun commented 7 years ago

i commit it , you can take a pull ..

private void init(boolean securityOpen, String security, String... clusterNodes) {
        try {

            Builder builder = Settings.builder().put("client.transport.sniff", true);

            if (securityOpen) {
                builder.put("xpack.security.user", security);
            }

            Settings settings = builder.put("client.transport.ignore_cluster_name", true).build();

change your config like this 

var ioc = {
    esClient: {
        type: "org.nlpcn.elasticsearch.site.ESClient",
       args: [true, 'name:passowrd',"127.0.0.1:9300"],
        events: {
            depose: 'destroy'
        }
    }
};
jaminlai commented 7 years ago

@ansjsun Could I run 'git clone https://github.com/NLPchina/elasticsearch-site' directly and effective?

ansjsun commented 7 years ago

yes you can

ansjsun commented 7 years ago

you can in elasticsearch-site dir git checkout . git pull open config to setting !

jaminlai commented 7 years ago

@ansjsun

I have checkout the new code and install the plugin on the step again, but it occour the exception when searched as below:

org.elasticsearch.plugins.PluginsService-44517 INFO [2017-02-17 10:06:59] loaded plugin [org.elasticsearch.script.mustache.MustachePlugin] org.elasticsearch.plugins.PluginsService-44517 INFO [2017-02-17 10:06:59] loaded plugin [org.elasticsearch.transport.Netty3Plugin] org.elasticsearch.plugins.PluginsService-44517 INFO [2017-02-17 10:06:59] loaded plugin [org.elasticsearch.transport.Netty4Plugin] 2017-02-17 10:06:59.444:WARN:oejs.ServletHandler:qtp1927950199-88: /api/SqlApi/execute org.nutz.ioc.IocException: [esClient] # FAIL to create Ioc Bean name=[esClient] at org.nutz.ioc.impl.ObjectMakerImpl.make(ObjectMakerImpl.java:149) at org.nutz.ioc.impl.NutIoc.get(NutIoc.java:210) at org.nutz.ioc.impl.NutIoc.get(NutIoc.java:239) at org.nlpcn.jcoder.run.java.JavaRunner._instance(JavaRunner.java:186) at org.nlpcn.jcoder.run.java.JavaRunner.instance(JavaRunner.java:155) at org.nlpcn.jcoder.run.mvc.ApiUrlMappingImpl.createInvoker(ApiUrlMappingImpl.java:162) at org.nlpcn.jcoder.run.mvc.ApiUrlMappingImpl.getOrCreate(ApiUrlMappingImpl.java:142) at org.nlpcn.jcoder.run.mvc.ApiUrlMappingImpl.getOrCreate(ApiUrlMappingImpl.java:108) at org.nlpcn.jcoder.run.mvc.ApiActionHandler.handle(ApiActionHandler.java:33) at org.nlpcn.jcoder.filter.JcoderFilter._doFilter(JcoderFilter.java:80) at org.nlpcn.jcoder.filter.JcoderFilter.doFilter(JcoderFilter.java:44) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:485) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:290) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:606) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:535) at java.lang.Thread.run(Thread.java:745) Caused by: org.nutz.lang.born.BorningException: Fail to born 'org.nlpcn.elasticsearch.site.ESClient'| by args: [| @(true)| @(elastic:changeme)| @([Ljava.lang.String;@675f9b57)] becasue:|java.lang.IllegalArgumentException: unknown setting [xpack.security.user] please check that any required plugins are installed, or check the breaking changes documentation for removed settings at org.nutz.lang.born.ConstructorCastingBorning.born(ConstructorCastingBorning.java:24) at org.nutz.ioc.weaver.DefaultWeaver.born(DefaultWeaver.java:67) at org.nutz.ioc.impl.ObjectMakerImpl.make(ObjectMakerImpl.java:114) at org.nutz.ioc.impl.NutIoc.get(NutIoc.java:210) at org.nutz.ioc.impl.NutIoc.get(NutIoc.java:239) at org.nlpcn.jcoder.run.java.JavaRunner._instance(JavaRunner.java:186) at org.nlpcn.jcoder.run.java.JavaRunner.instance(JavaRunner.java:155) at org.nlpcn.jcoder.run.mvc.ApiUrlMappingImpl.createInvoker(ApiUrlMappingImpl.java:162) at org.nlpcn.jcoder.run.mvc.ApiUrlMappingImpl.getOrCreate(ApiUrlMappingImpl.java:142) at org.nlpcn.jcoder.run.mvc.ApiUrlMappingImpl.getOrCreate(ApiUrlMappingImpl.java:108) at org.nlpcn.jcoder.run.mvc.ApiActionHandler.handle(ApiActionHandler.java:33) at org.nlpcn.jcoder.filter.JcoderFilter._doFilter(JcoderFilter.java:80) at org.nlpcn.jcoder.filter.JcoderFilter.doFilter(JcoderFilter.java:44) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)

ansjsun commented 7 years ago

I fix it , you can clone or pull ,

you can try it 

git checkout .
git pull

and restart ,

open http://localhost:8080/ username:admin password:admin

click Ioc Manager

image

visit http://localhost:8080/web/sql/

image

goodluck for you

ly853602 commented 7 years ago

@ansjsun i have configured as your suggest ,but not work, and what does mean "open http://localhost:8080/ username:admin password:admin", thanks in advance

ly853602 commented 7 years ago

another question confused me,how can i add username and password when i visit web/sql in brower

chenyg0911 commented 7 years ago

http://localhost:8080/?username=elastic&password=changeme

2017-07-27 16:29 GMT+08:00 ly853602 notifications@github.com:

another question confused me,how can i add username and password when i visit web/sql in brower

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/NLPchina/elasticsearch-sql/issues/383#issuecomment-318295472, or mute the thread https://github.com/notifications/unsubscribe-auth/APRMhIVzLSyl9SuoTOrOQPpJj-qRguQTks5sSEp9gaJpZM4MC0PF .

ly853602 commented 7 years ago

default default thanks a lot

ly853602 commented 7 years ago

@chenyg0911 this is my web and configure conditions

chenyg0911 commented 7 years ago

https://github.com/NLPchina/elasticsearch-sql/issues/455

ansjsun commented 7 years ago

用户名密码需要改ioc配置。。就是再后台改。。不能再url中传入

ly853602 commented 7 years ago

改后台配置后，其它的正常访问就行了，是吗

ly853602 commented 7 years ago

default

ansjsun commented 7 years ago

是的

JackLeeee commented 5 years ago

jdbc连接es5.5版本有xpack的兼容吗，没有的话需要手动修改哪里

shi-yuan commented 5 years ago

5.x最新版是有的，可以在最新版中打包一个5.5的，修改pom.xml的elasticsearch.version为5.5.0

JackLeeee commented 5 years ago

6.8.0的jdbc连接也可以通过修改pom进行兼容吗？非常感谢

shi-yuan commented 5 years ago

6.8.0的话，就从master 6.7.1分支上改

JackLeeee commented 5 years ago

jdbc如何修改才能用到PreBuiltXPackTransportClient，现在的配置怎么试都还是普通的TransportClient，无法进行身份核验

shi-yuan commented 5 years ago

5.x、6.x的最新版，都换到了PreBuiltXPackTransportClient，用的哪个版本？

JackLeeee commented 5 years ago

用的是6.7.0.0版本 图片

错误 Exception in thread "main" java.sql.SQLException: Error at com.alibaba.druid.pool.ElasticSearchDruidDataSource.handleConnectionException(ElasticSearchDruidDataSource.java:1142) at com.alibaba.druid.pool.DruidPooledConnection.handleException(DruidPooledConnection.java:127) at com.alibaba.druid.pool.DruidPooledStatement.checkException(DruidPooledStatement.java:68) at com.alibaba.druid.pool.ElasticSearchDruidPooledPreparedStatement.executeQuery(ElasticSearchDruidPooledPreparedStatement.java:51) at test.ConfTest.main(ConfTest.java:25) Caused by: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{dUnyFraERla-Io3mc7vlSQ}{192.168.120.129}{192.168.120.129:9300}]] at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352) at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248) at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60) at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:388) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:391) at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46) at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:53) at org.nlpcn.es4sql.query.SqlElasticSearchRequestBuilder.get(SqlElasticSearchRequestBuilder.java:29) at org.elasticsearch.plugin.nlpcn.QueryActionElasticExecutor.executeSearchAction(QueryActionElasticExecutor.java:25) at org.elasticsearch.plugin.nlpcn.QueryActionElasticExecutor.executeAnyAction(QueryActionElasticExecutor.java:53) at com.alibaba.druid.pool.ElasticSearchDruidPooledPreparedStatement.getObjectResult(ElasticSearchDruidPooledPreparedStatement.java:89) at com.alibaba.druid.pool.ElasticSearchDruidPooledPreparedStatement.executeQuery(ElasticSearchDruidPooledPreparedStatement.java:40) ... 1 more

es控制台中报错： [2019-05-29T05:37:05,512][WARN ][o.e.t.TcpTransport ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/192.168.120.1:49240}], closing connection io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000055000000000000000108004d3603010d417574686f72697a6174696f6e1a4261736963205a57786863335270597a6f784d6a4d304e54593d0016696e7465726e616c3a7463702f68616e647368616b650004d3bef202 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000055000000000000000108004d3603010d417574686f72697a6174696f6e1a4261736963205a57786863335270597a6f784d6a4d304e54593d0016696e7465726e616c3a7463702f68616e647368616b650004d3bef202 at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1182) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final] ... 15 more

把Security关掉就好了，没有用到XPackClient，无法进行权限验证，应该是配置的问题，可以给个参考吗。万分感谢

shi-yuan commented 5 years ago

如果没有Security，可以不用传user：

image

JackLeeee commented 5 years ago

用到security了，必须要进行身份核验，但是这些配置不管用

shi-yuan commented 5 years ago

不管用？请问，直接用PreBuiltXPackTransportClient，能连接上，能正常search么？

shi-yuan commented 5 years ago

6.7.0.0这个版本，不应该还是TransportClient的

shi-yuan commented 5 years ago

得依赖这个包： image

JackLeeee commented 5 years ago

有的，但是配置上了用户和密码没有使用到xpack这个client。Properties没有缺配置吧

shi-yuan commented 5 years ago

没有，是在这儿构造的 image

JackLeeee commented 5 years ago

图片

ES控制台还是报这个错 [2019-05-29T06:32:33,499][WARN ][o.e.t.TcpTransport ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/192.168.120.1:52005}], closing connection io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:

是因为新版本的xpack在使用tcp连接的时候必须启用ssl/tls验证吗？那么在java客户端是不是无法通过PreBuiltXPackTransportClient去连接es了

JackLeeee commented 5 years ago

图片 官方确实这样打算了，sad

shi-yuan commented 5 years ago

看来插件支持rest迫在眉睫了， 😄

JackLeeee commented 5 years ago

6.x版本使用xpack security会强制将ssl开启，需要在代码中添加如下配置，具体代码官方给出。生成证书和私钥的方式：https://www.elastic.co/guide/en/elasticsearch/reference/6.7/configuring-tls.html#node-certificates；java客户端连接所需配置：https://www.elastic.co/guide/en/x-pack/6.2/java-clients.html 使用jdbc连接es-sql的案例：

    Properties properties = new Properties();
    properties.put("url","jdbc:elasticsearch://192.168.120.129:9300");
    properties.put("connectionProperties","client.transport.sniff=true;client.transport.ignore_cluster_name=true;xpack.security.user=elastic:123456;xpack.security.transport.ssl.enabled=true;xpack.security.transport.ssl.verification_mode=certificate;xpack.security.transport.ssl.keystore.path=D:\\elastic-certificates.p12;xpack.security.transport.ssl.truststore.path=D:\\elastic-certificates.p12");
    DruidDataSource ds = (DruidDataSource) ElasticSearchDruidDataSourceFactory.createDataSource(properties);

    DruidPooledConnection connection = ds.getConnection();

    PreparedStatement pstm = connection.prepareStatement("select * from user");
    ResultSet rs = pstm.executeQuery();

    while(rs.next()){
        System.out.println(rs.getString(1)+":"+rs.getString(2)+":"+rs.getString(3));
    }
    rs.close();
    pstm.close();
    connection.close();

其中，connectionProperties中需要的参数比较重要，需要指明用户名密码，SSL所需证书和密码库文件等配置，具体可以参考elasticsearch.yml中的相关配置。官方文档给的比较模糊

91wangmeng commented 5 years ago

@JackLeeee 你好能否提供一份详细的介绍,我现在用的是阿里云的elasticsearch,该怎么获取elastic-certificates.p12呢

JackLeeee commented 5 years ago

参考官方文档https://www.elastic.co/guide/en/elasticsearch/reference/6.8/configuring-tls.html#node-certificates