Elasticsearch 2.0 support

[jmaitrehenry]

ES2.0 drop / rename:

• support for Java7
• package org.elasticsearch.common.cache
• package org.elasticsearch.common.netty.handler.ssl
• package org.elasticsearch.common.netty.handler.codec.http
• package org.elasticsearch.common.netty.channel
• package org.elasticsearch.common.os
• package org.elasticsearch.search.fetch.partial

[etfeet]

any idea on when 2.0 support will be available? really looking forward to this! Thanks for getting this up to date for 1.7.

Rory

[jmaitrehenry]

Sorry, I don’t know, I’m busy but I still working on it.

On Nov 19, 2015, at 1:17 PM, etfeet notifications@github.com wrote:

any idea on when 2.0 support will be available? really looking forward to this! Thanks for getting this up to date for 1.7.

Rory

— Reply to this email directly or view it on GitHub https://github.com/petaldevelopment/armor/issues/4#issuecomment-158144052.

[ld57]

Hi jmaitrehenry,

did you see major difference regarding security between elasticsearch 1.6-7 and 2.x ?

I know they dropped several integrated function, since they deployed these by using plugin control.

ld

[vyutpatti]

Hi Julien,

Thanks for sorting out the 2.0 compatibility aspects with the searchguard plugin. Do you mind sharing information on where the incompatibilities exist between ElasticSearch's 1.x and 2.x from searchguard plugin perspective?

Thanks again!

[jmaitrehenry]

For a quick status, I have the plugin running but I need to check the failing tests:

(Transport tests fail because I run a one node cluster for speedup test)

[jmaitrehenry]

I need to fixe SSL and it's done! I should be able to have a beta this weeks.

[ld57]

Genial! Thanks Jmaitrehenry!

[ld57]

@jmaitrehenry : They released elastic 2.1.0 ... they fixed major stuff inside, but no comment regarding security aspect. maybe your current build should work.

ld

[splitice]

I look forward to this, hopefully little or nothing is needed for the upgrade to 2.1 as well.

[SergeyBear]

2.1 fingercross :-)

[jmaitrehenry]

I just finish the 2.0 version, I'm creating the build and you can test it!

I will start the 2.1 version after that.

[jmaitrehenry]

You can now install the snapshot for ES 2.0: [root@es1 elasticsearch]# bin/plugin -i armor -u https://oss.sonatype.org/content/repositories/snapshots/com/petalmd/armor/2.0.0-SNAPSHOT/armor-2.0.0-20151211.011117-1.zip

It's a snapshot and may have some bugs.

[ersushantsood]

I got error while installing plugin as plugin-descriptor is mandatory for Elastic 2.0 and it is missing ./plugin install https://oss.sonatype.org/content/repositories/snapshots/com/petalmd/armor/2.0.0-SNAPSHOT/armor-2.0.0-20151211.011117-1.zip -> Installing from https://oss.sonatype.org/content/repositories/snapshots/com/petalmd/armor/2.0.0-SNAPSHOT/armor-2.0.0-20151211.011117-1.zip... Trying https://oss.sonatype.org/content/repositories/snapshots/com/petalmd/armor/2.0.0-SNAPSHOT/armor-2.0.0-20151211.011117-1.zip ... Downloading ...................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE Verifying https://oss.sonatype.org/content/repositories/snapshots/com/petalmd/armor/2.0.0-SNAPSHOT/armor-2.0.0-20151211.011117-1.zip checksums if available ... Downloading .DONE ERROR: Could not find plugin descriptor 'plugin-descriptor.properties' in plugin zip

[jmaitrehenry]

I try to fix a failing test on Travis and I will make a new build with the plugin-descriptor.

[ersushantsood]

Thanks for the quick response , i was just referring the source code , you replaced org.elasticsearch.common.netty with classes of netty3.10 , is every test worked fine after using netty3.10 instead of elasticsearch dependency?

[jmaitrehenry]

@ersushantsood Elasticsearch use netty3.10 with the version 2.0 and I use the netty dependency as define on elasticsearch.

In the plugin code, we change org.elasticsearch.common.netty for org.jboss.netty because, for the 2.0 version, ES stop use shading or relocation.

[ersushantsood]

@jmaitrehenry Thanks for the response , I just referred the Breaking Changes page of Elasticsearch 2.0 and found they discontinued shading and relocation . Please let me know once plugin-descriptor issue is resolved . Thanks for the great work for extending the support of Search guard to latest versions of ElasticSearch

[ersushantsood]

Hi @jmaitrehenry I debugged the issue and found out that plugin-descriptor.properties need to be bundled in zip and outsize your plugin jar then plugin gets installed . Right now descriptor file is not available in your distribution so you need to change the maven assembly to bundle plugin-descriptor in the zip .

[ersushantsood]

@jmaitrehenry I am getting below error as classloader is not able to load the below class .Did you get this issue at your build time

:[2015-12-14 01:03:23,770][ERROR][com.petalmd.armor.ArmorPlugin] Class enhancements for DLS/FLS not successful due to javassist.CannotCompileException: [source error] no such class: com.petalmd.armor.filter.level.SearchContextCallback javassist.CannotCompileException: [source error] no such class: com.petalmd.armor.filter.level.SearchContextCallback at javassist.CtField.make(CtField.java:167) at com.petalmd.armor.ArmorPlugin.(ArmorPlugin.java:88) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:389) at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:349) at org.elasticsearch.plugins.PluginsService.(PluginsService.java:109) at org.elasticsearch.node.Node.(Node.java:148) at org.elasticsearch.node.Node.(Node.java:129) at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145) at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178) at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35) Caused by: compile error: no such class: com.petalmd.armor.filter.level.SearchContextCallback at javassist.compiler.MemberResolver.searchImports(MemberResolver.java:469) at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:413) at javassist.compiler.MemberResolver.lookupClassByJvmName(MemberResolver.java:320) at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:337) at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:324) at javassist.compiler.Javac.compileField(Javac.java:133) at javassist.compiler.Javac.compile(Javac.java:93) at javassist.CtField.make(CtField.java:162)

[ld57]

@jmaitrehenry and @ersushantsood , thanks for your support for continuing/contributing dev on 2.0 and 2.1 ! I keep following petal armor version instead of search-guard, then do not stop working on it :)

I use MS windows edition of elastic(1.7.0)-kibana (3 and 4)-logstash(1.5.4) in cluster mode, using transport mode, and I expect switching on elastic(2.1.0)-kibana(3 and 4)-logstash(2.1.1) cluster in end of December-start of January.

I will be able to give a feedback on armor compatibility in a windows environment using transport node.

[jmaitrehenry]

I have a build that fix the plugin-descriptor.roperties, but ES2.0 add a check about dependencies of plugin and I fail on it. ERROR: java.lang.IllegalStateException: jar hell! class: com.sun.jna.AltCallingConvention jar1: /usr/share/elasticsearch/lib/jna-4.1.0.jar jar2: /tmp/5639933706592727895/temp_name57538323/jna-4.1.0.jar

Or, if I exclude this jar: [2015-12-11 10:27:29,986][ERROR][bootstrap ] Exception com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: com/sun/jna/platform/win32/Win32Exception

On Dec 14, 2015, at 4:58 AM, ld57 notifications@github.com wrote:

@jmaitrehenry https://github.com/jmaitrehenry and @ersushantsood https://github.com/ersushantsood , thanks for your support for continuing/contributing dev on 2.0 and 2.1 ! I keep following petal armor version instead of search-guard, then do not stop working on it :)

I use MS windows edition of elastic(1.7.0)-kibana (3 and 4)-logstash(1.5.4) in cluster mode, using transport mode, and I expect switching on elastic(2.1.0)-kibana(3 and 4)-logstash(2.1.1) cluster in end of December-start of January.

I will be able to give a feedback on armor compatibility in a windows environment using transport node.

— Reply to this email directly or view it on GitHub https://github.com/petaldevelopment/armor/issues/4#issuecomment-164395208.

[ersushantsood]

@jmaitrehenry I fixed this issue by removing jna jar from plugin and mentioned plugins jars explicitly in class path of elasticsearch in elasticsearch.in.sh file present in bin folder.

[ersushantsood]

There are certain java security Manager fixes need to be done in ArmorService.java as ES 2.0 enforces security manager on any code running in the ES context.

[jmaitrehenry]

Ok, I have a build I could install on ES2.0 and no error on startup. I do not made more tests for now, but if you want to test it, you could install it:

$ bin/plugin install https://oss.sonatype.org/content/repositories/snapshots/com/petalmd/armor/2.0.0-SNAPSHOT/armor-2.0.0-20151215.030126-10.zip

[ersushantsood]

Thanks @jmaitrehenry ,actually ES2.1 brought Security manager using ESPolicy.java in Elasticsearch which gets executed during bootstrap which breaks the creation of armor_node_key.key due to AccessController .Only option I could find is to disable SecurityManager for the time being using security.manager.enabled: false in elasticsearch.yml . I tried AccessController also from Java.security but ESPolicy.java was enforcing strong restrictions.

[ersushantsood]

It is running successfully with basic authentication module testing in Elasticsearch 2.1

[ersushantsood]

@jmaitrehenry I had opened a query on discuss.elastic.co and below is the response for Security manager : https://discuss.elastic.co/t/how-to-override-the-permissions-in-security-policy-of-elasticsearch-2-1/37179

[ersushantsood]

I tested the yesterday snapshot and you need to make s small fix in build process. Please remove only jna 4.1 jar and don't remove other jna system jar as the latter is not provided by elastic

[jmaitrehenry]

I don't understand how I could lost that change, my local build have jna-plateform but not jna. This build should be correct: https://oss.sonatype.org/content/repositories/snapshots/com/petalmd/armor/2.0.0-SNAPSHOT/armor-2.0.0-20151216.195914-11.zip

The assemblies file exclude *jna* unless just *:jna:jar:4.1.0

[ersushantsood]

Hi I further tested ldap scenarios and things look fine there is 1 issue I fixed locally in ArmorPlugin.java where TransportModule does not get disabled even if you set armor.enabled:false in elastic search.yml as the check is missing in TransportModule operation.

[jmaitrehenry]

Just merge in master a version for elasticsearch 2.3