Create Java security manager for forcible asserting behaviours in testing [LUCENE-4337]

[asfimport]

Following on from conversations about mutation testing, there is an interest in building a Java security manager that is able to assert / guarantee certain behaviours

---

Migrated from LUCENE-4337 by Greg Bowyer (@GregBowyer), resolved Aug 30 2012 Attachments: ChrootSecurityManager.java, ChrootSecurityManagerTest.java, LUCENE-4337.patch (versions: 3), LUCENE-4337-not-working.patch Linked issues:

• 5399

  • SOLR-4195

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

I checked already: We dont really need an own SecurityManager implementation at all. All this can be done from command line executing the tests: http://docs.oracle.com/javase/6/docs/technotes/guides/security/PolicyFiles.html

We just need to create a PolicyFile that allows READ,EXECUTE for all files, but READ,WRITE,EXECUTE,DELETE for all files in the working copy (using a FilePermission). This policy file can be passed to the underlying Test JVM with -Dargs=... and we are done. I will later do some tests to try out. No need to write any line of code.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Very simple patch:

This adds some sysprops to the test runner to throw a SecurityException, when a test tries the following:

• Create, delete or write to files outside ${build.dir}. This is mostly working, only some Solr tests directly start the example server without cloning it before, causing that the initial index is created! This should never happen, as it modifies checkout outside build.dir - Example: TestMultiCoreConfBootstrap
• Open ports on any other address than 127.0.0.1 (e.g., TestJmxMonitoredMap)

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

And a 3rd thing:

• SecurityPermission was not granted, but TIKA violates it: It reconfigures the security framework of Java (BouncyCastle does this). This is a no-go for a library, especially not thread safe (it changes a property shared by other threads). Not sure how to fix!

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Patch is really simple and nice.

Maybe we should think about adding exemptions for the known problems and get this in to prevent future problems?

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Attached is a patch that works around some problems:

• the JMX problem: Unfortunately this is all messy, as Java's RMI servers by default always bind to 0.0.0.0 and use their own hostname to find the IP address when creating bean proxies (see http://weblogs.java.net/blog/emcmanus/archive/2006/12/multihomed_comp.html, there is also a JDK bug open about that). The workaround was to use a SocketFactory and a sysprop. The good thing of this fix is that now my Windows Box does not always pop up a Firewall Requester when TestJmxMonitoredMap is running...

• The TIKA Problem with BouncyCastle was solved by adding another permission

The remaining issue is only one test spawning the MultiCore example server and writing to files inside the example server, which they should not do in tests. My idea would be to copy the example server to a tmpDir and start it from there?

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Now also the TestMultiCoreConfBootstrap test was fixed. The problem was that it (unlike other tests) missed to redirect the started server's dataDirs.

Now all tests pass with the policy file, a final run is going on, but I think it's ready to commit.

I will commit this tomorrow to keep an eye on Jenkins, because I don't trust 3rd party JVMs and SecurityManagers...

[asfimport]

Greg Bowyer (@GregBowyer) (migrated from JIRA)

I have a slightly different approach that I was testing last night that programatically creates the policy so that we dont have to allow other permissions all the time

[asfimport]

Greg Bowyer (@GregBowyer) (migrated from JIRA)

Gregs alternative approach (ChrootSecurityManager) (untested - just for discussion); not that I dont think we can use the approach Uwe worked on, just that I am not sure with going down the strict policy route

[asfimport]

Greg Bowyer (@GregBowyer) (migrated from JIRA)

Actually you know what, I think I just talked myself out of my approach, yes its a bit more generic but maybe its less transparent to people that dont spend their time reading JVM source code, at least the policy file is more obvious to those who are approaching this for the first time.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

The approach is also good - and can also be configured via sysprops! Funny: you also added readlink permission, but i left that out as we are on Java 6.

The good thing with being explicit is my experience from today: because today I found lots of places where e.g. Solr was opening ports on 0.0.0.0 instead of localhost(127.0.0.1)-only which is bad for tests (I was always clicking on Windows-Firewall to allow that, krrrrr). So we should limit the tests as much as possible - a strict policy rule is the best. It's also a good test of Lucene and Solr if it works with a real J2EE security manager...

If we add new features, the policy file is edited quickly. If later JDK versions add new Permissions, its also no problem, because we won't use them at the moment.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Actually you know what, I think I just talked myself out of my approach, yes its a bit more generic but maybe its less transparent to people that dont spend their time reading JVM source code, at least the policy file is more obvious to those who are approaching this for the first time.

I also spend my time in reading JDK source code today! But it was more when fixing Solr test violations (die, JMX/RMI, die, die, die).

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

This reminds me of fixing the contrib/remote test cases! I like this approach as a start (and the test fixes). Maybe we can iterate from here?

Perhaps it will clear up some false test fails.

[asfimport]

Greg Bowyer (@GregBowyer) (migrated from JIRA)

I also spend my time in reading JDK source code today! But it was more when fixing Solr test violations (die, JMX/RMI, die, die, die).

Right there with you on RMI

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Greg, I have a good idea, that prevents us from really specifying everything in the policy file: I would only make the AllOtherPermission available in test-framework and make it take a list of java class names (comma separated) as parameter. Then it can be added to the default policy file like:

permission org.apache.lucene.utils.AllOtherPermission "java.io.FilePermission,java.net.SocketPermission,java.security.SecurityPermission";

This would grant (like yours) all other permissions, so only the listed permissions are added to the policy file. The implies() would only check for all those classes.

I will come up with a new patch, that simpliefies this!

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

I tried that out but failed to do it, because the policy file was not able to load the custom Permission at all (it must be signed maybe). It did not even print a System.err inside a static {} block in the class!

I also think the current explicit permissions are much better to manage and we know what our code needs (as we have to enable additional permissions when new features are added).

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

I will commit the attached patch now to trunk and 4.x.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Committed trunk revision: 1378914 Committed 4.x revision: 1378915

Thanks Greg for the good idea and some help + hints :-)

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Just to conclude some problems we had and why I fixed it, also for ASF Jenkins:

1. The security policy does not prevent java classes from LISTEN on any ip adresses. Policys only check the port number. So if you bind to 0.0.0.0:123 or 127.0.0.1:123 or a.b.c.d:123 makes no difference if port 123 is allowed by security policy.
2. The security policy as we have it does not grant ACCEPT on any different address than localhost. This means although all Solr servers opening public ports on 0.0.0.0 while running tests, the Java 'Firewall' (LOL) protects you by only allowing connections from localhost not external IPs
3. The policy pattern to do this is defaulting by JDK's to "localhost:1024-", which means it may accept connections from localhost on remote ports >= 1024 (as client, means remote socket address as seen from acceptor) and it may listen only on ports >= 1024. This works perfectly unless you have a broken network configuration, so we added additional policies:

• As localhost may point not necessarily to 127.0.0.1 or ::1 (could be 127.0.0.2 or the public IP, some /etc/hosts are wrong in regard to this, sometimes windows), but Solr clients in tests always use "127.0.0.1" (hardcoded) as target address, the connection would be disallowed. So I added in addition to "localhost:1024-" we added "127.0.0.1:1024-" and "[::1]:1024-" as ACCEPT granted.
• On the ASF Jenkins server we had another address problem, see: http://freebsd.1045724.n5.nabble.com/jail-external-and-localhost-distinction-td3967320.html. To explain: In FreeBSD Jails the localhost interface does not exist inside Jails, its emulated by the kernel of the main OS by mapping it to the real-world global IP. This causes that connections to localhost appear to the server you are connecting to like they would come from the public outside IP, but the security policy wants localhost. As ASF Jenkins is the only Jail we are working with and this setup is broken (a lot of software does not like that), I added another grant entry just for the public IP of ASF Jenkins so it accepts connections from this IP.

Finally Robert fixed the recent Jenkins issue caused by the security filesystem sandbox, so the clover tests were not able to write the clover.db file outside the sandbox. Thanks Robert!

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Just for completeness: I add here my ideas to create an AllButPermission (a Permission that grants everything, except a dynamically configureable list of Permission classes). The class is quite big and the small improvement of the policy file does not really make this a good alternative (although it would be cool). Unfortunately the patch works if you use the Permission directly in Java code, but from the policy file it is simply ignored. From my studies it looks like a Permission class must be in a signed code base to be used by the policy parser (like rt.jar).

Maybe Greg, you can use that as an improvement to your class.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Closed after release.