Add openjdk.test.security in systemtest

joeyleeeeeee97 commented 3 years ago

Overview

Security is an important part that contains runtime access control like SecurityManager, communication security protocols like TLS. Here I propose we add openjdk.test.security to cover this.

How is it implemented?

TLS communication tests. Personally I think this is important :)

Using java client to connect to current business HTTPS servers.
Checking java HTTPS server to accept incoming java HTTPS client connection
Listing and veri the cipher suites and protocols current runtime supports.

SecurityManager (Maybe)

Run some Illegal access to check Java SecurityManager is working

Remote login agent(Maybe)

Using current runtime to log into a remote machine

sxa commented 3 years ago

Hmm I would have expected that the openjdk suite would already have some of those but we'd have to check to be sure. We do have the external tests which test with things like tomcat, but again I'd have to check to give a definite answer on whether they explcitly test over an https connection.

smlambert commented 3 years ago

Thanks for your interest and suggestion @joeyleeeeeee97 !

If you are referring to security tests from the openjdk repository, we include them in the openjdk group already as the following test targets:

jdk_security1 jdk_security2 jdk_security3 jdk_security4 jdk_security_infra

These are run as part of the top-level extended.openjdk target which we were running on the weekend (though it appears we may not have yet re-enabled from disabling for the July release).

Example of a test job running those targets: https://ci.adoptopenjdk.net/view/Test_openjdk/job/Test_openjdk11_dragonwell_extended.openjdk_aarch64_linux/13/tapResults/. (JUnit results of same test run: https://ci.adoptopenjdk.net/view/Test_openjdk/job/Test_openjdk11_dragonwell_extended.openjdk_aarch64_linux/13/testReport/)

In the top-level sanity.openjdk target, we include the jdk_tier1_cipher target which is applicable to jdk11+.

If you are referring to other test material, can you point to it so we can consider adding it? Or perhaps you are suggesting we write some new tests. If we have a gap, then certainly that would be a good plan. Please help to take a look at what we currently include and we can see if any other material should be added.

joeyleeeeeee97 commented 3 years ago

@smlambert Thanks for the suggestions, and yes! I am thinking of adding new tests. Here is some background on this, there are some clients switching to OpenJDK and I found it hard to answer some of the questions they asked.

For example, 'I was using TLSv1.3 and on my app, does it work after migrating to OpenJDK? Please prove that." Yes, we are having some (a lot) security unit tests, but I found sometimes it's not convincing enough to list these unit tests we passed. Just like all parts are working fine doesn't mean they could work together.

So after all we write a small app that mocks what they are using, and this kind of test I think might be useful to community.

I think this is some kind of 'system test' also. We can't rewrite all user's app but I do think there is some common pattern, and we also need to generalize these kinds of questions and tests for clarity.

smlambert commented 3 years ago

Sounds good re: defining the common user stories / patterns and augmenting our system tests - @joeyleeeeeee97 !

joeyleeeeeee97 commented 3 years ago

@smlambert Hi, I am starting to work on this now and please review what I am planning to add, in README I have a brief user story and what this test will contain:

https://github.com/joeyleeeeeee97/aqa-systemtest/blob/master/openjdk.test.security/doc/README.md

My progress is about 60%.

smlambert commented 2 years ago

Ok great, thanks @joeyleeeeeee97 ! From my quick glance, it looks like a great start, I will only get a chance to look more closely after this first week of Sept release is complete.

adoptium / aqa-systemtest