[Arquillian Cube OpenShift] namespace.use.existing + env.init.enabled=true fails to create test resources

[RadekKoubsky]

Issue Overview

I have my existing openshift project which si empty. I want arquillian cube to deploy test resources (e.g. openshift.yml) generated by FMP during resource goal. When running integration tests, arquillian cube throws the following exception about non-existing image stream:

Running io.openshift.booster.OpenShiftIT
Initializing Session:b13b197f-9d14-49f0-a628-571ccd39cef7
Using Kubernetes at: https://api.devel.xpaas:8443/
Did not find any kubernetes configuration.
Applying additional kubernetes configuration from: file:/home/rkoubsky/QE/MSA/forks/rkoubsky/rest_springboot-tomcat/target/spring-boot-http-is.yml
Could not annotate namespace: [rkoubsky] with status: [ERROR].
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.949 sec <<< FAILURE! - in io.openshift.booster.OpenShiftIT
io.openshift.booster.OpenShiftIT  Time elapsed: 3.948 sec  <<< ERROR!
java.lang.RuntimeException: io.fabric8.kubernetes.clnt.v2_2.KubernetesClientException: Failure executing: POST at: https://api.devel.xpaas:8443/oapi/v1/namespaces/rkoubsky/imagestreams. Message: ImageStream "spring-boot-http" is invalid: []: Internal error: imagestreams "spring-boot-http" is invalid: spec.tags[latest].from.name: Invalid value: "spring-boot-http@sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d": error generating tag event: imagestreamimage "sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d" not found. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=[], message=Internal error: imagestreams "spring-boot-http" is invalid: spec.tags[latest].from.name: Invalid value: "spring-boot-http@sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d": error generating tag event: imagestreamimage "sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d" not found, reason=InternalError, additionalProperties={})], group=null, kind=ImageStream, name=spring-boot-http, retryAfterSeconds=null, additionalProperties={}), kind=Status, message=ImageStream "spring-boot-http" is invalid: []: Internal error: imagestreams "spring-boot-http" is invalid: spec.tags[latest].from.name: Invalid value: "spring-boot-http@sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d": error generating tag event: imagestreamimage "sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d" not found, metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
Caused by: io.fabric8.kubernetes.clnt.v2_2.KubernetesClientException: Failure executing: POST at: https://api.devel.xpaas:8443/oapi/v1/namespaces/rkoubsky/imagestreams. Message: ImageStream "spring-boot-http" is invalid: []: Internal error: imagestreams "spring-boot-http" is invalid: spec.tags[latest].from.name: Invalid value: "spring-boot-http@sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d": error generating tag event: imagestreamimage "sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d" not found. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=[], message=Internal error: imagestreams "spring-boot-http" is invalid: spec.tags[latest].from.name: Invalid value: "spring-boot-http@sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d": error generating tag event: imagestreamimage "sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d" not found, reason=InternalError, additionalProperties={})], group=null, kind=ImageStream, name=spring-boot-http, retryAfterSeconds=null, additionalProperties={}), kind=Status, message=ImageStream "spring-boot-http" is invalid: []: Internal error: imagestreams "spring-boot-http" is invalid: spec.tags[latest].from.name: Invalid value: "spring-boot-http@sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d": error generating tag event: imagestreamimage "sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d" not found, metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).

Results :

Tests in error: 
  OpenShiftIT.io.openshift.booster.OpenShiftIT » Runtime io.fabric8.kubernetes.c...

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0

When I run:

mvn clean fabric8:deploy -Popenshift
mvn verify -Popenshift-it -Dnamespace.use.existing=my-project -Denv.init.enabled=false

It works fine.

Expected Behaviour

Integration test deploys its resources and test them

Current Behaviour

Integration test fails to deploy its resources.

Steps To Reproduce

1. clone this maven project
2. Checkout branch arquillian-cube-integration-test-openshift
3. run mvn clean verify -Popenshift,openshift-it -Dnamespace.use.existing=my-project -Denv.init.enabled=true

[lordofthejars]

Case 3 is a wrong configuration. Fabric8 automatically creates image stream. So if you set cube to create image stream it fails because IS is already created.

Is the same case as if you run oc create .... After running fabric8

El 11 sept. 2017 5:21 p. m., "Ladislav Thon" notifications@github.com escribió:

Unfortunately, this isn't enough yet :-( It works flawlessly when running tests in an existing project (which is our main usecase), but running in an Arquillian-created project doesn't work (which, obviously, is needed for OpenShift.io).

1. mvn clean verify -Popenshift,openshift-it -Dnamespace.use.existing=myproject -Denv.init.enabled=true works
2. mvn clean verify -Popenshift,openshift-it -Denv.init.enabled=true doesn't work, the build gets stuck at Applying kubernetes configuration from: jar:file:/home/lthon/tmp/rest_ springboot-tomcat/target/spring-boot-http-7-SNAPSHOT. jar!/META-INF/fabric8/openshift.json, no pod is running in the itest-xxxxx namespace, presumably because the image was build in another namespace?
3. mvn clean verify -Popenshift,openshift-it -Denv.init.enabled=true -DenableImageStreamDetection=true doesn't work, the test fails with java.lang.RuntimeException: io.fabric8.kubernetes.clnt.v2_6.KubernetesClientException: Failure executing: POST at: https://192.168.0.100:8443/ oapi/v1/namespaces/itest-6785a55b/imagestreams https://192.168.0.100:8443/oapi/v1/namespaces/itest-6785a55b/imagestreams. Message: ImageStream "spring-boot-http" is invalid, which seems to be the same error message this all started with.

Sounds like we're back to square 1?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/arquillian/arquillian-cube/issues/743#issuecomment-328563777, or mute the thread https://github.com/notifications/unsubscribe-auth/ABcmYf_FZR6om4X2wsOx9BPAvQrKk9ZYks5shVAOgaJpZM4OFVmn .

[Ladicek]

Indeed Fabric8 Maven plugin creates the image stream, but in a different namespace than the itest-xxxxx. So I was thinking I need to enable adding the image stream to the itest-xxxx namespace, and that seems to work manually.

[lordofthejars]

Hummm then I will need to take a look

El 11 sept. 2017 5:44 p. m., "Ladislav Thon" notifications@github.com escribió:

Indeed Fabric8 Maven plugin creates the image stream, but in a different namespace than the itest-xxxxx. So I was thinking I need to enable adding the image stream to the itest-xxxx namespace, and that seems to work manually.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/arquillian/arquillian-cube/issues/743#issuecomment-328570804, or mute the thread https://github.com/notifications/unsubscribe-auth/ABcmYQCHQiRPPysOmw2beRCeS4mBcfySks5shVVagaJpZM4OFVmn .

[Ladicek]

BTW, here's what I did to verify the behavior in the 3rd case (all with a snapshot build of current Cube master):

mvn clean verify -Popenshift,openshift-it -Denv.init.enabled=true -DenableImageStreamDetection=true -Dnamespace.cleanup.enabled=false -Dnamespace.destroy.enabled=false
# this fails with java.lang.RuntimeException: io.fabric8.kubernetes.clnt.v2_6.KubernetesClientException: Failure executing: POST at: https://192.168.0.100:8443/oapi/v1/namespaces/itest-4d3cff0b/imagestreams. Message: ImageStream "spring-boot-http" is invalid

oc get project
oc project itest-xxxxx

oc create -f target/spring-boot-http-is.yml
# this works just fine, and actually allows the deployment created by Cube before the test to proceed

[Ladicek]

Also, I've been successful with porting a WildFly Swarm booster test to Arquillian Cube (found https://github.com/arquillian/arquillian-cube/pull/803 in the process), but I stumbled upon this roadblock: https://github.com/Ladicek/wfswarm-rest-http/blob/arquillian-cube/src/test/java/io/openshift/booster/GreetingServiceTest.java#L37 Do you possibly know of any solution? Thanks!

[lordofthejars]

Hi @Ladicek about the second and third case that it is not working, this is related on how it works Cube and fabric8. What Fabric8 Maven plugin does is create and configure some stuff inside the default OpenShift namespace (ie myproject). And this occurs before Cube test is executed. Then Cube test is executed and it creates a random namespace (it-....) but this new namespace has not the content generated by fabric8 maven plugin previously, so when the -is.yml is executed it fails because it misses some content.

The solution for that requires some big development, basically an integration between Fabric8 and Cube like explained here https://github.com/arquillian/arquillian-cube/issues/743#issuecomment-327757345 so fabric8 bits are executed after the random namespace is created by cube.

[lordofthejars]

@Ladicek I don't understand exactly the problem, but if the problem is just enable tests in failsafe and not surefire, the only thing I guess you can do is playing with inclusions/exclusions in surefire plugin and failsafe plugin cc/ @bartoszmajsak

[lordofthejars]

https://github.com/arquillian/arquillian-cube/issues/806

[Ladicek]

Hi @lordofthejars, when it comes to the -is.yml thing, I think this should easily work, because the -is.yml contains a namespace, so when importing it to itest-xxxx, the image should be found just fine. And indeed when trying it manually, it works -- as described in https://github.com/arquillian/arquillian-cube/issues/743#issuecomment-328752684

When it comes to the second problem (when Cube is used in a Surefire test run) -- ah you are actually right, I can easily exclude Cube from test classpath with Surefire! THANKS A LOT! Funny I didn't think of it myself, as I've been dealing with test classpath exclusions just a while ago :-)

[lordofthejars]

Humm I tried with @dipak-pawar and it failed using oc. I'll continue investigating on this too, because in my case it failed as well.

[Ladicek]

Hmm weird. I'll check again.

[lordofthejars]

I have talked with @rhuss today and it seems that this use case should not be working by default if you are not pointing fabric8 and cube into same namespace.

[Ladicek]

So basically the ability of Cube to create a separate namespace is known to not work when using Fabric8 Maven plugin? That's unfortunate.

[lordofthejars]

It is not exactly like that I mean it is normal that this is happening since until now I am not sure that Cube was thought to be work together with fabric8 how we are doing now. Currently Fabric8 has its own bits and Cube their own. This means that Fabric8 does some stuff when it is run with the current namespace. On the other side Cube does some stuff on configured namespace which can be the current one or a new one. If you choose a new one, since a) Fabric8 runs before Cube and b) the namespace is generated randomly, we have no way to make it work with random stuff.

Tomorrow I'll try if there is any workaround, but as I suspect we will need to implement this integration so first Cube creates the namespace and second Fabric8 does its stuff and not the other way around.

[Ladicek]

Thanks for info, @lordofthejars!

[Ladicek]

I'm thinking we should probably open a new issue for the 2nd usecase (FMP + Arquillian Cube creating a new namespace), now that the 1st one works fine (FMP + Arquillian Cube running in current namespace). WDYT?

[lordofthejars]

@Ladicek this is exactly what I wanted to talk with you on mattermost. The first thing is to try to define a priority, currently case 1 is covered and it is working, then case 2 and 3 are something that is currently widely used or something that is planed to be used (just to see if it is also a blocker, or it is something that can wait one week).

For second case I created a new issue https://github.com/arquillian/arquillian-cube/issues/806 before starting implementing it, today I'll check if there is some quick workaround.

[Ladicek]

@lordofthejars Sorry, I don't frequent Mattermost, it's like a 5th chat I'd have to figure out... but if you ping me there, I can join.

The cases 2/3 -- well we don't need them at the moment, I believe, but it will become important soon because of OpenShift.io. So waiting one or two weeks should be OK.

[bartoszmajsak]

@Ladicek can you elaborate a bit on

but it will become important soon because of OpenShift.io

So with the fix 1. you can switch from OpenshiftTestAssistant to Cube, but those test will still not run on OSIO? https://github.com/openshiftio/booster-common/issues/8

[Ladicek]

I keep hearing that we need to switch to Arquillian Cube because it allows running tests in a different namespace, which is needed on OpenShift.io: https://github.com/openshiftio/booster-common/issues/8 I don't know if that's true or not, but assuming it is, then:

1. We can switch to Cube for RHOAR boosters tests, because we run them in "current" namespace. Well I'm pretty sure we'll find issues along the way (one thing I can think of right now: can we deploy a configmap manually before Cube starts deploying resources?), but that's probably expected.
2. We still won't be able to run RHOAR boosters tests on OpenShift.io, because running in a different namespace doesn't work.

[lordofthejars]

configmap manually

yes if you do it either manually before running mvn or in maven lifecycle before integration tests.

[Ladicek]

That's exactly what I hoped I won't hear :-) But we'll figure it out when we get there.

[lordofthejars]

or you can create an openshift additional file and configure cube to pick it up, but let's wait when it is required.

[lordofthejars]

@Ladicek check latest master because the problem described in 3) should be fixed. Notice that the use case number 2, is not valid here because fabric8 with OpenShift requires an ImageStream insert on given namespace. If you don't set the enableImageStreamDetection to true then Cube never inserts the ImageStream so the test will fail because it misses one part (the Image Stream part).

So I think that now everything is fixed. Just give a try.

[lordofthejars]

More about second case notice that in case 2 you are running mvn clean verify -Popenshift,openshift-it -Denv.init.enabled=true and in case 3 mvn clean verify -Popenshift,openshift-it -Denv.init.enabled=true -DenableImageStreamDetection=true since you want to use openshift.json and this file requires an ImageStream there, if you don't set the the flag to true when deploying the openshift.json you'll get an error (as it is happening). So the rule of thumb here is that if you are using OpenShift with ImageStreams (by default in F8 Maven Plugin are used) you need to set the enableImageStreamDetection flag to true or it won't work because of missing pieces on tested namespace.

Of course this do not apply when you are using the same namespace as used in F8 Maven plugin (case 1)

[lordofthejars]

@spisiakm Have you had time to try with latest master of Arquillian Cube? In my computer the project using method 3 passed correctly.

[Ladicek]

Hi, indeed case 2 is wrong per current design and I expect it to fail. But case 3 definitely should work.

And I just verified that with current Cube master, case 3 works just fine. I think Cube currently covers the usecases we need (though I guess we'll find other issues as we progress :-) ).

@spisiakm, could you please also check and confirm? Thanks.

[spisiakm]

Hi folks, case no. 3 works like a charm now with current master, so I guess this issue is resolved/fixed. Thanks a lot !

[lordofthejars]

You are welcome, thank you very much for your feedback, since it is really valuable. Anything you need please ping us, since we are trying to give top priority.

Today I am going to release a new Arquillian Cube version so you have on maven central.

[Ladicek]

Thanks @lordofthejars, having these fixes in an actual release will be great! We realized we'll need to switch to Cube pretty soon, we'll get in touch if we hit an issue. Thanks!

[bartoszmajsak]

@Ladicek grab it while it's hot http://search.maven.org/#search|ga|1|arquillian-cube