[Arquillian Cube OpenShift] namespace.use.existing + env.init.enabled=true fails to create test resources

RadekKoubsky commented 7 years ago

Issue Overview

I have my existing openshift project which si empty. I want arquillian cube to deploy test resources (e.g. openshift.yml) generated by FMP during resource goal. When running integration tests, arquillian cube throws the following exception about non-existing image stream:

Running io.openshift.booster.OpenShiftIT
Initializing Session:b13b197f-9d14-49f0-a628-571ccd39cef7
Using Kubernetes at: https://api.devel.xpaas:8443/
Did not find any kubernetes configuration.
Applying additional kubernetes configuration from: file:/home/rkoubsky/QE/MSA/forks/rkoubsky/rest_springboot-tomcat/target/spring-boot-http-is.yml
Could not annotate namespace: [rkoubsky] with status: [ERROR].
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.949 sec <<< FAILURE! - in io.openshift.booster.OpenShiftIT
io.openshift.booster.OpenShiftIT  Time elapsed: 3.948 sec  <<< ERROR!
java.lang.RuntimeException: io.fabric8.kubernetes.clnt.v2_2.KubernetesClientException: Failure executing: POST at: https://api.devel.xpaas:8443/oapi/v1/namespaces/rkoubsky/imagestreams. Message: ImageStream "spring-boot-http" is invalid: []: Internal error: imagestreams "spring-boot-http" is invalid: spec.tags[latest].from.name: Invalid value: "spring-boot-http@sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d": error generating tag event: imagestreamimage "sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d" not found. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=[], message=Internal error: imagestreams "spring-boot-http" is invalid: spec.tags[latest].from.name: Invalid value: "spring-boot-http@sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d": error generating tag event: imagestreamimage "sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d" not found, reason=InternalError, additionalProperties={})], group=null, kind=ImageStream, name=spring-boot-http, retryAfterSeconds=null, additionalProperties={}), kind=Status, message=ImageStream "spring-boot-http" is invalid: []: Internal error: imagestreams "spring-boot-http" is invalid: spec.tags[latest].from.name: Invalid value: "spring-boot-http@sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d": error generating tag event: imagestreamimage "sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d" not found, metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
Caused by: io.fabric8.kubernetes.clnt.v2_2.KubernetesClientException: Failure executing: POST at: https://api.devel.xpaas:8443/oapi/v1/namespaces/rkoubsky/imagestreams. Message: ImageStream "spring-boot-http" is invalid: []: Internal error: imagestreams "spring-boot-http" is invalid: spec.tags[latest].from.name: Invalid value: "spring-boot-http@sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d": error generating tag event: imagestreamimage "sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d" not found. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=[], message=Internal error: imagestreams "spring-boot-http" is invalid: spec.tags[latest].from.name: Invalid value: "spring-boot-http@sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d": error generating tag event: imagestreamimage "sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d" not found, reason=InternalError, additionalProperties={})], group=null, kind=ImageStream, name=spring-boot-http, retryAfterSeconds=null, additionalProperties={}), kind=Status, message=ImageStream "spring-boot-http" is invalid: []: Internal error: imagestreams "spring-boot-http" is invalid: spec.tags[latest].from.name: Invalid value: "spring-boot-http@sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d": error generating tag event: imagestreamimage "sha256:ae65653b28535fac376fc62f9b4fda24206152964ca1b6884c621cc2e633698d" not found, metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).

Results :

Tests in error: 
  OpenShiftIT.io.openshift.booster.OpenShiftIT » Runtime io.fabric8.kubernetes.c...

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0

When I run:

mvn clean fabric8:deploy -Popenshift
mvn verify -Popenshift-it -Dnamespace.use.existing=my-project -Denv.init.enabled=false

It works fine.

Expected Behaviour

Integration test deploys its resources and test them

Current Behaviour

Integration test fails to deploy its resources.

Steps To Reproduce

clone this maven project
Checkout branch arquillian-cube-integration-test-openshift
run mvn clean verify -Popenshift,openshift-it -Dnamespace.use.existing=my-project -Denv.init.enabled=true

RadekKoubsky commented 7 years ago

I have tried the code with Arquillian Cube 1.8.0, the issue with image stream is still present.

lordofthejars commented 7 years ago

@RadekKoubsky I have started working on this. Any thought where the error could happen? I am asking just to not going completely blind.

lordofthejars commented 7 years ago

@RadekKoubsky I have been reading the issue and running the example and I have noticed that I am not sure if an error on Cube or if you have another scenario in mind but look when you do:

mvn clean fabric8:deploy -Popenshift
mvn verify -Popenshift-it -Dnamespace.use.existing=my-project -Denv.init.enabled=false

It works. And it works because fabric8:deploy generates the kubernetes resource and deploy it to OpenShift and then test is run, and since everything is deployed it works.

Then you say that using just thins command:

mvn clean verify -Popenshift,openshift-it -Dnamespace.use.existing=my-project -Denv.init.enabled=true

Does not work. Basically it does not work because Arquillian is not deploying the scenario, but here is my question. There is no really advantage of using first instead of second approach. In terms of CI/CD it doesn't care since it is Jenkinsfile who does everything. In terms of development in terminal, well he can create a shell script and run it from build tool (Maven), create an alias or whatever. Then there is if developer is running from IDE, in this case is where it is interesting that Arquillian could deploy the services but you still need the manual step of generating the kubernetes file, so again if you need a manual step, it is not so automatic at all.

The only thing I see is that you can generate kubernetes file with maven call then switch to IDE and run the test from there, but then not sure it is valid from UX.

Am I missing something?

Ladicek commented 7 years ago

@lordofthejars The correct one to speak to is @spisiakm, as @RadekKoubsky is leaving Red Hat.

Actually the 1st scenario, the one with 2 Maven invocations, is something we don't want. We want the second one to work, the one with a single Maven invocation.

lordofthejars commented 7 years ago

Ok great, welcome @spisiakm, but what is the benefit of the second one if you still need one manual step?

Ladicek commented 7 years ago

Sorry, not sure I'm following. Here's a somewhat more recent description of the issue: https://github.com/openshiftio/appdev-planning/issues/43#issuecomment-318640464

Basically, we need mvn clean verify to work, in an existing namespace. Are we doing something wrong?

lordofthejars commented 7 years ago

but why not mvn fabric8:deploy verify -D..... ?

Ladicek commented 7 years ago

Arquillian Cube is supposed to deploy the application, no? I'm pretty sure I'm missing something.

lordofthejars commented 7 years ago

Yes when it is necessary (basically when you want to run your application from IDE or from CLI), basically when you create manually your kubernetes.json then it has sense to do it. But in this application you are not creating any kubernetes.json manually, you are letting fabric8 plugin to do it, so it means that you need to run at least one time (until you clean the project) the maven goal to generate these resources.

What I am saying is that if you want to run your tests from Maven, you don't need that Arquillian deploys anything you can just run mvn fabric8:deploy verify -D..... and that's all. Why would you need Arquillian does anything?

The other thing which might be more useful might be an integration between fabric8 plugin and arquillian so you can run safely from IDE and from CLI without worrying about calling the plugin first.

But this is a big change overall and I would like to listen comments about it from @bartoszmajsak and @iocanel and @jwendell

lordofthejars commented 7 years ago

@Ladicek if you want we can move the conversation to BJN as well.

Ladicek commented 7 years ago

So, in the linked repo, if I do mvn clean verify -Popenshift,openshift-it -Dnamespace.use.existing=myproject -Denv.init.enabled=true, the fabric8:resource and fabric8:build goals run and the {kubernetes,openshift}.{json,yml} files are created long before the test starts. So I expect Arquillian to take them, apply them, run the test, then cleanup.

You ask: why do you want to do this? Well I ask: why not? It's simpler, it's how Maven is typically used, and if I understand the docs correctly, it should work. And it seems it would, but the image stream created by fabric8:build gets deleted for some reason. Is Arquillian possibly doing some pre-test cleanup?

Ladicek commented 7 years ago

Oh thanks for the BJN offer, but I'm not very familiar with this issue, so unless you are fine with making it a pair debugging session, it wouldn't work :-)

lordofthejars commented 7 years ago

Exactly we can do this thing you mention but you could do mvn clean fabric8:deploy verify -Popenshift,openshift-it -Dnamespace.use.existing=myproject -Denv.init.enabled=false and you'll get the same, what I mean is why do you want that Arquillian deploys something generated by Maven plugin when the Maven plugin itself can do it.

Ladicek commented 7 years ago

Well, at the very least, I don't want to force users to add more commands to the cmdline, when the traditional mvn clean verify obviously should work.

But out of curiosity, I tried mvn clean fabric8:deploy verify -Popenshift,openshift-it -Dnamespace.use.existing=myproject -Denv.init.enabled=false on the linked repo, and fabric8:deploy doesn't even get a chance to run, because tests are executed first.

lordofthejars commented 7 years ago

humm ok then tomorrow morning I will take a look what you should do to make it work and provide you the CLI option to do it.

Ladicek commented 7 years ago

Thank you!

bartoszmajsak commented 7 years ago

We have discussed potential solutions for this functionality with @lordofthejars.

Quick win

To let you guys use one mvn command to run everything we will fix Cube to scan for files generated by fabric8-maven-plugin. @lordofthejars is on it.

Desired improvement

We have two options here if we consider use case when you run the test from the IDE. We need to have those {kubernetes,openshift}.{json,yml} files generated upfront.

we can re-use logic of fabric8-maven-plugin to
- generate those files
- create s2i image and register it in openshift registry
- this can be done under the hood based on e.g. @AutogenerateOpenshiftResources annotation put on the test class
we can provide Shrinkwrap descriptors so you can either create default files like fmp does but also open it up for customization "shrinkwrap style"
- this would mean we have to flesh out what kind of @Deployment methods we would like to have to make it simple and powerful enough

Thoughts?

Ladicek commented 7 years ago

Re "Quick win": my understanding from this section of the documentation is that Arquillian Cube should already be able to find the FMP-generated files. Am I wrong here? I'd consider this a bugfix, not an improvement. Actually you're using the word "fix" too, so it seems we're on the same page :-)

Re "Desired improvement": I don't think we considered running tests from IDE just yet :-) I think all our Kubernetes/OpenShift tooling revolves around FMP, so the 1st option sounds best to me. But I don't have all the details, I might be easily missing something.

And thanks for paying attention to this, it's gonna be a huge help for us!

bartoszmajsak commented 7 years ago

Ok, let's make the "Quick win" working then and we can discuss IDE experience when you will be focusing on it. I think we have a bug here which @lordofthejars is looking at.

And thanks for paying attention to this, it's gonna be a huge help for us!

Sure, our pleasure :)

lordofthejars commented 7 years ago

We have found the root cause. Currently fabric8 plugin generate resources at target/classes/META-INF/fabric8 classpath location. The problem is that surefire plugin does not add target/classes but target/test-classes ("surefire.test.class.path" -> "/Users/alex/git/rest_springboot-tomcat/target/test-classes:/Users/alex/git/rest_springboot-tomcat/target/spring-boot-http-7-SNAPSHOT.jar:/Users/alex/.m2/repository/org/apache/tomcat/tomcat-juli/8.0.36/tomcat-juli-8.0.36.jar:..." so for Arquillian is impossible to find these files.

What we are going to do is talk with fabric8 guys to see if the plugin can copy resources in both places.

Ladicek commented 7 years ago

That sounds weird. Per http://maven.apache.org/surefire/maven-surefire-plugin/examples/configuring-classpath.html, the target/classes directory is on test classpath. Is that maybe a Surefire/Failsafe bug? Let me check.

lordofthejars commented 7 years ago

Currently you are using failsafe which should work in same way, but checking all resources loaded debugging the code we saw that the resources of target/classes are not there and also surefire.test.class.path only contains test-classes place. Then we moved the generated files there and they were found by Cube.

Ladicek commented 7 years ago

Failsafe documentation is the same in this regard: http://maven.apache.org/components/surefire/maven-failsafe-plugin/examples/configuring-classpath.html

bartoszmajsak commented 7 years ago

Documentation is apparently wrong. In the surefire.test.class.path property, passed to Failsafe Runner, what you are getting is a reference to a jar file with your production code, but not target/classes. And apparently files generated by fabric8-maven-plugin land in META-INF after this jar is produced.

Ladicek commented 7 years ago

Ah, I see! That's again the thing with Failsafe preferring to use the JAR instead of target/classes!

Ladicek commented 7 years ago

OK, so if I downgrade to Failsafe 2.18.1, which still uses target/classes, I get further. The test fails when trying to apply target/spring-boot-http-is.yml with an error message of:

imagestreams "spring-boot-http" is invalid: spec.tags[latest].from.name: Invalid value: "spring-boot-http@sha256:cf768ac1152974a20a08a34c76066a97137a1f0c29c1d57215dad183b9215b3b

This image did exist before the test started, as it was created by the Fabric8 Maven plugin. I can see it in the OpenShift web console. I don't know who deleted it -- was it Arquillian Cube?

(This finding is consistent with the issue description in https://github.com/openshiftio/appdev-planning/issues/43#issuecomment-318640464)

lordofthejars commented 7 years ago

The problem here is that fabric8 executes this -is.yml file, and then in cube we are rexecuting again and you get this failure. In fact if you use CLI you'll get the same error. Figuring out how to fix this.

Ladicek commented 7 years ago

I think you can easily apply the same file twice. The issue seems to be that FMP creates the image, the image stream and the image stream tag, then someone deletes it all, and then Cube tries to apply the -is.yml file, which tries to create the image stream and the image stream tag, but the image is missing.

Anyway, great we're on the same page! :-)

lordofthejars commented 7 years ago

exactly this is what is happening, now I have commented the code in cube that detects -is.yml files and it has been able to deploy everything. So this is a really good step forward, so I need to think a bit but maybe we will make this -is.yml detection code optional being default to false.

cc/ @bartoszmajsak @dipak-pawar

Ladicek commented 7 years ago

Great! I'm just wondering, who and why deletes the image+imagestream+imagestreamtag? That shouldn't happen, should it?

Ladicek commented 7 years ago

Also, instead of downgrading to Failsafe 2.18.1, adding this to the Failsafe plugin configuration seems to work:

<configuration>
    <additionalClasspathElements>
        <additionalClasspathElement>${project.build.outputDirectory}/META-INF/fabric8</additionalClasspathElement>
    </additionalClasspathElements>
</configuration>

lordofthejars commented 7 years ago

Good news I have got a running version of the project. I will add a new flag, and release 1.9.0 version of cube which by default will ignore the -is.yml files.

Ladicek commented 7 years ago

Cool! Any idea why the image gets deleted and applying the -is.yml file subsequently fails? I still don't quite understand why that happens.

lordofthejars commented 7 years ago

I think that it is not really delated at all, for what we have seen is that is trying to create resources twice. Look if you want to try by yourself do this. mvn clean package and then go to target directory and do oc create -f ...-is.yml and you'll get the same error.

Ladicek commented 7 years ago

Well you're right, oc create -f ...-is.yml indeed also fails with error generating tag event: imagestreamimage.image.openshift.io "sha256:469b67311d56220795dec3c7a39ef2d9f61cd9d2cc25cca99752fb1cee2a0ebb" not found. That "not found" bit fooled me.

lordofthejars commented 7 years ago

yes I think that it is more how OpenShift treat this scenario more than something related to our stuff.

Ladicek commented 7 years ago

So the reason why META-INF/fabric8 stuff couldn't be found on test classpath is this line: https://github.com/snowdrop/spring-boot-http-booster/blob/master/pom.xml#L133 Once removed, it works just fine. (Where just fine = obviously I hit the issue with -is.yml.)

EDIT: and the reason is simple. With that line, fabric8:resources runs after the JAR is created, so the META-INF/fabric8 stuff simply wasn't there when the JAR was being created.

Ladicek commented 7 years ago

I sent PRs to Spring Boot boosters to avoid binding FMP to the package phase. That should solve our issues with finding META-INF/fabric8 on the test classpath, without actually changing FMP.

lordofthejars commented 7 years ago

Ok I have updated kubernetes client on cube to latest ones, also I have fixed the problem with -is.yml files. Tomorrow I will release a new version so you can work on Monday.

Ladicek commented 7 years ago

Thanks @lordofthejars! Oh and I'm totally fine with release on Monday, we all deserve a weekend off of work!

lordofthejars commented 7 years ago

The problem is that Monday is PTO in my case, then Tue and Wed I am at JavaZone so this leave us to Thursday.

bartoszmajsak commented 7 years ago

Well I can do computers too, so I can take care of the release.

lordofthejars commented 7 years ago

When this is released @Ladicek if you see that it works, could you just ping us so we can close this issue?

bartoszmajsak commented 7 years ago

Instead of releasing and trying it out only then maybe you could build snapshot against latest master and confirm if that works for you @Ladicek?

lordofthejars commented 7 years ago

I tried locally and it worked

El 9 sept. 2017 3:16 p. m., "Bartosz Majsak" notifications@github.com escribió:

Instead of releasing and trying it out only then maybe you could build snapshot against latest master and confirm if that works for you @Ladicek https://github.com/ladicek?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/arquillian/arquillian-cube/issues/743#issuecomment-328276698, or mute the thread https://github.com/notifications/unsubscribe-auth/ABcmYSGyAhYnmTOk9GqFDkvtzDixyZiLks5sgo-TgaJpZM4OFVmn .

Ladicek commented 7 years ago

I just tried locally as well and it works. Unless it breaks your schedules or something, I'd like to check a couple more boosters before proceeding with a release. I'll comment later today or tomorrow.

bartoszmajsak commented 7 years ago

Thanks @Ladicek. Please run more booster tests and let us know so we gonna release being sure it works for you.

spisiakm commented 7 years ago

Thanks a lot @bartoszmajsak @lordofthejars for looking into this and also @Ladicek for substituting me in this case, unfortunately I wasn't available last week. I'm going try it out and see what happens. @Ladicek has already tried this out with Spring Boot http booster and it worked just fine, so thank you again, you helped us quite a bit ! :)

bartoszmajsak commented 7 years ago

Just give us go/no-go and we will proceed accordingly :)

Ladicek commented 7 years ago

Unfortunately, this isn't enough yet :-( It works flawlessly when running tests in an existing project (which is our main usecase), but running in an Arquillian-created project doesn't work (which, obviously, is needed for OpenShift.io).

mvn clean verify -Popenshift,openshift-it -Dnamespace.use.existing=myproject -Denv.init.enabled=true works
mvn clean verify -Popenshift,openshift-it -Denv.init.enabled=true doesn't work, the build gets stuck at Applying kubernetes configuration from: jar:file:/home/lthon/tmp/rest_springboot-tomcat/target/spring-boot-http-7-SNAPSHOT.jar!/META-INF/fabric8/openshift.json, no pod is running in the itest-xxxxx namespace, presumably because the image was build in another namespace?
mvn clean verify -Popenshift,openshift-it -Denv.init.enabled=true -DenableImageStreamDetection=true doesn't work, the test fails with java.lang.RuntimeException: io.fabric8.kubernetes.clnt.v2_6.KubernetesClientException: Failure executing: POST at: https://192.168.0.100:8443/oapi/v1/namespaces/itest-6785a55b/imagestreams. Message: ImageStream "spring-boot-http" is invalid, which seems to be the same error message this all started with.

Sounds like we're back to square 1?

arquillian / arquillian-cube