Integration tests fail after change in devfile registry

feloy commented 2 years ago

/kind bug

Context

This PR changes metadata in Devfiles. https://github.com/redhat-developer/odo/pull/6137#issuecomment-1251953720

These changes break the integration tests

Acceptance criteria

See https://github.com/redhat-developer/odo/issues/6138#issuecomment-1270171350

[x] Spin up our own Devfile registry, that we control. Odo tests should test odo features, not that the Devfile stacks are valid/usable with odo, which should be the scope of Devfile registry tests (see: #6705)
[ ] Have one or just a few tests that validate that that odo can communicate with the default Devfile registry. They should not test other odo features, but could just check that the right Devfile has been downloaded. (see: https://github.com/redhat-developer/odo/issues/6138#issuecomment-1499090003)
[ ] Remove the NGINX server used for caching Devfiles (cf. #5922)
[x] We should not use starter projects because we might have issues there as well. Ensure that we use projects in our local code base.

See https://github.com/redhat-developer/odo/issues/6138#issuecomment-1499090003 for details on ACs completion.

rm3l commented 2 years ago

https://github.com/redhat-developer/odo/pull/6140 introduced a quick fix to unblock the current (v3.0.0-rc2) release process, but we need to find a more robust solution for such issues that can arise again at any later time.

kadel commented 2 years ago

There is also another thing. This change only broke our tests, and not odo itself. But next time there could be a change that breaks odo. We should consider running our tests against https://registry.stage.devfile.io/ every change in devfile registry gets first deployed into staging registry before going to production, we should try to catch as much as we can before changes are in production. Using staging could help with that.

rm3l commented 2 years ago

There is also another thing. This change only broke our tests, and not odo itself. But next time there could be a change that breaks odo.

That's a good point indeed.

We should consider running our tests against registry.stage.devfile.io every change in devfile registry gets first deployed into staging registry before going to production, we should try to catch as much as we can before changes are in production. Using staging could help with that.

I think we are already using the staging registry in our tests: https://github.com/redhat-developer/odo/blob/main/tests/helper/helper_generic.go#L326-L334 This is why we caught this issue today (https://github.com/devfile/registry/pull/123 was merged yesterday); I think this change has not been deployed yet to the production registry.

kadel commented 2 years ago

I think we are already using the staging registry in our tests: https://github.com/redhat-developer/odo/blob/main/tests/helper/helper_generic.go#L326-L334 This is why we caught this issue today (devfile/registry#123 was merged yesterday); I think this change has not been deployed yet to the production registry.

I forgot about this. That is good. So that is good that it broke our tests, it worked as expected :-D, it just came in the worst possible time it could.

valaparthvi commented 2 years ago

6140 introduced a quick fix to unblock the current (v3.0.0-rc2) release process, but we need to find a more robust solution for such issues that can arise again at any later time.

I remember we discussed some robust solution to this, but I can't recall what. Can you help me recall it? @rm3l @feloy

feloy commented 2 years ago

6140 introduced a quick fix to unblock the current (v3.0.0-rc2) release process, but we need to find a more robust solution for such issues that can arise again at any later time.

I remember we discussed some robust solution to this, but I can't recall what. Can you help me recall it? @rm3l @feloy

We discussed two possibilities:

pass a devfile version when running odo init (are different versions stored in registry?)
during integration tests, read the content of the devfile to get the values to assert, instead of hardcoding them in the test source code

valaparthvi commented 2 years ago

pass a devfile version when running odo init (are different versions stored in registry?)

I don't think this would work. There is only one version at a time. I say this by checking odo registry, if that is what you are pointing at.

odo registry output

```sh $ odo registry NAME REGISTRY DESCRIPTION dotnet50 StagingRegistry Stack with .NET 5.0 dotnet60 StagingRegistry Stack with .NET 6.0 dotnetcore31 StagingRegistry Stack with .NET Core 3.1 go StagingRegistry Go is an open source programming languag... java-maven StagingRegistry Upstream Maven and OpenJDK 11 java-openliberty StagingRegistry Java application Maven-built stack using... java-openliberty-gradle StagingRegistry Java application Gradle-built stack usin... java-quarkus StagingRegistry Quarkus with Java java-springboot StagingRegistry Spring Boot® using Java java-vertx StagingRegistry Upstream Vert.x using Java java-websphereliberty StagingRegistry Java application Maven-built stack using... java-websphereliberty-gradle StagingRegistry Java application Gradle-built stack usin... java-wildfly StagingRegistry Upstream WildFly java-wildfly-bootable-jar StagingRegistry Java stack with WildFly in bootable Jar ... nodejs StagingRegistry Stack with Node.js 16 nodejs-angular StagingRegistry Angular is a development platform, built... nodejs-nextjs StagingRegistry Next.js gives you the best developer exp... nodejs-nuxtjs StagingRegistry Nuxt is the backbone of your Vue.js proj... nodejs-react StagingRegistry React is a free and open-source front-en... nodejs-svelte StagingRegistry Svelte is a radical new approach to buil... nodejs-vue StagingRegistry Vue is a JavaScript framework for buildi... php-laravel StagingRegistry Laravel is an open-source PHP framework,... python StagingRegistry Flask is a web framework, it’s a Pytho... python-django StagingRegistry Django is a high-level Python web framew... ```

during integration tests, read the content of the devfile to get the values to assert, instead of hardcoding them in the test source code

I am inclined towards this solution, but then again, this cannot be done for all the test cases. We have been using local devfiles for most of our cases, but some tests in odo init and odo registry still call the command with devfile name to fetch it.

rm3l commented 2 years ago

Summary of Cabal and grooming discussions (2022-10-06):

Spin up our own Devfile registry, that we control. Odo tests should test odo features, not that the Devfile stacks are valid/usable with odo, which should be the scope of Devfile registry tests
Have one or just a few tests that validate that that odo can communicate with the default Devfile registry. They should not test other odo features, but could just check that the right Devfile has been downloaded.
Remove the NGINX server used for caching Devfiles (cf. #5922)
We should not use starter projects because we might have issues there as well. Ensure that we use projects in our local code base.

valaparthvi commented 1 year ago

Since this issue has come up recently way more frequently than we'd like, following changes have been implemented so far:

Spin up our own Devfile registry, that we control. Odo tests should test odo features, not that the Devfile stacks are valid/usable with odo, which should be the scope of Devfile registry tests

Setup our custom devfile registry on IBM cloud for our CI, this ensures that any issue caused by the Staging Registry does not block us from continuous development. See: https://github.com/redhat-developer/odo/pull/6705. On IBM cloud, we export the custom devfile registry via DEVFILE_REGISTRY and our tests capture this env var if available and use this custom registry.

Have one or just a few tests that validate that that odo can communicate with the default Devfile registry. They should not test other odo features, but could just check that the right Devfile has been downloaded.

We still want to catch any problems from the Staging Registry before they propagate to production, and so we continue to run our nightly tests with Staging Registry. See: https://github.com/redhat-developer/odo/pull/6725

Remove the NGINX server used for caching Devfiles (cf. https://github.com/redhat-developer/odo/pull/5922)

This was implemented, but then reversed because of frequent flaky behavior, IIRC.

We should not use starter projects because we might have issues there as well. Ensure that we use projects in our local code base.

This has been taken care of.

github-actions[bot] commented 1 year ago

A friendly reminder that this issue had no activity for 90 days. Stale issues will be closed after an additional 30 days of inactivity.

github-actions[bot] commented 1 year ago

This issue was closed because it has been inactive for 30 days since being marked as stale.

rm3l commented 1 year ago

/remove-lifecycle stale /remove-lifecycle rotten /reopen

Reopening because it is partially implemented, and there remains a few tasks still open.

openshift-ci[bot] commented 1 year ago

@rm3l: Reopened this issue.

In response to [this](https://github.com/redhat-developer/odo/issues/6138#issuecomment-1748274035): >/remove-lifecycle stale >/remove-lifecycle rotten >/reopen > >Reopening because it is partially implemented, and there remains a few tasks still open. Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

redhat-developer / odo