search

redhat-developer / odo

odo - Developer-focused CLI for fast & iterative container-based application development on Podman and Kubernetes. Implementation of the open Devfile standard.
https://odo.dev
Apache License 2.0
792 stars 244 forks source link

Make operator hub integration tests run in parallel #3244

Closed dharmit closed 3 years ago

dharmit commented 4 years ago

/kind bug /area testing

What versions of software are you using?

Operating System: all

Output of odo version: master

Trying to run Operator Hub tests in parallel is leading to numerous race conditions that me & @amitkrout tried hard to understand and fix. However, it's not making complete sense and since we're on deadline to make sure that these test failures don't block other open PRs, we're opening this issue to deal with it separately.

How to reproduce?

Try to run operator hub tests in parallel by using GINKGO_FLAGS in Makefile instead of GINKGO_FLAGS_SERIAL.

Logs:

Here project we start with is pknmkygnnu but later on, oc tries to fetch info from nkyqyuspbj. We tried adding -n <namespace> to command but that didn't fix the race condition. There are other race conditions happening as well.

Creating a new project: pknmkygnnu                                                                                                                                                  [115/2932]
Running odo with args [odo project create pknmkygnnu -w -v4]                                                                                                                                  
[odo]  •  Waiting for project to come up  ...                                                                                                                                                 
[odo] I0526 15:54:15.927259   56684 occlient.go:542] Status of creation of project pknmkygnnu is Active                                                                                       
[odo] I0526 15:54:15.927304   56684 occlient.go:547] Project pknmkygnnu now exists                                                                                                            
[odo] I0526 15:54:15.929666   56684 occlient.go:582] Status of creation of service account &ServiceAccount{ObjectMeta:{default  pknmkygnnu /api/v1/namespaces/pknmkygnnu/serviceaccounts/defau
lt 0a318607-4e13-4275-ba67-10b919985f8e 513620 0 2020-05-26 15:54:14 +0530 IST <nil> <nil> map[] map[] [] []  []},Secrets:[]ObjectReference{ObjectReference{Kind:,Namespace:,Name:default-toke
n-78lgb,UID:,APIVersion:,ResourceVersion:,FieldPath:,},ObjectReference{Kind:,Namespace:,Name:default-dockercfg-v5nk2,UID:,APIVersion:,ResourceVersion:,FieldPath:,},},ImagePullSecrets:[]Local
ObjectReference{LocalObjectReference{Name:default-dockercfg-v5nk2,},},AutomountServiceAccountToken:nil,} is ready                                                                             
 ✓  Waiting for project to come up [1s]                                                                                                                                                       
[odo]  ✓  Project 'pknmkygnnu' is ready for use                                                                                                                                               
[odo] I0526 15:54:15.938363   56684 odo.go:72] Could not get the latest release information in time. Never mind, exiting gracefully :)                                                        
[odo]  ✓  New project created and now using project: pknmkygnnu                                                                                                                               
Running odo with args [odo project set pknmkygnnu]                                                                                                                                            
[odo] Already on project : pknmkygnnu                                                                                                                                                         
[odo] I0526 15:54:16.079165   56724 odo.go:72] Could not get the latest release information in time. Never mind, exiting gracefully :)                                                        
Created dir: /tmp/977599626                                                                                                                                                                   
Setting current dir to: /tmp/977599626                                                                                                                                                        
Running odo with args [odo preference set Experimental true -f]                                                                                                                               
[odo] I0526 15:54:16.180139   56747 preference.go:188] The path for preference file is /tmp/977599626/config.yaml                                                                             
[odo] Global preference was successfully updated                                                                                                                                              
[odo] I0526 15:54:16.180353   56747 odo.go:72] Could not get the latest release information in time. Never mind, exiting gracefully :)                                                        
Running oc with args [oc get csv]                                                                                                                                                             
[oc] No resources found in nkyqyuspbj namespace.                                                                                                                                              
Running oc with args [oc get csv]                                                                                                                                                             
[oc] No resources found in nkyqyuspbj namespace.                                                                                                                                              
Running oc with args [oc get csv]                                                                                                                                                             
[oc] No resources found in nkyqyuspbj namespace.                                                                                                                                              
Running oc with args [oc get csv]                                                                                                                                                             
[oc] No resources found in nkyqyuspbj namespace.                                                                                                                                              
Running oc with args [oc get csv]                                                                                                                                                             
[oc] No resources found in nkyqyuspbj namespace.                                                                                                                                              
Running oc with args [oc get csv]                                                                                                                                                             
[oc] No resources found in nkyqyuspbj namespace.                                                                                                                                              
Running oc with args [oc get csv]                                                                                                                                                             
[oc] No resources found in nkyqyuspbj namespace.                                                                                                                                              
Running oc with args [oc get csv]                                                                                                                                                             
[oc] No resources found in nkyqyuspbj namespace.                                                                                                                                              
Running oc with args [oc get csv]                                                                                                                                                             
[oc] No resources found in nkyqyuspbj namespace.                                                                                                                                              
Running oc with args [oc get csv]                                                                                                                                                             
[oc] No resources found in nkyqyuspbj namespace.                                                                                                                                              
Running oc with args [oc get csv]
[oc] No resources found in nkyqyuspbj namespace.
Running oc with args [oc get csv]
[oc] No resources found in nkyqyuspbj namespace.
Running oc with args [oc get csv]
[oc] No resources found in nkyqyuspbj namespace.
Running oc with args [oc get csv]
[oc] No resources found in nkyqyuspbj namespace.
Running oc with args [oc get csv]
[oc] No resources found in nkyqyuspbj namespace.
Running oc with args [oc get csv]
[oc] NAME                        DISPLAY   VERSION   REPLACES   PHASE
[oc] mongodb-enterprise.v1.5.2   MongoDB   1.5.1                
Running oc with args [oc get csv]
[oc] NAME                        DISPLAY   VERSION   REPLACES   PHASE
[oc] mongodb-enterprise.v1.5.2   MongoDB   1.5.1                
Running oc with args [oc get csv]
[oc] NAME                        DISPLAY   VERSION   REPLACES   PHASE
[oc] mongodb-enterprise.v1.5.2   MongoDB   1.5.1                
Running oc with args [oc get csv]
[oc] NAME                        DISPLAY   VERSION   REPLACES   PHASE
[oc] mongodb-enterprise.v1.5.2   MongoDB   1.5.1                
Running oc with args [oc get csv]
[oc] NAME                        DISPLAY   VERSION   REPLACES   PHASE
[oc] mongodb-enterprise.v1.5.2   MongoDB   1.5.1                
Running oc with args [oc get csv]
[oc] NAME                        DISPLAY   VERSION   REPLACES   PHASE
[oc] mongodb-enterprise.v1.5.2   MongoDB   1.5.1                
Running oc with args [oc get csv]
[oc] NAME                        DISPLAY   VERSION   REPLACES   PHASE
[oc] mongodb-enterprise.v1.5.2   MongoDB   1.5.1                
Running oc with args [oc get csv]
[oc] NAME                        DISPLAY   VERSION   REPLACES   PHASE
[oc] mongodb-enterprise.v1.5.2   MongoDB   1.5.1                
Running oc with args [oc get csv]
[oc] NAME                              DISPLAY   VERSION             REPLACES   PHASE
[oc] etcdoperator.v0.9.4-clusterwide   etcd      0.9.4-clusterwide              
[oc] mongodb-enterprise.v1.5.2         MongoDB   1.5.1                          
Running oc with args [oc get csv]
[oc] NAME                              DISPLAY   VERSION             REPLACES   PHASE
[oc] etcdoperator.v0.9.4-clusterwide   etcd      0.9.4-clusterwide              
[oc] mongodb-enterprise.v1.5.2         MongoDB   1.5.1                          
Running odo with args [odo catalog list services]
[odo] I0526 15:54:43.690386   57478 preference.go:188] The path for preference file is /tmp/977599626/config.yaml
[odo] I0526 15:54:43.719291   57478 operators.go:17] Fetching list of operators installed in cluster
[odo] I0526 15:54:43.739232   57478 preference.go:188] The path for preference file is /tmp/977599626/config.yaml
[odo] I0526 15:54:43.739320   57478 preference.go:188] The path for preference file is /tmp/977599626/config.yaml
[odo] Operators available in the cluster
[odo] NAME                                CRDs
[odo] etcdoperator.v0.9.4-clusterwide     EtcdCluster, EtcdBackup, EtcdRestore
[odo] mongodb-enterprise.v1.5.2           MongoDB, MongoDBUser, MongoDBOpsManager
[odo] I0526 15:54:43.739402   57478 odo.go:72] Could not get the latest release information in time. Never mind, exiting gracefully :)
Running odo with args [odo service create etcdoperator.v0.9.4-clusterwide --crd EtcdCluster]
[odo] I0526 15:54:43.869421   57511 preference.go:188] The path for preference file is /tmp/977599626/config.yaml
[odo] I0526 15:54:43.869510   57511 preference.go:188] The path for preference file is /tmp/977599626/config.yaml
[odo] I0526 15:54:43.869548   57511 operators.go:17] Fetching list of operators installed in cluster
[odo] I0526 15:54:43.885244   57511 preference.go:188] The path for preference file is /tmp/977599626/config.yaml
[odo] Deploying service of type: EtcdCluster
[odo]  •  Deploying service  ...
[odo] I0526 15:54:43.885396   57511 preference.go:188] The path for preference file is /tmp/977599626/config.yaml
 ✓  Deploying service [14ms]
[odo]  ✓  Service 'example' was created
[odo] 
[odo] Progress of the provisioning will not be reported and might take a long time
[odo] You can see the current status by executing 'odo service list'
[odo] Optionally, link etcdoperator.v0.9.4-clusterwide to your component by running: 'odo link <component-name>'
[odo] I0526 15:54:43.900395   57511 odo.go:72] Could not get the latest release information in time. Never mind, exiting gracefully :)
Running oc with args [oc get pods -n pknmkygnnu]
[oc] No resources found in pknmkygnnu namespace.
Running oc with args [oc get pods  -o template="{{.status.phase}}" -n pknmkygnnu]
[oc] error: resource name may not be empty
Deleting project: pknmkygnnu
Running odo with args [odo project delete pknmkygnnu -f]
[odo] I0526 15:54:45.316595   57570 application.go:49] Unable to list Service Catalog instances: unable to list ServiceInstances: serviceinstances.servicecatalog.k8s.io is forbidden: User "d
eveloper" cannot list resource "serviceinstances" in API group "servicecatalog.k8s.io" in the namespace "pknmkygnnu"
[odo]  ✓  Deleted project : pknmkygnnu
[odo]  ⚠  Warning! Projects are deleted from the cluster asynchronously. Odo does its best to delete the project. Due to multi-tenant clusters, the project may still exist on a different nod
e.
[odo] I0526 15:54:45.323445   57570 odo.go:72] Could not get the latest release information in time. Never mind, exiting gracefully :)

• Failure [30.809 seconds]
odo service command tests for OperatorHub
/home/dshah/go/src/github.com/openshift/odo/tests/integration/operatorhub/cmd_service_test.go:17
  When creating an operator backed service
  /home/dshah/go/src/github.com/openshift/odo/tests/integration/operatorhub/cmd_service_test.go:69
    should be able to create EtcdCluster from its alm example [It]
    /home/dshah/go/src/github.com/openshift/odo/tests/integration/operatorhub/cmd_service_test.go:79

    No future change is possible.  Bailing out early after 0.111s.
    Running oc with args [oc get pods  -o template="{{.status.phase}}" -n pknmkygnnu]
    Expected
        <int>: 1
    to match exit code:
        <int>: 0

    /home/dshah/go/src/github.com/openshift/odo/tests/helper/helper_generic.go:54
openshift-bot commented 4 years ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot commented 3 years ago

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten /remove-lifecycle stale

openshift-bot commented 3 years ago

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen. Mark the issue as fresh by commenting /remove-lifecycle rotten. Exclude this issue from closing again by commenting /lifecycle frozen.

/close

openshift-ci-robot commented 3 years ago

@openshift-bot: Closing this issue.

In response to [this](https://github.com/openshift/odo/issues/3244#issuecomment-744077832): >Rotten issues close after 30d of inactivity. > >Reopen the issue by commenting `/reopen`. >Mark the issue as fresh by commenting `/remove-lifecycle rotten`. >Exclude this issue from closing again by commenting `/lifecycle frozen`. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.