[Epic] crc should pass OpenShift conformance tests

[guillaumerose]

An OpenShift cluster is an OpenShift cluster as long as it passes conformance tests. Therefore, crc should pass all those tests which a single node openshift cluster can support.

• [ ] Find the list of broken tests https://github.com/code-ready/snc/pull/399
• [ ] Ignore the broken tests if those are failing because of known reasons (like HA specific one)
• [ ] Merge that list to snc (which is we are trying with https://github.com/code-ready/snc/pull/399)

Once those tests are green in the CI we will automatically have it part of installer e2e side because it also uses same ci.sh script which snc repo contains.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[cfergeau]

Still in progress

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[praveenkumar]

At least it should pass test matrix what openshift single can run.

[cfergeau]

I ran the conformance tests (list of tests ./openshift-tests run --dry-run openshift/conformance, openshift-tests comes from github.com/openshift/origin) against crc 2.0.1:

Conformance.log:error: 5 fail, 0 pass, 0 skip (200ms)started: (0/1/5) "[Conformance][sig-api-machinery][Feature:APIServer] local kubeconfig \"lb-ext.kubeconfig\" should be present on all masters and work [Suite:openshift/conformance/parallel/minimal]"
Conformance.log:error: 5 fail, 0 pass, 0 skip (200ms)started: (0/1/5) "[Conformance][sig-api-machinery][Feature:APIServer] local kubeconfig \"localhost.kubeconfig\" should be present on all masters and work [Suite:openshift/conformance/parallel/minimal]"
Conformance.log:error: 5 fail, 0 pass, 0 skip (200ms)started: (0/1/5) "[Conformance][sig-api-machinery][Feature:APIServer] local kubeconfig \"lb-ext.kubeconfig\" should be present on all masters and work [Suite:openshift/conformance/parallel/minimal]"
Serial.log:error: 1 fail, 0 pass, 11 skip (10m57s)
sig-apps.log:error: 1 fail, 107 pass, 7 skip (9m10s)
sig-arch.log:error: 31 fail, 14 pass, 2 skip (6.4s)
sig-auth.log:error: 2 fail, 75 pass, 2 skip (3m11s)
sig-autoscaling.log:error: 1 fail, 0 pass, 7 skip (15m0s)
sig-builds.log:error: 2 fail, 38 pass, 2 skip (15m49s)
sig-cli.log:error: 57 fail, 51 pass, 0 skip (1m38s)
sig-cluster-lifecycle.log:error: 4 fail, 2 pass, 0 skip (15m11s)
sig-etcd.log:error: 1 fail, 0 pass, 0 skip (1.3s)
sig-imageregistry.log:error: failed because an invariant was violated, 29 pass, 0 skip (1m43s)
sig-instrumentation.log:error: 16 fail, 4 pass, 0 skip (23.9s)
sig-network.log:error: 11 fail, 107 pass, 49 skip (10m54s)
sig-node.log:error: 1 fail, 127 pass, 2 skip (15m18s)
sig-operator.log:error: 1 fail, 9 pass, 0 skip (42.4s)
sig-storage.log:error: 12 fail, 353 pass, 1928 skip (29m5s)

The cluster did not have monitoring enabled, so some of the failures are expected. The sig-arch and sig-cli tests are the ones with the most failures, need to take a closer look.