add generating auth files for jenkins profisioned rosa

[qiliRedHat]

1. https://issues.redhat.com/browse/OCPQE-15925, updated generate_auth_files.sh and README.md.
2. Update Tasks.py to add a universal __get_kubeconfig(self,user).
3. Add a fix to scale_deployment function in Tasks.py to avoid the Warning for DeploymentConfig in 4.14.
4. Exclude insights operator as it is degraded on ocm staging env as expected

[qiliRedHat]

Test PASSed on rosa-hypershift

% ./generate_auth_files.sh                        
CLUSTER_ID, OCM_TOKEN and OCM_ENV are provided for ocm-profile-ci Jenkins job provisioned rosa cluster.
Checking ocm cli
Checking rosa cli
I: Logged in as 'svetsa-ocm' on 'https://api.stage.openshift.com'
Creating admin user ....
admin file is created
Waiting for admin user login and generate kubeconfig ....
admin user login successfully and kubeconfig file is created.
Generating 80 IDP users under the htpasswd idp ....
users file is created
The credential files for reliability test are created under folder path_to_auth_files.

% ls path_to_auth_files 
admin               create_idp_users_rosa.log   kubeconfig          users

Negative path

1. Create admin failed

   # ./generate_auth_files.sh 
   CLUSTER_ID, OCM_TOKEN and OCM_ENV are provided for ocm-profile-ci Jenkins job provisioned rosa cluster.
   Checking ocm cli
   Checking rosa cli
   I: Logged in as 'svetsa-ocm' on 'https://api.stage.openshift.com'
   Creating admin user ....
   ERR: Cluster '253puj7ucnkk459ih2co3ufipdpt0833' already has an admin
   rosa create admin cmd failed.

2. Create idp users failed

   # ./generate_auth_files.sh 
   CLUSTER_ID, OCM_TOKEN and OCM_ENV are provided for ocm-profile-ci Jenkins job provisioned rosa cluster.
   Checking ocm cli
   Checking rosa cli
   I: Logged in as 'svetsa-ocm' on 'https://api.stage.openshift.com'
   Creating admin user ....
   admin file is created
   Waiting for admin user login and generate kubeconfig ....
   admin user login successfully and kubeconfig file is created.
   Generating 50 IDP users under the htpasswd idp ....
   Error happened during create_idp_users_rosa, please check path_to_auth_files/create_idp_users_rosa.log.

[qiliRedHat]

@svetsa-rh PTAL A running rosa-hypershift staging reliability test https://redhat-internal.slack.com/archives/C0266JJ4XM5/p1689929537589349

[qiliRedHat]

Test on classic rosa

# ./generate_auth_files.sh 
CLUSTER_ID, OCM_TOKEN and OCM_ENV are provided for ocm-profile-ci Jenkins job provisioned rosa cluster.
Checking ocm cli
Checking rosa cli
I: Logged in as 'svetsa-ocm' on 'https://api.stage.openshift.com'
Creating admin user ....
admin file is created
Waiting for admin user login and generate kubeconfig ....
admin user login successfully and kubeconfig file is created.
Generating 50 IDP users under the htpasswd idp ....
users file is created
The credential files for reliability test are created under folder path_to_auth_files.

# ls path_to_auth_files/
admin  create_idp_users_rosa.log  kubeconfig  users

[svetsa-rh]

Hi @qiliRedHat I added couple of minor comments, but looks good overall. Couple of questions: 1) If someone were to look at reliability test results, for ex https://redhat-internal.slack.com/archives/C0266JJ4XM5/p1689929537589349, how can they know if the test ran on Self managed/Classic Rosa or Hypershift cluster? 2) Are we printing any output to reliability logs, like oc describe cluster or rosa describe cluster? Can you share sample logs if you have them? Do you think this will be helpful?

FYI, to distinguish between Classic Rosa cluster and Hypershift cluster, you could run a command such as rosa describe cluster -c "$CLUSTER_ID" -o json | jq -r ".hypershift.enabled"

[qiliRedHat]

Hi @qiliRedHat I added couple of minor comments, but looks good overall. Couple of questions:

1. If someone were to look at reliability test results, for ex https://redhat-internal.slack.com/archives/C0266JJ4XM5/p1689929537589349, how can they know if the test ran on Self managed/Classic Rosa or Hypershift cluster?
2. Are we printing any output to reliability logs, like oc describe cluster or rosa describe cluster? Can you share sample logs if you have them? Do you think this will be helpful?

FYI, to distinguish between Classic Rosa cluster and Hypershift cluster, you could run a command such as rosa describe cluster -c "$CLUSTER_ID" -o json | jq -r ".hypershift.enabled"

What we currently print to log are just api info and server version.

Reliability-v2 test has started on cluster: [https://api.sdq-ci-khiua.7elj.s3.devshift.org:443](https://api.sdq-ci-khiua.7elj.s3.devshift.org/)
Server Version: 4.13.4
Kubernetes Version: v1.26.5+7d22122

I also thought about this, but some concern/limitation stop me from doing that

1. For a non rosa cluster, we don't require user to install ocm and rosa command line.
2. Inside the reliability test, we currently does not support com and rosa command (this can be done in the future out of this pr if there is a need). I don't think from oc cli we can identify if the cluster is classic rosa or rosa hypershift, a tricky way to identify from the cluster name/api - the cluster installed by the ocm Jenkins job has a special name 'sdq-ci-'. But again no rosa cli support now, so not able to check if it is hypershift. I think we can consider this in later pr if not urgent.

[svetsa-rh]

Hi @qiliRedHat I added couple of minor comments, but looks good overall. Couple of questions:

1. If someone were to look at reliability test results, for ex https://redhat-internal.slack.com/archives/C0266JJ4XM5/p1689929537589349, how can they know if the test ran on Self managed/Classic Rosa or Hypershift cluster?
2. Are we printing any output to reliability logs, like oc describe cluster or rosa describe cluster? Can you share sample logs if you have them? Do you think this will be helpful?

FYI, to distinguish between Classic Rosa cluster and Hypershift cluster, you could run a command such as rosa describe cluster -c "$CLUSTER_ID" -o json | jq -r ".hypershift.enabled"

What we currently print to log are just api info and server version.

Reliability-v2 test has started on cluster: [https://api.sdq-ci-khiua.7elj.s3.devshift.org:443](https://api.sdq-ci-khiua.7elj.s3.devshift.org/)
Server Version: 4.13.4
Kubernetes Version: v1.26.5+7d22122

I also thought about this, but some concern/limitation stop me from doing that

1. For a non rosa cluster, we don't require user to install ocm and rosa command line.
2. Inside the reliability test, we currently does not support com and rosa command (this can be done in the future out of this pr if there is a need). I don't think from oc cli we can identify if the cluster is classic rosa or rosa hypershift, a tricky way to identify from the cluster name/api - the cluster installed by the ocm Jenkins job has a special name 'sdq-ci-'. But again no rosa cli support now, so not able to check if it is hypershift. I think we can consider this in later pr if not urgent.

That sounds good to me Qiujie. I was thinking that this info might be helpful for anyone doing scale-ci monitoring, especially when you are on PTO or holidays.

[svetsa-rh]

/lgtm