cloud-native-toolkit / software-everywhere

Automation modules to deploy IBM Software on IBM Cloud, AWS, and Azure
https://modules.cloudnativetoolkit.dev
Apache License 2.0
9 stars 9 forks source link

Tubnonomic automation no longer working properly #354

Closed triceam closed 2 years ago

triceam commented 2 years ago

The turbnomic automation fails to deploy on both OCP 4.8 and 4.10 on IBM Cloud.

Automation: https://github.com/IBM/automation-turbonomic

The deployment never becomes healthy b/c all of the PVCs (which use the storage class created by the 202 layer) never become healthy.

The error events coming from the PVCs are similar to:

Generated from vpc.block.csi.ibm.io_ibm-vpc-block-csi-controller-0_be0c2dbc-2418-4f0a-8069-3a17a2503793
6 times in the last 1 hour
(combined from similar events): failed to provision volume with StorageClass "ibmc-vpc-block-mzr": rpc error: code = InvalidArgument desc = {RequestID: 84bf2f91-8e7d-4c93-af0b-b22daff331e5 , Code: InvalidParameters, Description: Failed to extract parameters, BackendError: <sizeRange> is an invalid parameter, Action: Please provide valid parameters}

The storage class has a volume size 10-2000Gi. The volumes have a size of 1Gi, so I suspect that is the issue.

image.png image.png

timroster commented 2 years ago

is there any reason why the default storage (block) or maybe ibmc-vpc-block-metro-10iops-tier since wait for first customer is specified in https://github.com/turbonomic/t8c-install/wiki/Storage-Class-Requirements ?

triceam commented 2 years ago

I was able to recover the deployment by changing the storage class used by Turbonomic to the cluster default: ibmc-vpc-block-10iops-tier

Before changing the automation, I need to understand why we created that storage class in the first place... and I do not know why. @tcskill @mjperrins @seansund Any idea why we create the ibmc-vpc-block-mzr storage class for IBM Cloud deployments? This is what is causing turbo to fail.

tcskill commented 2 years ago

@triceam yes. That is needed on ibm cloud only when you have a vpc deployment or multi region deployment because by default Ibm cloud creates block storage with bind immediately and need it to be wait for first consumer (note only for the use case described above) when on classic cluster not needed, also when on single zone not needed

timroster commented 2 years ago

standard metro classes will work for this use case, no need for a custom storageclass

triceam commented 2 years ago

OK, well, it doesn't work anymore. The <sizeRange> is an invalid parameter, Action: Please provide valid parameters} error is b/c turbo is requesting 1Gi volumes, and the minimum size is 10Gi. I will update our automation to use the metro class.

tcskill commented 2 years ago

Yes I see new metro has wait for first consumer so should work fine

triceam commented 2 years ago

Removed 202 layer (no custom storage class for IBM Cloud, switched to use ibmc-vpc-block-metro-10iops-tier storage class on IBM Cloud.

Fixed in: