Only deploy Ceph operator and CRDs when using Ceph storage

[Adam-D-Lewis]

Context

Currently Rook Ceph operator is deployed even if Rook Ceph is not being used for storage. Since we've disallowed changing from one storage type to another without a redeployment, we should make the ceph operator and CRDs only deployed if Rook Ceph is being used for storage on that deployment.

Value and/or benefit

Less unneeded clutter and resources being used in the cluster.

Anything else?

No response

[marcelovilla]

@kenafoster mentioned offline this could be a potential issue blocking them from upgrading their deployments. @kenafoster would you consider this a release blocker?

[kenafoster]

I do consider it a release blocker, at least as it relates to my team's use case. Given that we operate in a security-audited environment, we will be very hesitant to deploy the Nebari release into our environment if it is unnecessarily deploying the rook-ceph operator chart... which creates a bunch of unnecessary accounts and RBAC items that may then have to go through audit

[Adam-D-Lewis]

This would be a useful thing to add. Would you be interested in creating a PR for it @kenafoster? I don't think it should take long and I'm happy to provide guidance as needed.

[kenafoster]

@Adam-D-Lewis I think it's as simple as the PR above, right? I'm testing now in AWS to make sure I'm not missing anything

[Adam-D-Lewis]

@kenafoster Yeah, I think that'd be sufficient. We need to test a fresh deployment with rook ceph storage set to ensure that works as well e.g. Deploys successfully and able to store a file/build a conda environment. I don't think there'll be any issues though. If your testing goes well then I can try a Ceph enabled one. Then it should be good to go.