[FEATURE] Allow user to set resource constraint for Longhorn components

[PhanLe1010]

Is your feature request related to a problem? Please describe (👍 if you like this request)

In the current implementation of Longhorn, there is no setting to control the CPU and memory limits as we believe that Longhorn, as a critical storage layer, should not be limited or killed by OOMKill. The reason is that if Longhorn struggles, the application workloads cannot function well. However, this assumption might not be correct for the edge environment in which the user might want to set limits for the Longhorn system toa void Longhorn reader the system unusable.

There are multiple tickets related to the resource constraint topic currently:

1. Share manager: https://github.com/longhorn/longhorn/issues/8314
2. Instance manager: https://github.com/longhorn/longhorn/issues/4009
3. And I imagine that user would likely want to customized the setting for other components like longhorn-manager, csi-plugin,...
4. https://github.com/longhorn/longhorn/issues/6351

This ticket serves as the master ticket to combine them all together.

Additional Context

@innobead mentioned that Rook resource constraints strategy is: https://github.com/rook/rook/blob/master/design/ceph/resource-constraints.md . Maybe we can take a look at their approach

[benfiola]

Apologies in advance, but could this be expanded to support - broadly - 'pod metadata' (using terminology from the Alertmanager/Prometheus operators)? For my personal use case, I'd like to be able to attach additional labels to Longhorn managed pods (in addition to defining pod requests).

Also, I'm more than happy to implement this if you think it's an appropriate first-issue for a newcomer like myself and if there's an appetite for external contributions.