Differences between podified and appliance

[Fryguy]

This is a checklist of the things we know do not work in podified, but do work in appliances. If you know of more features, please comment here.

• [ ] Embedded Ansible - currently synchronous and thus tied to a 10 minute timeout as a workaround. Otherwise it needs the concept of "server", which is different in pods. See https://github.com/ManageIQ/manageiq/issues/19957
• [ ] oVirt/RHV fleecing - Currently depends on a directLUN hook to the appliance that can't be used in pods, needs redesign https://github.com/ManageIQ/manageiq-pods/issues/596
• [ ] hot fixes (rpm or patched ruby files) for severe bugs cannot be applied to pods, it requires a newly deployed image
• [x] Cockpit role - ManageIQ/manageiq-pods#531
• [ ] HA in pods (what it means or doesn't mean), related to: https://github.com/ManageIQ/manageiq-pods/issues/583. repmgr and automatic postgres failover is an appliance only concept.
• [ ] Region replication is baked into the appliance console and the UI for adding subscriptions. It's unclear how this is accomplished in podified and needs to be tested and documented.
• [ ] Multiple LDAP systems per tenant (was implemented at the appliance level)
• [ ] Log presentation in the UI - #697 (technically also a deficiency in the appliances as well - https://github.com/ManageIQ/manageiq/issues/19582)
  • [x] https://github.com/ManageIQ/manageiq-ui-classic/pull/8525
• [ ] Host provisioning via PXE - not sure if the IPMI tools work in pods, but the NFS mount to write the PXE configs shouldn't.
• [x] https://github.com/ManageIQ/manageiq-ui-classic/pull/8128
• [ ] httpd default is OIDC / SAML only
  • [ ] httpd-init container is needed for other SSSD based external auth

This section describes features that work differently in podified vs. appliances. These differences could mean a less desirable feature set and/or require different documentation.

• [ ] Documentation for enabling SSA in podified vs. appliances is different. As an example:
  • For VMware SSA in podified, the user must create their own container image based on ours, then they install the vddk and follow any steps required to agree to vmware's terms and redistribution rules, and finally, they will need to publish and test this image for deployment. This PR added a hook to look for the vddk installation materials in the image: https://github.com/ManageIQ/manageiq-pods/pull/666
  • For VMware in appliances, the process is to agree to the terms and install the VDDK in the base appliance we ship, then use that image as a template for which they deploy in their environment.
• [ ] Documentation for upgrading postgresql versions is different.
  • In podified, the pvc is a shared resource in the CR, so there is a workflow process to backup the existing database, bring down the old pg pod, clear the pvc, and bring up the new pg pod and restore the database.
  • In appliances, the storage is not shared and the database backup is all that needs to be brought over to the new appliance.
• [ ] Resource limits in podified and appliances are different
  • In appliances, resource limits are handled by the operating system (swap usage) and our internal worker management where we limit memory usage before workers are requested to restart and to disable starting new workers if the swap usage is excessive.
  • In podified, resource limits are specified in the CR and follow kubernetes rules. Within the pod, we have no real idea how much CPU or memory we're using. Limits on CPU cause throttling of the pod which is only evident by looking at /sys/fs/cgroup/cpu/cpu.stat. Limits on memory cause the pod to be OOMKilled via kill -9 with no graceful cleanup. CPU and memory request are used for autoscaling when we get there.
• [ ] Log retention for podified is different
  • In podified, it depends on the cluster level settings and if they have setup clustered logging. It's up to them to set this up and verify it's working. Otherwise, it looks like the default is pods have a current and previous log. If it's restarted more than once, we lose the prior previous log. If pods are deleted due to a deployment being removed, we also lose those.
  • In appliances, it's more universal that logrotate will keep logs x number of days on each appliance. The only time we lose logs is if it's been logrotate x number of days since the event we care about or if we fill up the disk.
• [ ] System events / notification is different in podified compared to appliances. It might have a similar workaround for podified as log retention.
  • In podified, system events linked up to current pods/deployments/etc. (oc get events) are retained, perhaps 4 hours or more. If an object is removed from the system, these events might disappears sooner. If a node or storage had an issue, there might not be an event that there's something wrong. Even worse, that event may not live very long so we could lose post-mortem information.
  • In appliances, mostly everything works through journald so you can actually get a system-wide view for most things, including events for other failing services, by looking at the journald logs. /var/log likely contains other logs that could be relevant. Additionally, these event like logging should be retained in a similar logrotate approach as normal application logging.

[agrare]

I believe we had issues with VM remote consoles as well right @skateman ?

[skateman]

I don't know, Nick might have sorted them out.

[Fryguy]

remote console support was added in https://github.com/ManageIQ/manageiq-pods/pull/540, but Nick said he didn't see it working end to end since RHV consoles were different and he didn't have the VMware libs. There's https://github.com/ManageIQ/manageiq-pods/issues/538 to document how to install the right libraries.

[skateman]

Not sure if we can support html5 consoles in RHV any longer, the provider has this option disabled :confused:

[miq-bot]

This issue has been automatically marked as stale because it has not been updated for at least 3 months.

If you can still reproduce this issue on the current release or on master, please reply with all of the information you have about it in order to keep the issue open.

Thank you for all your contributions! More information about the ManageIQ triage process can be found in the triage process documentation.