[Hardening aspect] Missing IPIs can go undetected in CoCo Linux guest

[ereshetova]

Problem

For a CoCo guest a malicious host/VMM can prevent IPIs to be delivered across vCPUs. We need to ensure that all missing IPIs can be detected or force waiting for failed deliveries of IPIs.

Solution

Analyze the respected APIs and its users within kernel, analyse the consequences of a missing IPI. Ideally develop a method to take the IPIs outside of host/VMM control by utilizing methods such as IPI virtualization https://lwn.net/Articles/863190/ or similar.

Initial analysis The below is initial analysis done a while back. It is not completed and needs checking by someone with a better knowledge.

Here is the list of x86 callers of smp_call_function that do not wait:

1. xen_pv_stop_other_cpus. Xen code is currently excluded.
2. sysrq_showregs_othercpus. Just printing info, safe to be lost.
3. scftorture_invoke_one & scf_torture_cleanup. Torture test for smp_call_function() and friends. We do not care about torture test module. Rest are using wait=true and therefore should be safe.

Here is the list of x86 callers of smp_call_function_many that do not wait:

1. raise_mce. MCE feature should be disabled in CoCo guest by respected CPUID bit.
2. scftorture_invoke_one. Torture test for smp_call_function() and friends. We do not care about torture test module.
3. kvm_kick_many_cpus. Kernel-based Virtual Machine driver for Linux, we dont care about this one in TDX guest Rest are using wait=true and therefore should be safe.

Here is the list of x86 callers of on_each_cpu that do not wait (note this omits the generic drivers users since we had an assumption that we only enable minimal set of drivers. If this assumption is not correct, this list must be extended):

1. fix_erratum_688. ??
2. lapic_update_tsc_freq. TSC re-calibration. Does not seem to have any strong implications if this fails.
3. rcu_gp_init & rcu_gp_cleanup. These are only for RCU_STRICT_GRACE_PERIOD, which as far as I understand are used for debug only due to big performance degradation. However, if this is used for TDX guest kernel, i think this might create problems in rcu operation. Rest are using wait=true and therefore should be safe.

on_each_cpu_cond_mask, on_each_cpu_cond, smp_call_function_any & on_each_cpu_mask do not have x86 callers that do not wait.

Here is the list of x86 callers of smp_call_function_single that do not wait:

1. aperfmperf_snapshot_cpu. Aperf/Mperf should be disabled in CoCo guest by respected CPUIDs.
2. do_inject. if MCE feature is disabled (see above), injection should be actually turned off also.
3. cpuhp_report_idle_dead. if CPU hot plug is disabled, this is also disabled?
4. trc_wait_for_one_reader. This I think will disturb the correctness of how rcu works, if not delivered. However, I dont know if there any direct security consequences.
5. scftorture_invoke_one. Torture test for smp_call_function() and friends. We do not care about torture test module.
6. sync_rcu_exp_select_node_cpus & sync_sched_exp_online_cleanup. Same as above trc_wait_for_one_reader. Not sure of consequences for proper rcu operation.

Rest are using wait=true and therefore should be safe.

x86 callers of smp_call_function_single_async:

1. cpuid_read. We are ok with such reads failing.
2. rdmsr_safe_on_cpu. We are ok with such reads failing.
3. blk_mq_complete_send_ipi. ??
4. ingenic_tcu_cevt_cb. Ingenic SoCs TCU IRQ driver. ??
5. cpuidle_coupled_poke. Wake up a cpu that may be waiting: seem ok to fail from security pov.
6. liquidio_napi_drv_callback. Cavium LiquidIO code. ??
7. kgdb_roundup_cpus. kgdb, dont care for security
8. hrtick_start. Called to set the hrtick timer state. Seems like a DoS for task switching if IPI is lost, but nothing worse as a consequence.
9. kick_ilb. Kick cpu to do nohz balancing. Doesn't seem to have security implications if fails.
10. net_rps_send_ipi. Send pending IPI's to kick RPS processing on remote cpus. Looks ok, the package just wont get processed if IPI is dropped.