When used together with serial, tasks marked as run_once will be run on one host in each serial batch. If the task must run only once regardless of serial mode, use when: inventory_hostname == ansible_play_hosts_all[0] construct.
Which explains why run_once wasn't working for us.
According to a note in the ansible docs: https://docs.ansible.com/ansible/latest/user_guide/playbooks_strategies.html#running-on-a-single-machine-with-run-once
Which explains why
run_oncewasn't working for us.