This was because JS from the old version of the Stripe payment processor plugin was overriding the new and therefore included this bug. It was being included by CiviCRM's cached templates!
So we just needed to click the Cleanup Caches button on this page.
Critical issue: