Unable to disable express checkout setting address with publishable key

[grantsingleton]

What happened?

When I use ExpressCheckout, I update the address on the payment intent on the server with the secret key. When I call stripe.confirmPayment, it tries to update the address with the publishable key and fails with this error:

The shipping information on this PaymentIntent was last set with a secret key and therefore cannot be changed with a publishable key. Please use your secret key instead.

There is no way to tell confirmPayment that the address has been set already to avoid this error.

Here is my implementation for reference.

    <ExpressCheckoutElement
      onConfirm={async event => {
        try {
          const name = event.billingDetails?.name;
          const email = event.billingDetails?.email;
          const address: CheckoutBuyerAddress = {
            name,
            email,
            address_line1: event.shippingAddress?.address?.line1,
            address_line2: event.shippingAddress?.address?.line2,
            city_locality: event.shippingAddress?.address?.city,
            state_province: event.shippingAddress?.address?.state,
            postal_code: event.shippingAddress?.address?.postal_code,
            country_code: event.shippingAddress?.address?.country
          };

          const { error: submitError } = await elements.submit();
          if (submitError) {
            throw new Error(submitError.message);
          }

         // this is where I update the payment intent with the address
          await updateCheckout({ buyerAddress: address });

          // ** Throws the error here **
          const { error: confirmError } = await stripe.confirmPayment({
            elements,
            clientSecret: checkoutData.paymentIntent,
            confirmParams: {
              return_url: window.location.href
            }
          });

          if (confirmError) {
            throw new Error(confirmError.message);
          }
        } catch (e: any) {
          console.error(e.message);
          console.error(e);
        }
      }}
    />

What's needed: the ability to tell confirmPayment not to set the address.

Environment

No response

Reproduction

No response

[grantsingleton]

Closing this since i'm taking a different approach that doesnt set the address on the server.