Shopper subscription payment failing for unknown cause

[lbwright22]

Description

For two weeks in a row shopper (user ID: 220064 ) subscription payment request has failed to be sent to Stripe for collection.

order IDs R755515222 and R357704180

System is showing that the shopper has a valid bank card associated with their OFN account and has authorised charges to enterprise: ID 200270.

On the Stripe platform the shopper's registered bank card is within date (ie not expired).

Expected Behavior

Order cycle closes and subscription payments are collected via Stripe. Enterprise manager and shopper are informed of the cause of a failed payment.

Actual Behaviour

Order cycle closes and subscription payments are collected via Stripe. Enterprise manager and shopper are not informed of the cause of a failed payment.

Steps to Reproduce

1. 2. 3. 4.

Animated Gif/Screenshot

Workaround

Severity

Your Environment

• Version used:
• Browser name and version:
• Operating System and version (desktop or mobile):

Possible Fix

[lin-d-hop]

Failed payment = S2 :-( Hopefully not too complex in the context of related issues all over the place...

[andrewpbrett]

Digging into this... each of those orders have two payments associated with them, one in the "failed" state and one in "complete". In both cases the failed one was created about five days before the completed one, but was updated around the same time as the completed one was created.

Looking at the Stripe logs I see the two completed payments for the customer for £10.00 each.

There's definitely something buggy going on here, but @lbwright22 could you add some detail on where you're seeing that the payments aren't collected? I haven't logged in to view the backoffice for this order, just looked directly at the database, but I would expect it to show one failed and one completed payment for each of these orders.

[lbwright22]

@andrewpbrett Cultivate subscription order cycle opens on Wednesday morning at 6 am and payments are collected (when it closes) at noon on Monday. In both cases for this shopper the automated payment collection on Monday failed and I created a payment after noon on Monday via the Stripe dashboard. There was no trace for the failed payment in Stripe nor on the OFN platform (in Stripe the manually created orders which I make I label with just the OFN order ID, the automated orders which come through OFN are named with 'SPREE' and the order ID). I think the manually created payment i collected (so the hub was not out of pocket) might be confusing things?

[andrewpbrett]

Got it. Yes, that was confusing - I saw that the successful payments happened very quickly after the order cycle closed, and assumed it must have been created by the code, but you are just that fast :)

On the admin dashboard for this order, I do see two payments. The failed payment is created on Wednesday morning, 6 am (not 3am) and updated right before the OC closes.

The SubscriptionPlacementJob moves the order to the completed state, but it should skip calling process_payments on the order until the order cycle closes. I'm looking into whether it's expected that the payment is created at all, or if that's potentially where things are going wrong.

[lbwright22]

Hi @andrewpbrett the subscription payment failed again for this customer and I still have no idea why.

I am creating a payment manually through Stripe now so that the payments are up to date for the enterprise but is there any trace as to why this one customer keeps having problems?

[Matt-Yorkley]

There's a clue in the transaction log:

Is it only happening with one order though?

[Matt-Yorkley]

So in this case that URL /v1/payment_intents//capture should have a payment intent ID in it, like: /v1/payment_intents/pi_12983718273/capture, but it was obviously nil for some reason when the API call was being made...

[andrewpbrett]

That's a good find Matt - so we're failing to create the payment intent in the first place it seems. The failed payment is getting created when the order cycle opens, on Wednesday morning. I'll take a look at it tomorrow and check the state of the payment, because I think we shouldn't even attempt to process it at that time.

[andrewpbrett]

The payment for this week's subscription order is currently still in the checkout state, so it doesn't look like we try to process it prematurely 👍.

However, in taking a look at it, I notice that the source for the payment is nil. There are several other payments for other subscriptions in the order cycle with a nil source as well, so this may or may not be the problem.

For this specific issue, I looked at the card we have on file for this customer. In our database its expiration date is 05/2021. It looks like the card was added on 23/04/2021. On Stripe, however, it looks like the card was automatically updated on 28/04/2021 and the expiration date and last four digits are now different.

It seems very likely that this why we're not processing the payment. We'll probably need to set up a webhook and have Stripe alert us when the payment_method.automatically_updated event happens so that we can take the (yet to be determined) appropriate action.

For right now, @lbwright22, I think that if we were to ask the customer to go to /account#/cards, 1) delete the old card, 2) add the new one and 3) reauthorize Cultivate Oxford, the payments will start working. This is obviously not the solution long term! But it would help to confirm this theory if they follow those steps and then the payment works this week.

[lbwright22]

Hi @andrewpbrett After the customer updated their card details again (as you suggested above) their payment went through successfully this morning.

However, this week there are three other customers for which the automated payments failed (I have created a payment in Stripe for each manually). Is this the same issue? I'm wondering if these people updated their cards due to expiry dates at the end of the month?
R468444585 R568361327 R413824420

I can see if I was a customer who 'did the right thing' by updating my card details (due to a card expiring) and then to be told I had to do it again, I would get annoyed.

[andrewpbrett]

Thanks for the update @lbwright22.

I can see if I was a customer who 'did the right thing' by updating my card details (due to a card expiring) and then to be told I had to do it again, I would get annoyed.

In that first case, the problem (at least as I currently understand it) was that Stripe did an automatic update of the card number/expiry on their end because they saw it was about to expire, and we didn't make the corresponding change on the OFN side. I'm not sure exactly how this works to be honest, that the card could be updated without the customer doing it themselves, but I do know that Stripe has this ability in some cases.

So I don't think we'd ever have to ask a customer to update their card details twice (that would definitely be annoying). Once the customer updates them in OFN, they're good (until the card expires, anyway).

I'm now looking into the three orders that had failed payments that you listed. One thing I notice right away is that we have lots (as in a dozen or more) credit card objects attached to each of those customers. In one case it's the same card info, in another it looks like there's two or three unique cards that get repeated.

In two cases (R468444585 and R413824420), the card that's listed as the default one for the customer expires June 2021, so that seems like the explanation: we tried to charge an expired card. In the case of R568361327, the default card expires June 2025, but that card was just created today, so I think that customer must have realized what happened and updated their card.

It's still definitely buggy that the reason never makes it back to the customer! Improving the error message seems S2-level to me. I should be able to have a fix in the next couple days.

[lbwright22]

Sounds great @andrewpbrett . Thank you for this hard work.

The customer who placed order R568361327 got in contact to say that they updated their credit card, which corresponds to your findings.

From talking to customers the 'default' card toggle is a problem or at least that section of the OFN account for shoppers (the left hand column where they can add and list cards is). I'm wondering if it is worth (at some stage) reviewing the UX of this?

(I may be wrong: I know I am making a big assumption that when you say that two of the customers have multiple cards attached to them that this is due to the way they have added lots to their OFN account rather than a fault in which our system has stored data. It may be very much more ingrained than my assumption, in which case the UX of that part of the account is not so much of an issue.)

[lin-d-hop]

Hi @lbwright22 Can you clarify how that wishlist item is related? Also, its not standard ecommerce to delete cards on behalf of the user. It does sound like a good idea to do a UX review on this aspect of the app though let's not jump to conclusions on actions to take ;-)

[sigmundpetersen]

I guess it's the openfoodfoundation/openfoodnetwork#8897 ?

[andrewpbrett]

@lbwright22 I took a look at how we handle error messages on subscription payments, and from what I can tell if we get a message from Stripe we should be passing it along to the customer in the failed payment email. This only goes to the customer, however, and not the hub. The hub should, in theory, see the specific error (that the card has expired) in the summary email.

Since I don't see anything that's obviously wrong in the code, and since it does look like we intend to send the specific error message to both the hub and the customer in these emails, could we maybe try to track down those emails and see what they look like? If it's possible to reach out to the hub directly, screenshots would help. Or if not, I can also try taking a look at SendGrid logs, but I think they only keep them for 7 days (and I'll need to wait until someone from the UK can get me in with 2FA).

[lbwright22]

Hi @andrewpbrett One of the customer's for whom the payment failed last week also failed again this week (order ID R731805748). The others went through. The card registered to this customer's Stripe expires in 2024 and so is this a case where they had a card which was due to expire on 1st July, they updated on OFN but it hasn't linked with the update on Stripe in some way?

Screen shots of emails:

All error messaging has been missing from hub emails from 7th June onwards. Before then, emails sent to hubs looked like:

[andrewpbrett]

Thanks for those screenshots @lbwright22 - very helpful!

I looked at Order R731805748. The default card for that customer is listed, in the OFN database, as expiring in June 2021. I also noticed that the database record was created in March 2020 but updated 30 June 2021, presumably because it was about to expire. So that makes me suspect that there might be something going wrong when the customer updates their card on the OFN side, like maybe we update the number but not the expiration date, and the mismatch causes the payment not to go through. I also noticed that this particular user has 39 (!) cards added to OFN - I don't know whether this is a cause or maybe just a symptom.

I do see the same as you're seeing on the Stripe side, that there is a single card listed for this customer and that it expires in 2024. I didn't see any log messages from Stripe this time saying that the card had automatically been updated - does anyone from support ever add or update a card directly in Stripe?

I do still think that our short term fix should work, i.e. that if this particular customer deletes their default credit card on the OFN side and re-adds it, payments will go through for them.

[lin-d-hop]

I am still confused about this one. We have an increasing number of subscriptions that require manual payment each week. The payment fails through OFN but then successfully completes when I make the payment manually via Stripe.

The work around of asking the user to login and reupload their card details is not an overly practical solution for subscriptions. The whole point of subscriptions is that users set it up and forget about it.

This s2 has been open forever. I'd really like us to actually get to the bottom of it, but while Stripe is so broken this feels lower on the list. Closing for now.

[lin-d-hop]

Closing this but it does have relevant information for #8534