investigate 402s

[chadwhitacre]

Two recent reports of 402s when trying to add a credit card:

http://gittip.freshdesk.com/helpdesk/tickets/414 http://gittip.freshdesk.com/helpdesk/tickets/433

[dcwalk]

I just had a 402 myself... failed on Gittip 100. Would be nice if there was some way to make these failures less silent (akin to #2211 ) as I only discovered it by accident.

Resolution was to re-enter the same CC and it is now indicated as working. Will see if it 'takes' next payday.

[patcon]

cc: @derhechi

FYI, not documented in @balanced docs, but found in Strip docs that 402 is:

402 Request Failed - Parameters were valid but request failed.

https://stripe.com/docs/api#authentication

[patcon]

Sorry to pester you @steveklabnik, but any thoughts on this? (Feel free to cc someone else in, just wasn't sure who to @mention :)

@whit537 do we log Balanced API errors anywhere? Balanced doesn't seem to log failed api calls, so I guess it's up to us, but not sure how to track it down if we do it.

[steveklabnik]

I can't see in the helpdesk, so I have very little context.

HTTP 402 is "Payment Required." It happens when a charge fails.

[patcon]

+1 from user experiencing this in http://gittip.freshdesk.com/helpdesk/tickets/465

[chadwhitacre]

I wonder if this is related to Ghostery (#1669).

[chadwhitacre]

Debugging hint: look for relevant usernames in Sentry?

[chadwhitacre]

+1 on FD498.

This is getting bad. :-(

[Boran]

• as a user. I tried to add money with mastercard yesterday, and got a "402 Payment required" on 2 attempt. This will cost customers :-). The error messages need to be much friendlier and more precise... maybe a button to "Report and followup" would be nice?

[chadwhitacre]

Sounds like we have two separate problems:

1. @jpcaissy (465) is hitting So sorry!.
2. @hirojin(?) (414), @derhechi (433) , and @Boran (498) all report 402 Client Error: PAYMENT REQUIRED).

Re: (1). I've asked @jpcaissy via Freshdesk if he has Ghostery installed (#1669). If that's not the explanation then let's reticket, keeping this ticket for the 402s.

Re: (2). A clue: @derhechi reports getting the error "even before I enter any credit card details." Also interesting that we don't have any Balanced request id like we have for other Balanced error conditions. Also interesting that it's specified as a "Client Error."

Hey @matthewfl, any insight on the conditions under which balanced.js would throw a 402 Client Error: PAYMENT REQUIRED)?

Where is the error propagating from? Are we getting it from a client-side call to Balanced or a server-side call to Balanced?

[mjallday]

@whit537 revision 1.1 of balanced.js should not generate 402s since it doesn't attempt to verify the card, it just stores the details for later retrieval.

do you have any request IDs or can you log the request IDs if not? we can use these to take a closer look at the exact requests that are coming though. the request ID is returned in a header called X-Balanced-GURU and begins with OHM.

[jpcaissy]

I had Ghostery installed, and I have just whitelisted gittip. I was finally able to add my credit card.

[matthewfl]

@whit537 the 1.1 version of balanced.js that gittip is using should only report an error on basic validation issues, eg the card doesn't pass luhn check or the date is expired. However, I think that these errors would be reported as a 400 because the payload is invalid. The 402 might be coming from when the card is validated against the card network which happens when you being to operate on it server side. (eg here). if you know the card identifier then we might be able to look into the issue and see what is happening.

[chadwhitacre]

There are 420 participants with a last_bill_result of 402 Client Error: PAYMENT REQUIRED).

[chadwhitacre]

@jpcaissy Cool, thanks for the confirmation. :-)

[chadwhitacre]

Those 420 402 Client Error: PAYMENT REQUIRED)s represent 65% of our bad credit cards (649 total) and 14% of our active users (2916 total).

We definitely get 402 Client Error: PAYMENT REQUIRED) during payday (I just acked the payday logs), so this isn't a problem unique to connecting a card for the first time.

[msherry]

@whit537 You'll get that for every failed debit (or credit) -- it's what Balanced returns when a transaction is declined for any reason, e.g. insufficient funds. The body of the response should have more information as to what exactly was wrong.

[chadwhitacre]

@mjallday Ah, okay. I'm seeing a request id inline in the error message for 97 out of 649 failing cards (15%). Did the request id get moved to a header with version 1.1 of the API? I've reticketed logging that as #2441.

[chadwhitacre]

We record last_bill_result in two places:

1. When we first associate a card (as @matthewfl points out), and
2. ... during payday.

We don't differentiate between those two places in our logging, and we should.

The body of the response should have more information as to what exactly was wrong.

Currently in both contexts we're pulling the error message out of err.message.message. I'm hearing from @msherry that there's additional info in other attributes that we should also record.

Afaict the bottom line here is that we don't currently have robust-enough error logging to properly debug these 402s. I've reticketed more comprehensive improvements to our logging around exchanges as #2443.

[chadwhitacre]

Aaaaaand we don't have enough info to debug the specific 402s that @hirojin, @derhechi, and @Boran are seeing. :disappointed: Amiright?

[chadwhitacre]

@dcwalk Looks like you were able to get yourself back on track, at least? I've added +1s for you to #583 and #1746.

[Boran]

FYI I just tried to add my CC once again, same result, "402 Client Error: PAYMENT REQUIRED". Perhaps if additional logging was added in the meantime you can see something useful?

[mjallday]

Here's what I'm seeing for @Boran's card:

{
  "errors": [
    {
      "status": "Payment Required",
      "category_code": "card-declined",
      "additional": "This transaction was declined by the card issuer. Customer please call bank.",
      "status_code": 402,
      "category_type": "banking",
      "extras": {},
      "request_id": "OHMb26256b4e74a11e3ac9102b12035401b",
      "description": "349: This transaction was declined by the card issuer. Customer please call bank. Your request id is OHMb26256b4e74a11e3ac9102b12035401b."
    }
  ]
}

IIRC this is a decline from the issuing bank for this card, not from Balanced. The response is coming back from the underlying processor with the very generic "Do Not Honor" message. @msherry any suggestions for getting this card to go through? i'm pretty sure it's outside of Balanced's control but maybe you have a better idea than I do?

it looks like csc and address are both being passed through tho the processor is returning an AVS unsupported message so I can't tell if that's a match or not.

[msherry]

A 349 is the second-most frustrating declination message we get, because there is so little information in it (as we can see here). In the past we've seen that passing full name, address and CSC information can help reduce these slightly, but if those are already being passed and the card is still being declined, one really has to take the error message's advice and have the customer call their bank to straighten things out.

[Boran]

Well I did give a full name and address and use this card many times a year. I've never had to call the bank in the ten or more years I've had this MasterCard. However if it would help here I can do so. On 29 May 2014 19:27, "Marc Sherry" notifications@github.com wrote:

A 349 is the second-most frustrating declination message we get, because there is so little information in it (as we can see here). In the past we've seen that passing full name, address and CSC information can help reduce these slightly, but if those are already being passed and the card is still being declined, one really has to take the error message's advice and have the customer call their bank to straighten things out.

— Reply to this email directly or view it on GitHubhttps://github.com/gittip/www.gittip.com/issues/2372#issuecomment-44559114 .

[Boran]

P.a. perhaps it had something to do with my being based in Switzerland. On 29 May 2014 20:59, "Sean Boran" sean@boran.com wrote:

Well I did give a full name and address and use this card many times a year. I've never had to call the bank in the ten or more years I've had this MasterCard. However if it would help here I can do so. On 29 May 2014 19:27, "Marc Sherry" notifications@github.com wrote:

A 349 is the second-most frustrating declination message we get, because there is so little information in it (as we can see here). In the past we've seen that passing full name, address and CSC information can help reduce these slightly, but if those are already being passed and the card is still being declined, one really has to take the error message's advice and have the customer call their bank to straighten things out.

— Reply to this email directly or view it on GitHubhttps://github.com/gittip/www.gittip.com/issues/2372#issuecomment-44559114 .

[mjallday]

thanks for the info about the country @Boran. @msherry for @Boran's card I'm seeing the incorrect country code being passed through. could that hinder the auth?

this is in the precog logs going down to knox:

{
...
 "address": {
   "city": null,
   "region": "VD",
   "postal_code": "1807",
   "street_address": "<REMOVED>",
   "country_code": "USA"
 },
...
}

[msherry]

Incorrect address information can lead to higher decline rates, absolutely. 349 is basically the issuer's fraud/risk systems kicking in and rejecting the transaction, so any information that's incorrect (like a nonsensical US postal code, or a missing city, as in this example) will probably make this transaction more likely to be flagged as fraud/risky.

[mjallday]

@whit537 are you able try and get the payload to contain a revision 1.1 compliant payload for tokenization? i'd like to try tokenizing the card with the country code correctly passed through so it doesn't default to USA.

the original tokenization payload you sent through looks like

{
...
      "address": {
        "state": "VD",
        "postal_code": "1807",
        "line1": "<REMOVED>"
      }
...
}

what you want it to look like would be (documentation)

{
...
            "address": {
                "city": null, 
                "country_code": "CHE", 
                "line1": "<REMOVED>", 
                "line2": null, 
                "postal_code": "1807", 
                "state": "VD"
            }, 
...
}

looks like the country code field is the only thing missing. alternatively you can omit the address data entirely. i'm not sure if that would increase the likelyhood of verifying the card but i do think it will work better than passing (inadvertently) incorrect address data. maybe balanced should not default the country code to USA as well.

[dcwalk]

@whit537 I am back on track: Re-entered the same card and a payment went through in payday 102 ( #2399 ).

I am based out of the US (in Canada) perhaps that was the reason in my case as well...? Regardless there has been no other issues with the Card outside of gittip and I'm back using it here as well.

Edit: Thanks for adding my comments to the related issues around error notification. I think better notifications around this is extremely important!

[chadwhitacre]

@mjallday Where did you find that error json? I looked on @Boran's customer page on the Balanced dashboard but I'm seeing "No results" for activity and "No results" for logs.

are you able try and get the payload to contain a revision 1.1 compliant payload for tokenization?

Reticketed as #2446.

[chadwhitacre]

Well I did give a full name and address and use this card many times a year. I've never had to call the bank in the ten or more years I've had this MasterCard. However if it would help here I can do so.

P.a. perhaps it had something to do with my being based in Switzerland.

@Boran I know that @daankortenbach, who is based in the Netherlands, had to call his bank to convince them to allow Gittip's charges to go through. Sorry for the trouble. :-(

[msherry]

The fact that gittip seems to be essentially hardcoding the address country to 'USA' surely isn't helping here.

[chadwhitacre]

The fact that gittip seems to be essentially hardcoding the address country to 'USA' surely isn't helping here.

Picking up with that on #2448.

[Boran]

Ok, tell me when the country has been fixed and I'll try again. Seems like a major issue if only US donations can be accepted..

[seanlinsley]

2448 has now been deployed, so please give it another try once you get the chance.

[Boran]

I tried again, same error.

Sean

On 31 May 2014 01:17, Sean Linsley notifications@github.com wrote:

2448 https://github.com/gittip/www.gittip.com/pull/2448 has now been

deployed, so please give it another try once you get the chance.

— Reply to this email directly or view it on GitHub https://github.com/gittip/www.gittip.com/issues/2372#issuecomment-44709557 .

[chadwhitacre]

Here's what I'm seeing for @Boran's card:

@mjallday Where are you seeing this? When I search "boran" in the dashboard I get no customer records. The customer record linked from the Gittip account has no cards attached nor any name info.

[mjallday]

@whit537 i looked at the log data for that card. currently we're not associating cards to the marketplace when they cannot be verified but i think that's the wrong thing to do and needs to be changed.

I'm seeing this come through again with the super generic "Do Not Honor" from the issuing bank. @Boran did you manage to call the bank and ask them if there was a reason for the decline?

All the address data that we're passing through to the bank looks correct on this end.

[chadwhitacre]

Thanks @mjallday.

[Boran]

I called the bank, my card was blocked. After unblocking, I can now use the card, even for gittip :-) So issue fixed, from my point of view.

[chadwhitacre]

Awesome! Thanks for your persistence, @Boran! :D

[ghost]

Anti-fraud systems flagged and blocked Gittip at my Visa provider because of the many small amounts. After calling they unblocked it for me.

[chadwhitacre]

+1 on FD571.

[chadwhitacre]

Closing. We've got better logging for 402 failures now, with #2508.

[encukou]

I still get "402 Client Error: PAYMENT REQUIRED" when trying to connect a card. Not sure if it's logged anywhere.

[chadwhitacre]

@encukou Have you tried calling your bank? Some banks have blacklisted Gratipay and need to hear from their customers in order to whitelist us. Sorry for the trouble. :-/