Clarify abandoned payments

[artfulrobot]

If someone fills in a CiviCRM Contribution form and clicks through to the GoCardless external website but then abandons the process, CiviCRM is left with a "Pending"/incomplete Contribution and Contribution Recur record.

However, if they complete the process, the contrib and contrib recur records still show this in their status, until the first payment has been successfully claimed, at which point the recur record is set to status: "In Progress"

It might be clearer for CiviCRM admins to see the recuring contribution set to "In Progress" at the time that the mandate and subscription were successfully set up, since then any "Pending" records that are older than an hour or so could be assumed to be abandoned processed, and could be ignored (or have their status set to cancelled or such).

Would this help you? Does anyone want this feature? Might anyone want to contribute to making it happen? I think there's perhaps 3 hours work in it, including updating the test code, documentation etc.

[artfulrobot]

Proposed change 1: Change the status of the Recurring Contribution record to "In Progress" as soon as the mandate has been successfully completed.

I think this is a good thing to do; I can't really see a problem with doing it, I'll declare it as a change in the README in case anyone was relying on the old behaviour. Please comment here if you think this will be a problem for you

✔ Benefits: you can distinguish between mandates that have been successfully setup (but just haven't had a payment yet - it takes about a week with Direct Debit) and those that haven't been properly setup.

Proposed change 2: Scheduled job changes status from Pending to Cancelled

If a contribution recur record is set up (Pending) then we'd expect the user to enter their card details shortly after. If they've not done so after -say- 30 mins or 1 hour, then we can assume they've abandoned it.

A scheduled job could look for Pending Contributions attached to a GoCardless Pending Contribution Recur record that are older (at the time of running the job) than 1 hour, and change the Contribution status to Cancelled and the Contribution Recur status to Abandoned.

✔ Benefits: we now have a way of following up 'abandoned' carts and distinguishing between recurring contribs that worked for a while but then ended (Cancelled) and those that were never setup fully.

So the abandoned case looks like this:

Your input please

• Would this cause you any problems?
• Would this help you (at least 2 of my clients are interested)?
• Would adding the "Abandoned" status to Contribution recur records be OK? It's not a standard, core status, so the extension would need to make sure this existed itself. Is that OK?
• Should we go further and mark the contribution as Abandoned, (instead of Cancelled, as proposed) too? What benefit would that give?

[Upperholme]

Hi. This sounds like a good step forward in clarifying the state of play of contributions. I can't see it causing problems in the (so far) couple of implementations where I'm using GoCardless. In my opinion 'abandoned' is a better description of the status of the contribution record. This might also enable easier identification and differentiation between contributions that have been explicitly cancelled and those that have simply been abandoned, which in turn might ease reporting and follow up action. Any scheduled job that is cleaning up would need to be able to differentiate records by payment processor so that it didn't inadvertently pick up/change records relating to other payment channels.

I don't believe that either of the organisations that I'm working with using this extension would be willing/able to make any significant contribution by way of funding this development. Both are processing a low number of transactions through GoCardless and aren't facing any issues that this proposal would affect in any substantive way.

[artfulrobot]

@Upperholme Thanks for your input Graham.

Any scheduled job that is cleaning up would need to be able to differentiate records by payment processor so that it didn't inadvertently pick up/change records relating to other payment channels.

Quite! I'll edit the proposal to clarify that's what I meant.

[simonjohnparker]

Hi @artfulrobot

Very keen for this to help us see when a user doesn't complete the payment.

If a contribution recur record is set up (Pending) then we'd expect the user to enter their card details shortly after. If they've not done so after -say- 30 mins or 1 hour, then we can assume they've abandoned it.

I'm just thinking in the rare scenario that maybe the user does still want to complete, but has just left the GC form hiding in one of their tabs. If the civi status goes to abandoned, and they come back to it say 1.5hr later to finish. How would Civi react in this scenario if the user would come back and complete the form or does the GC user input page have any kind of timeout its self?

Happy to fund this.

[artfulrobot]

@simonjohnparker Hmmm.

You could ask GC: Is there a time limit between creating a redirect flow and completing it?

I think (it would need testing) what would happen would be

1. The ContributionRecur record would be updated as normal - set to In Progress.

2. The Contribution record would stay as Cancelled, and a new one created when the payment came in, since when a payment comes in it looks for a Pending one (which it won't find) and if not found it creates a new one.

But as I say, this is an edge case that would need detailed testing once we have these new features in place.

[JoeMurray]

tl;dr I didn't review this in detail.

My focus is on maintaining proper handling of accounting entries in CiviCRM. From that perspective, a contribution being changed from Pending to Cancelled, Failed and Abandoned are all synonyms for the same bookkeeping entries. From my perspective it would be best if a contribution that had one of those three statuses never transitioned to a different status, but I may have previously relented, or the as-built code might not enforce that. It's not easy to implement a new Contribution Status either in core or an extension so that accounting data is not corrupted. The https://github.com/JMAConsulting/biz.jmaconsulting.customcontribstatuses extension allows them to be changed for organizations that don't care about accounting data in CiviCRM. With effort it might be possible to enhance core and / or the extension to support hooks on the transitions into and out of custom contribution statuses. This is desirable and would be quite useful, but I don't think it will be easy. It hasn't been done up until now due to resource limitations. A relatively safe change would be to replicate the handling of Cancelled / Failed for a new status of Abandoned. This new status would be useful for users to understand what has gone on, and probably could be the basis for new workflows to support re-submission etc.

I don't think it is a best practice for some generally relevant changes to complex core workflows around accounting to be done just for a single payment processor. How to handle abandoned payments is a problem for many, maybe most of the payment processors integrated with CiviCRM. I also think it is important that payment processor extensions not break core financial accounting functionality.

Could you do the work to handle the abandonment in a separate extension?

I haven't commented on Contribution Recur status since they don't directly affect accounting entries.

Good luck.

[artfulrobot]

thanks @JoeMurray for sharing your considerable experience! much appreciate!

Am I right in thinking that if we simply change the status of Contributions to Cancelled using the API then there's no corruption of the accounts stuff?

If so, let's just do that for now.

And if the recurring contribution status does not affect the accounts stuff then that's sounding like there's nothing stopping is going ahead with that.

[JoeMurray]

Yes, setting Contributions to Cancelled is fully supported by API and if there are any problems we will fix. We're trying to encourage a move to using Order API and Payments API, but I am not certain how that maps to your work.

Recurring contributions only affect accounting via associated contributions, so I think this approach is good.

[artfulrobot]

Ha! on inspection, I believe that Proposal 1 is the original behaviour!

[simonjohnparker]

Hi Rich,

Ha! on inspection, I believe that Proposal 1 is the original behaviour!

Is it?!

I'm not sure that's happening with ours, contributions have remained in pending state.

[artfulrobot]

Hi @simonjohnparker pls email me a link to such a contact - are you sure that's not an abandoned donor?

• pending contribution, pending recurring contribution = abandoned

• pending contribution, In Progress recurring contribution = DD set up but first payment not come in yet

• pending contribution, Cancelled recurring contribution = DD cancelled before first payment

[artfulrobot]

There's now a diagram showing the lifecycle of Contribution Recur and Contribution records:

link

From this you can see I've suggested that if a user abandons the process then the Contribution Recur record gets changed to Failed, instead of the proposed "Abandoned". The Contribution itself is set to Cancelled. This is done with a scheduled job after 24hrs of being in state Pending.

• Failed is a CiviCRM built-in status, which is a good thing; as we learnt ↑, these statuses result in special processing when applied to contributions and because they share the option set (at the mo) with recurring contributions, adding an unsupported one could cause serious problems if people were to use it on contributions.

• Failed is not used for Contribution Recur records by this extension for anything else (if a payment failes we use Overdue instead) - therefore we can know that Failed CRs means abandoned.

• Failed also works for users who didn't "abandon" it out of choice - e.g. their data was rejected (e.g. bank disallows Direct Debits, or such).

Conclusion

With the changes including the new scheduled job, we now have a way to identify 'abandoned' signups: search for Recurring [GoCardless] Contributions with status Failed.

I propose to close this now and releasse this (along with many other improvements) as v1.8.

[simonjohnparker]

Hi Rich,

Hi @simonjohnparker pls email me a link to such a contact - are you sure that's not an abandoned donor?

Yes was referring to this. But, if my understanding is correct, Cancelled status is automatically applied if the DD set up fails, and you are going to implement Failed status for those that abandon.

Which I'm supportive of.

Thanks.

[artfulrobot]

@simonjohnparker I think we're on the same page but just to be super clear (?!)...

"abandoned" here means users who started the process of setting up a mandate online but stopped on the GoCardless page that asks for their bank details.

The Recurring Contribution record for such a person will show: Pending until it's 24 hours old and a cron job runs which will now mark it as Failed, and the Contribution (as opposed to the Recurring Contribution) is then marked as Cancelled.

The Recurring Contribution record will only be marked as Cancelled if it was set up successfully at first, but then cancelled (e.g. by the bank) later.

I'll close this now.