Unhandled Error Editing Cart - Deadlock

[lin-d-hop]

Description

Regression with Bye Bye Spree

On editing cart the user sees the slug, when due to a deadlock on an adjustment update.

It has impacted 16 users and occurred 33 times since the release in the UK. The Slug during the checkout process certainly makes users distrust the platform.

https://app.bugsnag.com/yaycode/openfoodnetwork-uk/errors/5fb38d570c4aff0017d95586?filters[event.since][0]=30d&filters[error.status][0]=open&pivot_tab=event

Expected Behavior

No slugs in the checkout process.

Actual Behaviour

Slug on editing cart

Steps to Reproduce

I am unsure. It is an edge case. Hopefully the Bugsnag will hold enough details. 1. 2. 3. 4.

Animated Gif/Screenshot

Workaround

Severity

bug-s2: a non-critical feature is broken, no workaround bug-s3: a feature is broken but there is a workaround

I am unsure whether to give this s2 or s3. You can workaround if you keep trying, however shoppers have reported having to try a number of times. As this has been introduced by Bye Bye Spree and is shopper facing during checkout process it might qualify as s2..?

Your Environment

• Version used: 3.4.1
• Browser name and version:
• Operating System and version (desktop or mobile):

Possible Fix

[Matt-Yorkley]

We do have some serious issues around deadlocks, but I'm not convinced they're related to the Bye Bye Spree changes, I think they're triggered with higher shopper activity. There's patterns of spikes going back to October that look similar to the spikes over the last week.

Totally agree it needs to be fixed though :100:

[Matt-Yorkley]

Anatomy of a Deadlock

• Two processes are acquiring locks on the same two associated resources, but in reversed order, eg: process 1 is trying to update order->line_item and process 2 is trying to update line_item->order (with associated records).
• Alternatively, two processes are trying to update multiple records at once, but in an indeterminate order, eg process 1 is trying to update [product1, product2, product3] and process 2 is trying to update [product2, product1, product3].

Possible Solutions / Next Steps

• Try some deadlock debugging tools to get more info and find the specific code creating both sides of the locks. Possibly not needed, if we can get enough info from Bugsnag, logs, and APM.
• In some cases; if the order in which locks are acquired in two different bits of code is altered so they both use the same order, the deadlock will not occur.
• Updates triggered in callbacks can be a major source of locks. Any additional transactions happening inside callbacks are also wrapped inside the parent transaction, extending it and making it potentially blow up. Moving logic from #after_create to #after_commit for certain callback code (where possible) can fix/improve it, as the additional updates are handled after the lock on the origianl transaction is released.
• Alternatively, the processing of certain transactions can be passed into a queueing system, but whether or not we want to do that and whether or not it makes sense in each specific case is TBD...

Picking off the low-hanging fruit from these available options might get us some relatively quick wins on the nice end of the ease-value matrix...

[Matt-Yorkley]

This is another "it will get worse as the holiday season ramps up" issue. :santa: :christmas_tree: :fire:

[Matt-Yorkley]

There are two different sets of deadlocks here, and they've affected 54 users on 82 occasions in the last 3 weeks on UK Prod.

https://app.bugsnag.com/yaycode/openfoodnetwork-uk/errors/5f98a42344b486001883e959?filters[event.since][0][type]=eq&filters[event.since][0][value]=30d&filters[error.status][0][type]=eq&filters[error.status][0][value]=open&filters[app.release_stage][0][value]=production&filters[app.release_stage][0][type]=eq

https://app.bugsnag.com/yaycode/openfoodnetwork-uk/errors/5fb38d570c4aff0017d95586?filters[event.since][0][type]=eq&filters[event.since][0][value]=30d&filters[error.status][0][type]=eq&filters[error.status][0][value]=open&filters[app.release_stage][0][value]=production&filters[app.release_stage][0][type]=eq

[lin-d-hop]

Yeah we have big deadlock issues. They seemed to get worse with the rollout... but you are right it might be coincidence with increased load - xmas and lockdown.

Perhaps we can talk about this in delivery train tonight?