FR - Growing latencies on copy receipts

[mertcanyalhi]

Summary

Although the recent change (#192) improved the performance of the signature requests for copy receipts, we're observing an increasing trend for copy receipt latencies, which started to cause performance issues in our integrations.

We have outlets with >200K queue items and a CNumerator of nearly 5K. We noticed that the latencies have direct correlation with the number of queue items, which can be seen in the table below.

The table shows the correlation of copy receipt latency (red, left y-axis) and number of queue items (yellow, right y-axis) for multiple outlets (x-axis):

Impact

The impact can be seen in the sub-sections for each store. Each sub-section shows the queue size, CNumerator, and a latency heatmap for:

• All receipts - signature requests for all receipts
• Copy receipts - signature requests for copy receipts
• Other receipts - signature requests for non-copy receipts

TL;DR:

• Copy receipt latency has a direct correlation with the queue size.
• While the copy receipt request is being processed, it blocks the queue and impacts latency of other receipt requests.
  • This is expected due to the blocking nature of the queue; the main idea is to highlight the spikes in other receipt types are actually a result of copy receipt requests.
• Growing copy receipt latencies can be clustered in 2 groups;
  • First group that has higher latency
  • Second group that has lower latency, which is mainly a subsequent copy receipt request. Subsequent copy receipt requests that are made in a certain window is always lower. This can be the result of an index cache on the data layer

Store A

• Queue size: 216K
• CNumerator: 4.7K

Latency heatmap for all receipts:

Latency heatmap for copy receipts:

Latency heatmap for other receipts:

When we focus on the copy receipt latencies, it can be seen that the latency of the subsequent copy receipt requests that are made in a certain window is always lower. This can be the result of an index cache on the data layer. The table below shows copy receipt requests that are grouped in equal time windows:

Findings:

• The latency heatmap for copy receipts indicates that the latencies can be clustered in levels; 4-5s and 10-15s
• The latency heatmap for other receipts indicates that the latencies are mostly in the same level (~500ms).
  • Latencies of other receipts seem to be similar (~500ms) for all receipt types. This can also be seen within table under the summary section.
  • In some cases, an increasing latency can be observed for other receipts. This is a result of the previous copy receipt request blocking the middleware while it is being processed.

Store B

• Queue size: 194K
• CNumerator: 3.7K

Latency heatmap for all receipts:

Latency heatmap for copy receipts:

Latency heatmap for other receipts:

Findings:

• Same as store A.

Store I

• Queue size: 42K
• CNumerator: 0.9K

Latency heatmap for all receipts:

Latency heatmap for copy receipts:

Latency heatmap for other receipts:

Findings:

• Copy receipt request latencies are on the same level (~2.4s-3.1s).
• Other receipt request latencies are on the same level (~500ms) with spikes up to ~2.8s that is a side effect of the copy receipt request that is processed in the background.

Expectations

As SaaS and PaaS providers, our expectation is to have stable latencies regardless of the queue size. Growing queue size will cause delays or even interruptions in receipt printing, and as a result, degraded customer experience.

[mertcanyalhi]

Just wanted to check if there is any update, or anything we can help with 🙏

CC: @TSchmiedlechner @volllly

[TSchmiedlechner]

Hi @mertcanyalhi, thanks a lot for these very detailed analysis, very appreciated! Indeed it looks like the scaling of the intermediate fix we had deployed for #192 did improve the situation, but still doesn't scale perfectly - although the rate of the performance deterioration became much slower, that's very obviously not enough.

We've implemented a second fix that decouples the copy receipt performance from the number of processed receipts and should change the complexity from O(n) to O(1). We're currently working on deploying this, ETA is somewhat about ~2 weeks in my opinion - I'll keep you posted. Sorry about the inconvenience, and again, thanks for the super helpful problem description. 😊

[mertcanyalhi]

Hey @TSchmiedlechner, thanks for the information and your efforts 🙏 please let me know if we can provide any further input or help you in any way