Closed smadbe closed 4 months ago
Here's the current investigations. For now, still no clues.
service
, it takes 250sconsole
, it takes 0.2sservice
and by the console
are the sameconsole
with SQL_NO_CACHE
, it still takes 0.2s onlyservice
concurrently, with get a deadlock
, because of the FOR UPDATE
console
concurrently (tried with 10), we get the result in ~0.2s per query, one result after another because of the lock FOR UPDATE
service
is called and then we call it in the console
at the same time, we get deadlock
, as expected by the FOR UPDATE
service
and console
: it's in a transaction. But if cal the query inside a transaction in the console
, we also get the result in 0.2s.service
before the big query, in the same order, in the console
, we also get the result in 0.2sFOR UPDATE
and call the service
: 260sFOR UPDATE
and call the service
, and concurrently in the console
, we get the result of the service
in 260s and the result of the console
in 0.2s, not deadlock
anymore because the lock is removedFOR UPDATE
and call the service
concurrently two times, we get the result after 360sORDER BY
to try: it's the sameint64
so I doubt it comes from this), or encoding when using text (but the query doesn't use any text field)Note: The lock FOR UPDATE
is questionable. Do we really need to lock the rows of all items in the path for ALL SELECT
statements on items of other queries in the app while this query is running? It's possible to lock only for UPDATE
with FOR SHARE
. This would make other queries selecting items in the path return their result, and only block queries that want to update those items.
Thanks for this analysis.
How do you explain the deadlock with the transaction? Anyway we should avoid having deadlocks at all cost (even if that's probably not the cause of the issue as the CPU is fully used during the 250s). So having some extra waiting because of that lock is normal but the deadlock is a bug.
with update
and then create the attempts... but with FOR UPDATE
, it should let the transaction complete its insert.resultStore.ScheduleResultsPropagation()
? it schedules a run of ResultStore::propagate() on transaction commit.
, cannot it cause a problem with the other transaction ?The lock FOR UPDATE is questionable. Do we really need to lock the rows of all items in the path for ALL SELECT statements on items of other queries in the app while this query is running?
The INSERT (to create the attempt path) is not supposed to be a long query and it is important that no other attempt path is created at the same time, so letting the parallel start-result-path
working on the same rows (so for the same user and item) wait looks like a good idea!
It's possible to lock only for UPDATE with FOR SHARE. This would make other queries selecting items in the path return their result, and only block queries that want to update those items.
If I understand well you proposal, it would mean that you may have this scenario if 2 service calls are done "at the same time": the service-call-1 would do the select, detect what is missing, then the service-call-2 would do the select, detect what is missing... and then service-call-1 would do the INSERT... followed by the service-call-2 doing the same insert, so we might duplicate and so issues.
So that seems important to understand the deadlock as this should not happen according to my understanding (so that means my understanding is wrong). I am not sure that is related with the 250s... but that's possible.
For the deadlock again (and maybe for the timing): Probably you could try to remove the latest queries from your service to see if it changes something:
resultStore.ScheduleResultsPropagation()
resultStore.InsertIgnoreMaps("results_propagate", rowsToInsertPropagate)
resultStore.InsertOrUpdateMaps(rowsToInsert, []string{"started_at", "latest_activity_at"}
For now I put the lock on the side because it's not the cause, we can come back to it later.
I continued the investigations, and found something interesting:
instantly
, but in a different format.Result when the query is constructed in the service (slow):
result: []map[string]interface {}{map[string]interface {}{"attempt_id0":0, "attempt_id1":0, "attempt_id2":0, "attempt_id3":0, "attempt_id4":0, "attempt_id5":0, "attempt_id6":0, "has_started_result0":1, "has_started_result1":1, "has_started_result2":1, "has_started_result3":1, "has_started_result4":1, "has_started_result5":1, "has_started_result6":1}}
We have numbers as a result.
Result when the query is pasted and executed in the service (fast):
result: []map[string]interface {}{map[string]interface {}{"attempt_id0":"0", "attempt_id1":"0", "attempt_id2":"0", "attempt_id3":"0", "attempt_id4":"0", "attempt_id5":"0", "attempt_id6":"0", "has_started_result0":"1", "has_started_result1":"1", "has_started_result2":"1", "has_started_result3":"1", "has_started_result4":"1", "has_started_result5":"1", "has_started_result6":"1"}}
We have strings as a result.
My best bet now is that it's due to the way the parameters are binded in the sql package, and that somehow the int64 are converted into strings, and that's what makes it very slow. But it's just a guess, I haven't investigated the idea further yet. I'll try next week.
Other things I tried:
Ongoing investigation:
is timeout after 15sec.
Geoffrey, launching it from his computer, has the request completing after 250sec.
Most of the time is spent on:
but that request is taking 0.2sec to run in the MySQL CLI. 🤷♂️