Closed finestructure closed 3 weeks ago
Another one from last night:
| | 2024-06-06 00:33:47.400 | *** Signal 11: Backtracing from 0x56187107d031... done *** |
| | 2024-06-06 00:33:47.400 | |
| | 2024-06-06 00:33:47.400 | *** Program crashed: Bad pointer dereference at 0x0000000000000008 *** |
| | 2024-06-06 00:33:47.400 | |
| | 2024-06-06 00:33:47.401 | Thread 10 "NIO-SGLTN-0-#1" crashed: |
| | 2024-06-06 00:33:47.401 | |
| | 2024-06-06 00:33:47.401 | rax 0x0000000000000000 0 |
| | 2024-06-06 00:33:47.401 | rdx 0x00007f4239df0900 98 ce 15 72 18 56 00 00 01 00 00 01 00 00 00 00 ·Î·r·V·········· |
| | 2024-06-06 00:33:47.401 | rcx 0x0000000000000003 3 |
| | 2024-06-06 00:33:47.401 | rbx 0xffffffffffffe708 18446744073709545224 |
| | 2024-06-06 00:33:47.401 | rsi 0x00007f423c496f40 00 00 00 00 00 00 00 00 80 ce 10 44 42 7f 00 00 ·········Î·DB··· |
| | 2024-06-06 00:33:47.401 | rdi 0x00007f42269f5f40 c9 00 00 00 e4 00 00 00 ff ff ff ff 18 00 00 00 É···ä···ÿÿÿÿ···· |
| | 2024-06-06 00:33:47.401 | rbp 0x00007f42269f4e20 00 00 00 00 00 00 00 00 83 1e cf 70 18 56 00 00 ··········Ïp·V·· |
| | 2024-06-06 00:33:47.401 | rsp 0x00007f42269f30a0 00 09 df 39 42 7f 00 00 59 8b 0c 71 18 56 00 00 ··ß9B···Y··q·V·· |
| | 2024-06-06 00:33:47.402 | r8 0x00007f42442008c0 c8 00 00 00 c8 00 00 00 c8 00 00 00 c8 00 00 00 È···È···È···È··· |
| | 2024-06-06 00:33:47.402 | r9 0x0000000000000500 1280 |
| | 2024-06-06 00:33:47.402 | r10 0x0000000000000018 24 |
| | 2024-06-06 00:33:47.402 | r11 0x00007f42269f7640 40 76 9f 26 42 7f 00 00 10 a5 62 3a 42 7f 00 00 @v·&B····¥b:B··· |
| | 2024-06-06 00:33:47.402 | r12 0x00000000ffffffff 4294967295 |
| | 2024-06-06 00:33:47.402 | r13 0x00007f42269f38d0 65 6e 5f 55 53 5f 50 4f 53 49 58 00 00 00 00 00 en_US_POSIX····· |
| | 2024-06-06 00:33:47.402 | r14 0x00007f42269f31a8 00 90 16 72 18 56 00 00 65 6e 00 00 00 00 00 00 ···r·V··en······ |
| | 2024-06-06 00:33:47.402 | r15 0x00000000ffffffff 4294967295 |
| | 2024-06-06 00:33:47.402 | rip 0x000056187107d031 ff 50 08 48 89 df 5b e9 33 3a 1f 00 0f 1f 00 0f ÿP·H·ß[é3:······ |
| | 2024-06-06 00:33:47.402 | |
| | 2024-06-06 00:33:47.402 | rflags 0x0000000000010202 |
| | 2024-06-06 00:33:47.402 | |
| | 2024-06-06 00:33:47.402 | cs 0x0033 fs 0x0000 gs 0x0000 |
| | 2024-06-06 00:33:47.402 | |
| | 2024-06-06 00:33:47.402 | 0 0x000056187107d031 icu_69_swift::Calendar::~Calendar() + 33 in Run |
| | 2024-06-06 00:33:47.402 | |
| | 2024-06-06 00:33:47.402 |
I notice
0x000055b0ef753c00 icu_69_swift::TimeZone::~TimeZone() in Run
in the first and
0x000056187107d031 icu_69_swift::Calendar::~Calendar() + 33 in Run |
in the second. Both referencing icu_69_swift
🤔
The crash is in Swift's ICU copy, almost certainly because Foundation was reaching into ICU. I reckon the reason you don't get backtraces is presumably because Foundation hasn't been compiled with frame pointers, @al45tair is this expected? @finestructure what Swift version is this and where is it from?
@finestructure As a pretty uninformed guess this is likely because some object in corelibs-foundation isn't thread safe that should be thread safe. For example could it be this? https://github.com/apple/swift-corelibs-foundation/issues/3166
I assume you're using DateFormatters somewhere, are you sure you're not sharing them across threads?
Thanks for the quick response, @weissi !
This is swift:5.10.0-jammy
on x86.
We do use DateFormatters
in a couple of places, let me double check their situation.
I don't think we're sharing formatters, unless the following pattern is problematic
private extension DateFormatter {
static let iso8601Full: DateFormatter = {
let formatter = DateFormatter()
formatter.dateFormat = "yyyy-MM-dd'T'HH:mm:ss.SSSZZZZZ"
formatter.calendar = Calendar(identifier: .iso8601)
formatter.timeZone = TimeZone(secondsFromGMT: 0)
return formatter
}()
}
My understanding is that the static initialisation is guarded by something like dispatch_once
and ok - or could that in fact be the culprit? We're using that pattern quite a bit.
I don't think we're sharing formatters, unless the following pattern is problematic
private extension DateFormatter { static let iso8601Full: DateFormatter = { let formatter = DateFormatter() formatter.dateFormat = "yyyy-MM-dd'T'HH:mm:ss.SSSZZZZZ" formatter.calendar = Calendar(identifier: .iso8601) formatter.timeZone = TimeZone(secondsFromGMT: 0) return formatter }() }
My understanding is that the static initialisation is guarded by something like
dispatch_once
and ok - or could that in fact be the culprit? We're using that pattern quite a bit.
Yes, this is problematic, particularly on Linux. On macOS this is better but still difficult because DateFormatter
is a class
(https://developer.apple.com/documentation/foundation/dateformatter).
You're caching one instance of a DateFormatter
and then you're sharing it across threads. So if it's ever mutated anywhere, that'd be a problem.
You might want to change it to
static var iso8601Full: DateFormatter {
let formatter = DateFormatter()
formatter.dateFormat = "yyyy-MM-dd'T'HH:mm:ss.SSSZZZZZ"
formatter.calendar = Calendar(identifier: .iso8601)
formatter.timeZone = TimeZone(secondsFromGMT: 0)
return formatter
}
and see if that improves things. This will create a fresh one every time.
Ah right, of course. I don't think we mutate them but I better just make sure we don't use this pattern to rule it out. Thanks, @weissi !
FWIW, the DateFormatter changes on our end didn't have any noticeable effect on the crash rate.
FWIW, the DateFormatter changes on our end didn't have any noticeable effect on the crash rate.
Right. I appreciate that we're looking for needle in the haystack here but it might be worth looking into other places that share objects from corelibs-foundation that are related to date/time. Just because we know it's in the bundled ICU libraries in the TimeZone::~Timezone
destructor. I believe corelibs-foundation is the only client of icu_69_swift
...
There have been no backtraces in our logs over at least the last 7 days, if not longer (our logs don't go back further and I didn't check in a while). I do see a number of server restarts, many of which will be deployments and maintenance restarts.
In order to be sure this is fixed, I'm going to enable backtraces in prod where we have fewer deployments, making it easier to line them up with restarts.
Ok, definitely still there. Two crashes this morning: https://gist.github.com/finestructure/e0e8dedd9cc15648e2f0b3bdba054938
One of them is inside our dependency Plot
, which is also using the problematic static dateFormatter
closure:
internal extension RSS {
static let dateFormatter: DateFormatter = {
let formatter = DateFormatter()
formatter.dateFormat = "E, d MMM yyyy HH:mm:ss Z"
formatter.locale = Locale(identifier: "en_US_POSIX")
return formatter
}()
}
(We're already using a fork of Plot, so that's something we can easily address.)
I reckon the reason you don't get backtraces is presumably because Foundation hasn't been compiled with frame pointers, @al45tair is this expected?
Hmmm. Yes, probably. I think I turned them on for the Swift runtime itself, but I don't think I've done anything for Foundation. (rdar://132390751 tracks turning them on for Foundation.)
I think we've squashed them all now - there have been no crashes on either env since we deployed the latest fix. I'll close this for now, will reopen in case this pops up again.
Thanks again @weissi for your help!
I think we've squashed them all now - there have been no crashes on either env since we deployed the latest fix. I'll close this for now, will reopen in case this pops up again.
Thanks again @weissi for your help!
Great work!! And no worries, honestly I can mostly be replaced with a bot that says "threading bug" when people report somewhat non-deterministic or prod-only crashes in Swift programs ;)
We do see server restarts occasionally (a couple of times per day perhaps, across 3 prod nodes).
Example backtrace below, there's nothing I can glean from it.
Backtrace settings are: