Precondition Failed: Inconsistency error occurred in OrderedDictionary

[xanderdunn]

Thanks for creating and maintaining this library, it's been very useful to me.

I recently encountered the below crashing error in OrderedDictionary while running my server:

Precondition failed: Inconsistency error occurred in OrderedDictionary: file /home/xander/dev/my_project/.build/checkouts/OrderedDictionary/Sources/OrderedDictionary.swift, line 291

which corresponds to this line. It appears that it's attempting to return nil for an existing key.

My full backtrace:

(lldb) thread backtrace
* thread #1, name = 'MyApp', stop reason = Precondition failed: Inconsistency error occurred in OrderedDictionary
  * frame #0: 0x00007ffff7c37110 libswiftCore.so`_swift_runtime_on_report
    frame #1: 0x00007ffff7cacb85 libswiftCore.so`_swift_stdlib_reportFatalErrorInFile + 213
    frame #2: 0x00007ffff79484a2 libswiftCore.so`closure #1 (Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in closure #1 (Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in closure #1 (Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in Swift._assertionFailure(_: Swift.StaticString, _: Swift.String, file: Swift.StaticString, line: Swift.UInt, flags: Swift.UInt32) -> Swift.Never + 242
    frame #3: 0x00007ffff79480e6 libswiftCore.so`closure #1 (Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in closure #1 (Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in Swift._assertionFailure(_: Swift.StaticString, _: Swift.String, file: Swift.StaticString, line: Swift.UInt, flags: Swift.UInt32) -> Swift.Never + 86
    frame #4: 0x00007ffff7948685 libswiftCore.so`function signature specialization <Arg[1] = [Closure Propagated : closure #1 (Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in closure #1 (Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in Swift._assertionFailure(_: Swift.StaticString, _: Swift.String, file: Swift.StaticString, line: Swift.UInt, flags: Swift.UInt32) -> Swift.Never, Argument Types : [Swift.StaticStringSwift.UnsafeBufferPointer<Swift.UInt8>Swift.UIntSwift.UInt32]> of generic specialization <()> of Swift.String.withUTF8<τ_0_0>((Swift.UnsafeBufferPointer<Swift.UInt8>) throws -> τ_0_0) throws -> τ_0_0 + 181
    frame #5: 0x00007ffff7946d10 libswiftCore.so`Swift._assertionFailure(_: Swift.StaticString, _: Swift.String, file: Swift.StaticString, line: Swift.UInt, flags: Swift.UInt32) -> Swift.Never + 528
    frame #6: 0x0000555555e95e9b MyApp`OrderedDictionary._unsafeValue(key=MyLib.MyData @ 0x00007fffffffc4b0, self=OrderedDictionary.OrderedDictionary<MyLib.MyData, MyLib.MyDataStream> @ 0x00007fffffffc460) at OrderedDictionary.swift:291:9
    frame #7: 0x0000555555e962e1 MyApp`OrderedDictionary.subscript.getter(position=219, self=OrderedDictionary.OrderedDictionary<MyLib.MyData, MyLib.MyDataStream> @ 0x00007fffffffc6a0) at OrderedDictionary.swift:313:21
    frame #8: 0x0000555555e9be5b MyApp`OrderedDictionary.subscript.read(position=219, self=OrderedDictionary.OrderedDictionary<MyLib.MyData, MyLib.MyDataStream> @ 0x00007fffffffc770) at OrderedDictionary.swift:0
    frame #9: 0x0000555555e9bb75 MyApp`protocol witness for Collection.subscript.read in conformance OrderedDictionary<A, B> at <compiler-generated>:0
    frame #10: 0x00007ffff797597f libswiftCore.so`protocol witness for Swift.IteratorProtocol.next() -> Swift.Optional<τ_0_0.Element> in conformance Swift.IndexingIterator<τ_0_0> : Swift.IteratorProtocol in Swift + 463
    frame #11: 0x00007ffff7a347a5 libswiftCore.so`Swift.LazyMapSequence.Iterator.next() -> Swift.Optional<τ_0_1> + 197
    frame #12: 0x0000555556043caf MyApp`MyDataHandler.stopAndFlushData(self=0x000055555751f3a0) at MyDataHandler.swift:360:9
    frame #13: 0x00005555560b7c07 MyApp`normalRun() at TimeSeriesRun.swift:96:34
    frame #14: 0x00005555560b89c9 MyApp`runOnlineLearner() at TimeSeriesRun.swift:130:5
    frame #15: 0x0000555555fee8ef MyApp`main() at main.swift:12:5
    frame #16: 0x0000555555fee834 MyApp`main at main.swift:15:1
    frame #17: 0x00007fffabb09b97 libc.so.6`__libc_start_main + 231
    frame #18: 0x00005555557f83aa MyApp`_start + 42

MyDataHandler.swift:360 is this code in my project:

        for myDataStream in self.dataStreams.orderedValues { // This is line 360
            stopStreamsFutures.append(myDataStream.close())
        }

self.dataStreams is:

    var dataStreams: OrderedDictionary<MyData, MyDataStream> = []

and MyData is a struct:

struct MyData: Decodable & CustomStringConvertible & Hashable & Encodable & Equatable {
    let name: String
...

    static func == (lhs: MyData, rhs: MyData) -> Bool {
        return lhs.name == rhs.name
    }
...
}

I don't see this error on every server run. The only unusual aspect of this server run is that it hit a rare case where MyData keys had to be removed from the OrderedDictionary. There is a situation where I will set values to nil for existing keys when those MyData objects are no longer needed: self.dataStreams[myData] = nil, but this is indicated as the correct way to remove key-value pairs in the example code. One possibility is that the keys that were set to nil were later added back with a new value, but I don't think that should matter.

My server is heavily multi-threaded, but I don't currently see any potential concurrency issues here. This particular call in my code is at the end of the server's life when a stop command has been issued to flush and close all data streams. The call is made on the main thread. Running with the thread sanitizer doesn't show any issues.

Do you have any thoughts on what might be going wrong here?

[lukaskubanek]

Hi @xanderdunn, first of all, I have to say that I don’t have experience with Swift on server and such multithreaded environments. Also, I haven’t tested OrderedDictionary in such scenarios, so there might be potential issues within the library itself.

Do you have any thoughts on what might be going wrong here?

From what I can see in your very detailed description of the issue, it looks like it has something to do with access from multiple threads. Are you modifying the ordered dictionary dataStreams while iterating over it, possibly from other threads, i.e. setting values to nil for certain keys in the loop?

OrderedDictionary.orderedValues returns a lazy sequence that operates on the underlying structure. In the upcoming version of the library (v4.0) I plan to change that to creating a copied Array, but it’ll take some time until I release it. In the meantime, I’d suggest you to create a copy of the values (data streams) and iterate over it.

for myDataStream in Array(self.dataStreams.orderedValues) {
    stopStreamsFutures.append(myDataStream.close())
}

There is a situation where I will set values to nil for existing keys when those MyData objects are no longer needed: self.dataStreams[myData] = nil, but this is indicated as the correct way to remove key-value pairs in the example code.

Yes, that’s correct. As mentioned above, the question is where this is triggered.

[xanderdunn]

@lukaskubanek Thanks very much for the input. We recently switched over to InsertionOrderedDictionary here, a part of a library that we were already using for other data structures.

We had another, unrelated situation where setting keys equal to nil and then later setting that same key to a non-nil value on OrderedDictionary caused inconsistencies. The .count of the OrderedDictionary was off and our checksums were failing as a result of it. This situation was not multi-threaded. The OrderedDictionary was read or written to only from the main thread. Unfortunately we haven't created a minimally reproducing example. If I were to start with a test for this, I'd try iterating through hundreds of thousands of non-unique key-value pairs, with random values that could include nil, and repeatedly set the keys on the OrderedDictionary to any non-unique value. Check that .count remains sane. You can close this issue when you'd like since we're no longer using the library.