Closed nielsenko closed 2 months ago
➤ PM Bot commented:
Jira ticket: RCORE-2139
~This does not happen with xcode 14.3.1 for some reason.~ This was a faulty observation. I can make this fail with xcode 14.3.1 as well.
This looks like an assertion failure - can we get the message of the assertion?
@nirinchev Recreated with a debug build. I don't get more info.
Are you able to identify which assertion is throwing? @nielsenko
@nicola-cab It is not one of the assertions. The stack-trace would have included realm::util::terminate_with_info
.
@nielsenko I suspect it is one of these guys: REALM_ASSERT_EX(), they throw. I'll try to repro locally.
@nielsenko I suspect it is one of these guys: REALM_ASSERT_EX(), they throw. I'll try to repro locally.
Why don't we see realm::util::terminate_with_info
in the stack-trace then?
Running again under the xcode debugger I get a slightly different stacktrace:
#0 0x00000001054f93b0 in __pthread_kill ()
#1 0x0000000105367124 in pthread_kill ()
#2 0x00000001801655c0 in abort ()
#3 0x00000001802a7798 in abort_message ()
#4 0x0000000180298ed0 in demangling_terminate_handler() ()
#5 0x00000001800634a8 in _objc_terminate() ()
#6 0x00000001802a6c70 in std::__terminate(void (*)()) ()
#7 0x00000001802a98dc in __cxxabiv1::failed_throw(__cxxabiv1::__cxa_exception*) ()
#8 0x00000001802a989c in __cxa_throw ()
#9 0x0000000108f70230 in realm::util::File::open_internal(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, realm::util::File::AccessMode, realm::util::File::CreateMode, int, bool*) at /Users/kasper/Projects/mongodb/realm-dart/packages/realm_dart/src/realm-core/src/realm/util/file.cpp:559
#10 0x000000010894f7fc in realm::util::File::open(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, realm::util::File::AccessMode, realm::util::File::CreateMode, int) at /Users/kasper/Projects/mongodb/realm-dart/packages/realm_dart/src/realm-core/src/realm/util/file.hpp:1058
#11 0x000000010894f570 in realm::util::File::open(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, realm::util::File::Mode) at /Users/kasper/Projects/mongodb/realm-dart/packages/realm_dart/src/realm-core/src/realm/util/file.hpp:1053
#12 0x0000000108d42ff0 in realm::util::InterprocessMutex::set_shared_part(realm::util::InterprocessMutex::SharedPart&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) at /Users/kasper/Projects/mongodb/realm-dart/packages/realm_dart/src/realm-core/src/realm/util/interprocess_mutex.hpp:178
#13 0x0000000108d40c2c in realm::DB::open(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, realm::DBOptions const&) at /Users/kasper/Projects/mongodb/realm-dart/packages/realm_dart/src/realm-core/src/realm/db.cpp:1116
#14 0x0000000108d44504 in realm::DB::open(realm::Replication&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, realm::DBOptions const&) at /Users/kasper/Projects/mongodb/realm-dart/packages/realm_dart/src/realm-core/src/realm/db.cpp:1503
...
and the following context:
libc++abi: terminating due to uncaught exception of type realm::FileAccessError: Failed to open file at path '/Users/kasper/Library/Developer/CoreSimulator/Devices/BC728C97-01D8-4EC5-9E3A-284EB93486A1/data/Containers/Data/Application/7CC281EB-4AF5-489F-9000-2D2793CE3390/tmp/realm_test_iHROtQ/mongodb-realm/flexible-local-uviabmj/server-utility/metadata/sync_metadata.realm.management/access_control.versions.mx': Too many open files
So it seems the crash is due to an uncaught exception. Obviously it should not be uncaught and should not crash, but it is also weird that we get into this "Too many open files" situation.
@nielsenko can you see if a simulator or test process is leaving open files on your system? Try lsof "/Volumes/Macintosh HD"
(Note: REALM_ASSERT_EX()
doesn't throw)
@ironage There are two things here:
The reason it crashes is because it's an unhandled exception on the sync client thread - I don't believe we're actively trying to handle all exceptions in Sync, because they're unexpected/extremely rare and in most cases, it's best to just crash the process rather than leave it in some weird state.
For situations where we cannot recover, I think it would be good to make it more explicit that we are terminating on purpose and why it is happening. Like do an explicit abort on any uncaught exception, and log using the static logger what the exception is first, so that the SDK is notified first?
Hmm.. if I downgrade the github runner to macos-12, then the Flutter iOS tests pass without hitting this issue. Here realm-core is still build with xcode 15.4, but the actual tests are run on a simualtor on macos-12 (which is x64), as opposed to this run which is run on macos-14 runner (which is arm64)
Okay. It seems something changed in later versions of macos (or maybe the simulators that come with later versions of xcode).
Since the simulator is just a process on macos it used to be that we could just use ulimit to increase the available file descriptors on the host and have it reflect in the simulator. We have used that trick for our tests since they juggle a lot of files, but it no longer seems to work.
I have tried increasing the limit with setrlimit
from within the librealm_dart
and then it works just fine.
@nirinchev is probably right that there is little we can do, if this happens during a client reset, but I still think we should bail in a more informative way. Fx, by logging a fatal error before aborting.
Normally, if there is an error during client reset, we do indeed log the error and forward that on to the user's sync error handler. However, this is an unusual case of error-upon-error: while handling a client reset error, we experienced an error while updating the metadata realm to track the recovery path for the manual client reset. You could check the logs for the original reason why the client reset failed, but I'd suspect it also has to do with maxing out the file handles. I'm not sure what further graceful steps we could take when we cannot access the file system, at some point we need to just fail the process. But perhaps logging the additional failure would be a good place to start.
Note that if we do implement a high level try...catch for cases like these in order to log the exception message, we risk losing the stack trace which can sometimes be more valuable for figuring out what actually happened.
Can't we just rethrow the original exception after logging, thus not modifying the stacktrace?
No, c++ exceptions don't work that way. The exception doesn't capture the stack trace, and the stack trace you get is the result of the process being killed without unwinding the call stack.
I see, that's unfortunate and then I agree that probably preserving the stacktrace would be more valuable than logging the error.
➤ ironage commented:
I'm closing this as I do not see any further actions to take.
SDK and version
SDK : Dart Version: 2.3.0-12-g25c79676 (using realm-core v14.7.0)
Observations
Crash log / stacktrace
Steps & Code to Reproduce
Assuming you have xcode 15.4 installed and selected, and a simulator running with iOS 17.5, and BAAS_URL points to a running BAAS instance.