dart-lang / native

Dart packages related to FFI and native assets bundling.
BSD 3-Clause "New" or "Revised" License
82 stars 27 forks source link

Make ObjC memory management errors more debuggable #1111

Closed liamappelbe closed 1 week ago

liamappelbe commented 2 weeks ago

When retaining or releasing a reference to an ObjC object or block, assert that the pointer is valid. The goal is to catch memory management errors (most importantly, retaining a ref to a dead object) in Dart where we can get a nice stack trace. Without this change, the retain/release call will seg fault in ObjC land, which gives no useful debugging information.

The key functions are isValidBlock (in objective_c.c) and isValidObject (in internal.dart).

isValidBlock is based on the getBlockRetainCount function from the ffigen tests. Blocks have a field named isa, which describes what kind of block it is. I tracked down all possible values of this field (there are 6), and so I can just check that the value is one of these. When a block is destroyed the isa field is cleared, so the odds of falsely declaring a block as valid are pretty low.

isValidObject is similar. Objects have an 8 byte header, part of which is a pointer to a class object. The object_getClass function extracts that pointer, and then we check if it's valid by looking it up in a set that contains every valid class object (there are typically a few thousand). We construct this set using objc_copyClassList. Like blocks, this header is cleared when the object is destroyed, so false positives are unlikely. See https://github.com/dart-lang/native/issues/1038 for more info.

Fixes https://github.com/dart-lang/native/issues/1038

liamappelbe commented 2 weeks ago

Looks like I used the wrong base branch. Commits e4a844e and later are real.

github-actions[bot] commented 2 weeks ago

PR Health

Changelog Entry :heavy_check_mark:

Details | Package | Changed Files | | :--- | :--- | Changes to files need to be [accounted for](https://github.com/dart-lang/ecosystem/wiki/Changelog) in their respective changelogs.

Coverage :warning:

Details | File | Coverage | | :--- | :--- | |pkgs/objective_c/lib/src/c_bindings_generated.dart| :broken_heart: Not covered | |pkgs/objective_c/lib/src/internal.dart| :broken_heart: Not covered | This check for [test coverage](https://github.com/dart-lang/ecosystem/wiki/Test-Coverage) is informational (issues shown here will not fail the PR). This check can be disabled by tagging the PR with `skip-coverage-check`.

License Headers :heavy_check_mark:

Details ``` // Copyright (c) 2024, the Dart project authors. Please see the AUTHORS file // for details. All rights reserved. Use of this source code is governed by a // BSD-style license that can be found in the LICENSE file. ``` | Files | | :--- | | _no missing headers_ | All source files should start with a [license header](https://github.com/dart-lang/ecosystem/wiki/License-Header).
Unrelated files missing license headers | Files | | :--- | |pkgs/ffi/example/main.dart| |pkgs/ffigen/example/libclang-example/generated_bindings.dart| |pkgs/ffigen/example/shared_bindings/generate.dart| |pkgs/ffigen/example/shared_bindings/lib/generated/a_gen.dart| |pkgs/ffigen/example/shared_bindings/lib/generated/a_shared_b_gen.dart| |pkgs/ffigen/example/shared_bindings/lib/generated/base_gen.dart| |pkgs/ffigen/example/simple/generated_bindings.dart| |pkgs/ffigen/lib/src/config_provider/config_spec.dart| |pkgs/ffigen/lib/src/header_parser/clang_bindings/clang_bindings.dart| |pkgs/ffigen/test/collision_tests/expected_bindings/_expected_decl_decl_collision_bindings.dart| |pkgs/ffigen/test/collision_tests/expected_bindings/_expected_decl_symbol_address_collision_bindings.dart| |pkgs/ffigen/test/collision_tests/expected_bindings/_expected_decl_type_name_collision_bindings.dart| |pkgs/ffigen/test/collision_tests/expected_bindings/_expected_reserved_keyword_collision_bindings.dart| |pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_comment_markup_bindings.dart| |pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_dart_handle_bindings.dart| |pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_forward_decl_bindings.dart| |pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_functions_bindings.dart| |pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_imported_types_bindings.dart| |pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_native_func_typedef_bindings.dart| |pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_opaque_dependencies_bindings.dart| |pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_packed_structs_bindings.dart| |pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_regress_384_bindings.dart| |pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_struct_fptr_fields_bindings.dart| |pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_typedef_bindings.dart| |pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_unions_bindings.dart| |pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_varargs_bindings.dart| |pkgs/ffigen/test/large_integration_tests/_expected_cjson_bindings.dart| |pkgs/ffigen/test/large_integration_tests/_expected_libclang_bindings.dart| |pkgs/ffigen/test/large_integration_tests/_expected_sqlite_bindings.dart| |pkgs/ffigen/test/native_test/_expected_native_test_bindings.dart| |pkgs/jni/lib/src/lang/jcharacter.dart| |pkgs/jni/lib/src/third_party/generated_bindings.dart| |pkgs/jni/lib/src/third_party/global_env_extensions.dart| |pkgs/jni/lib/src/third_party/jni_bindings_generated.dart| |pkgs/jnigen/android_test_runner/lib/main.dart| |pkgs/jnigen/example/in_app_java/lib/android_utils.dart| |pkgs/jnigen/example/kotlin_plugin/example/lib/main.dart| |pkgs/jnigen/example/kotlin_plugin/lib/kotlin_bindings.dart| |pkgs/jnigen/example/kotlin_plugin/lib/kotlin_plugin.dart| |pkgs/jnigen/example/pdfbox_plugin/lib/pdfbox_plugin.dart| |pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/pdmodel/PDDocument.dart| |pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/pdmodel/PDDocumentInformation.dart| |pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/pdmodel/_package.dart| |pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/text/PDFTextStripper.dart| |pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/text/_package.dart| |pkgs/jnigen/lib/src/bindings/descriptor.dart| |pkgs/jnigen/lib/src/elements/elements.g.dart| |pkgs/jnigen/test/jackson_core_test/third_party/bindings/com/fasterxml/jackson/core/_package.dart| |pkgs/jnigen/tool/command_runner.dart|

Package publish validation :heavy_check_mark:

Details | Package | Version | Status | | :--- | ---: | :--- | | package:ffi | 2.1.2 | already published at pub.dev | | package:ffigen | 12.0.0-wip | WIP (no publish necessary) | | package:jni | 0.9.0 | already published at pub.dev | | package:jnigen | 0.9.0 | already published at pub.dev | | package:native_assets_builder | 0.6.1 | already published at pub.dev | | package:native_assets_cli | 0.5.4 | already published at pub.dev | | package:native_toolchain_c | 0.4.1 | already published at pub.dev | | package:objective_c | 0.0.1-wip | WIP (no publish necessary) | Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.
coveralls commented 2 weeks ago

Coverage Status

coverage: 91.246% (+0.6%) from 90.676% when pulling 49de3a4a306cf24d62631788f93b8a17ec52383e on memerror into 6a9282c7e2abd5d5ad89a2de1eb985ee9fc63262 on main.

liamappelbe commented 2 weeks ago

Looks like the object ref counting test sometimes crashes (2/7 times I've run it). The pointer passed to object_getClass doesn't need to be a valid object, but it does need to be readable. I guess the memory page that the object was sitting in was unmapped right after the object was deleted.

I can work around this for the test by using one of the Mac virtual memory functions to test if the pointer is readable. These return an error code on failure rather than crashing. I'll do this for both the object and block tests.

I could potentially do this virtual memory test in the assert itself, but I feel like that would be overkill. I've never seen this crash on my local machine, so I think it's fairly rare. And unlike the test, if an unreadable pointer is passed to the assert, we were about to crash anyway, and the assert is just to give a best effort stack trace.

dcharkes commented 2 weeks ago

I could potentially do this virtual memory test in the assert itself, but I feel like that would be overkill. I've never seen this crash on my local machine, so I think it's fairly rare. And unlike the test, if an unreadable pointer is passed to the assert, we were about to crash anyway, and the assert is just to give a best effort stack trace.

Hm, maybe we should do it in asserts. There could be a chance that it's more common on a different type of machine. We don't pay any costs for asserts in release mode, so I'm not worried about the speed.

We should probably also make some clear documentation on the FFIgen and ObjectiveC package readme's that if there's segfaults the first thing users should do is run with asserts enabled.

liamappelbe commented 2 weeks ago

Hm, maybe we should do it in asserts. There could be a chance that it's more common on a different type of machine. We don't pay any costs for asserts in release mode, so I'm not worried about the speed.

Done. Actually, it looks like that check is too strict to use in general code. ObjC sometimes uses "tagged pointers" (this means something different than in Dart, so I can't just mask off the tag).

We should probably also make some clear documentation on the FFIgen and ObjectiveC package readme's that if there's segfaults the first thing users should do is run with asserts enabled.

Yeah, I need to write a readme for package:objective_c. At the moment it just has a placeholder readme. When I do I'll include a note about this.