Implement accessibility for tizen platform

wanchao-xu commented 2 years ago

Transplant linux part of chromium accessibility.
- Screen reader reads widget information by atk/atspi2 on tizne platform, and chromium accessibility support atk/atspi2
- Screen reader only read highlighted widget information on tizen mobile/wearable, modify the platform node of chromium accessibility to support it.
Implement the accessibility for tizen platform
- To use the flutter tizen accessibility, the screen reader should apply the patch https://review.tizen.org/gerrit/#/c/platform/core/accessibility/screen-reader/+/270107/.

swift-kim commented 2 years ago

Problem 1) The engine is being included as part of the embedder binary.

You can clearly see that the size of the embedder is larger than that of the engine, which is unexpected.

swift@swift-760XDA:~/Git/engine/src$ ls -lh out/linux_release_arm/lib*.so
-rwxrwxr-x 1 swift swift 7.8M  3월  4 13:31 out/linux_release_arm/libflutter_engine.so
-rwxrwxr-x 1 swift swift 8.2M  3월  4 13:31 out/linux_release_arm/libflutter_tizen_common.so
-rwxrwxr-x 1 swift swift 8.2M  3월  4 13:31 out/linux_release_arm/libflutter_tizen_mobile.so
-rwxrwxr-x 1 swift swift 8.2M  3월  4 13:31 out/linux_release_arm/libflutter_tizen_tv.so
-rwxrwxr-x 1 swift swift 8.2M  3월  4 13:31 out/linux_release_arm/libflutter_tizen_wearable.so

Here's a temporary solution:

diff --git a/shell/platform/common/BUILD.gn b/shell/platform/common/BUILD.gn
index 74c200d90a..1bbf2c8eb8 100644
--- a/shell/platform/common/BUILD.gn
+++ b/shell/platform/common/BUILD.gn
@@ -89,9 +89,13 @@ source_set("common_cpp_accessibility") {

   public_deps = [
     "//flutter/fml:fml",
-    "//flutter/shell/platform/embedder:embedder_as_internal_library",
+    "//flutter/shell/platform/embedder:embedder_headers",
     "//flutter/third_party/accessibility",
   ]
+
+  if (!build_tizen_shell) {
+    public_deps += [ "//flutter/shell/platform/embedder:embedder_as_internal_library" ]
+  }
 }

 source_set("common_cpp") {

Problem 2) Non-public API usage detected:

Symbols not allowed (out/linux_release_arm/libflutter_tizen_wearable.so):
 U atk_action_get_type
 U atk_bridge_adaptor_init
 U atk_component_get_type
 U atk_document_get_type
 U atk_hyperlink_get_type
 U atk_hyperlink_impl_get_type
 U atk_hypertext_get_type
 U atk_image_get_type
 U atk_object_get_name
 U atk_object_get_type
 U atk_object_initialize
 U atk_object_notify_state_change
 U atk_relation_set_add_relation_by_type
 U atk_relation_set_new
 U atk_relation_type_get_type
 U atk_selection_get_type
 U atk_state_set_add_state
 U atk_table_get_type
 U atk_text_attribute_get_name
 U atk_text_get_character_count
 U atk_text_get_type
 U atk_util_get_type
 U atk_value_get_type
 U atk_window_get_type
 U kFlutterSemanticsCustomActionIdBatchEnd
 U kFlutterSemanticsNodeIdBatchEnd

This means that apps built with the embedder binary will be automatically rejected by the watch app store. You may need to either exclude the code (related to the accessibility feature) from the wearable embedder build (by modifying tizen/BUILD.gn and using #ifndef WEARABLE_PROFILE) or send a request to add the symbols to the store's API allowlist.

wanchao-xu commented 2 years ago

@swift-kim , about "Problem 2) Non-public API usage detected", wearable support accessibility(screen reader), and if I exclude the code, screen reader will not read the widget information of flutter app, I think it's better to add the symbols to the store's API allowlist.

swift-kim commented 2 years ago

Please format the GN file (shell/platform/common/BUILD.gn) correctly so that the CI build can run.

I think it's better to add the symbols to the store's API allowlist.

By the way, do you know why the symbols used by the embedder itself (kFlutterSemanticsCustomActionIdBatchEnd, kFlutterSemanticsNodeIdBatchEnd) are externally exported? We should avoid adding these symbols to the allowlist as much as possible.

Also, is there any possibility of any other atk_* symbols that are not listed above being included in the future? I'd like to request updating the allowlist file only once. Do you have a full list of atk_* symbols?

wanchao-xu commented 2 years ago

about kFlutterSemanticsCustomActionIdBatchEnd and kFlutterSemanticsNodeIdBatchEnd, flutter engine will get all semantics node and custom actions, the symbols are used to notice the end.

bool FlutterTizenEngine::RunEngine(const char* entrypoint) {
  ...
  args.update_semantics_node_callback = OnUpdateSemanticsNode;
  args.update_semantics_custom_action_callback = OnUpdateSemanticsCustomActions;

And I have no list of atk_* symbols, is there any way to show such list?

swift-kim commented 2 years ago

about kFlutterSemanticsCustomActionIdBatchEnd and kFlutterSemanticsNodeIdBatchEnd

Anyway you need to find a way to prevent these symbols from being exported.

And I have no list of atk_* symbols, is there any way to show such list?

You may take a look at the src/build/linux/debian_sid_amd64-sysroot/debian/libatk1.0-0/DEBIAN/symbols file in your buildroot or https://gnome.pages.gitlab.gnome.org/atk/api-index-full.html. According to https://github.com/flutter-tizen/tizen_tools/runs/5418572124?check_suite_focus=true, you are using the libatk version 2.30.0.

swift-kim commented 2 years ago

@wanchao-xu Where did you copy the Chromium Linux accessibility code from? I need to be able to reproduce what you've done in order to upgrade the engine and accessibility code in the future. I need very detailed information on each file (e.g. why ax_platform_node_base.cc has been modified) but not just a link to the repo.

wanchao-xu commented 2 years ago

I copied the chromium linux accessibility from https://chromium.googlesource.com/chromium/src/+/4579d5538f06c5ef615a15bc67ebb9ac0523a973. Please refer to src/flutter/third_party/accessibility/README.md.

The original chromium accessibility has many dependencies, I should modify some code to reduce the dependencies after transplanted. Google also modify the chromium accessibility to support mac and windows, not just a link to the repo.

wanchao-xu commented 2 years ago

To use the flutter tizen accessibility, the screen reader should apply the patch https://review.tizen.org/gerrit/#/c/platform/core/accessibility/screen-reader/+/270107/.

swift-kim commented 2 years ago

I think it's better to add the symbols to the store's API allowlist.

@wanchao-xu I discussed with Mr. Kwon today and decided not to enable accessibility support for wearable devices (mainly due to MX division's recent security issue).

You can apply this patch to disable the feature from the wearable embedder: https://github.com/swift-kim/engine/commit/dc00575d9f8edbd7611a512ac6ff17017e43b0e7. (Please cherry-pick the commit but do not manually modify the code to avoid human errors.)

Please make a suggestion if you have a better idea.

wanchao-xu commented 2 years ago

@swift-kim , Can I merge this request into branch now?

flutter-tizen / engine

Implement accessibility for tizen platform #254