Skip resizing image if downsampleFactor is one

abaiadse commented 2 weeks ago

I'm using Appium-ocr-plugin with Appium server running on Docker. Most of the time, iOS tests fail with the following error:

2024-10-08 19:13:13.893 [944069a6][Plugin [ocr]] Using 3.12 as the screenshot-to-screen size ratio
2024-10-08 19:13:13.894 [944069a6][Plugin [ocr]] Using downsample factor of 3.12
2024-10-08 19:13:13.896 munmap_chunk(): invalid pointer
2024-10-08 19:13:14.409 Aborted (core dumped)

As a workaround, I want to set the downsample factor to 1 (via driver settings) to prevent resizing the screenshot image. However, even when I set it to 1, the code still executes the section that leads to this error.

abaiadse commented 2 weeks ago

@jlipps @KazuCocoa @rsin46 @sujithkp795 Please review :)

KazuCocoa commented 2 weeks ago

Could you share the same log part when you set ocrDownsampleFactor and get the texts? Maybe it would be helpful to share your system info as well such as https://github.com/appium/appium/issues/new?assignees=&labels=Bug%2CNeeds+Triage&projects=&template=bug_report.yml&title=bug%3A+%3Ctitle%3E 's template asks.

The change itself makes sense to skip if 1 scale (as not need to resign), but I'd like to know what environment could cause the error.

abaiadse commented 2 weeks ago

@KazuCocoa Here's the log when ocrDownsampleFactor is set "driver.setSetting("ocrDownsampleFactor", 1);"

2024-10-10 11:43:32.704 [724e8be4][XCUITestDriver@ace7] Responding to client with driver.getContexts() result: ["NATIVE_APP","OCR"]
2024-10-10 11:43:32.707 [724e8be4][HTTP] <-- GET /wd/hub/session/724e8be4-4614-4763-acf5-a4ef4f33e33f/contexts 200 615 ms - 30
2024-10-10 11:43:32.791 [724e8be4][HTTP] --> POST /wd/hub/session/724e8be4-4614-4763-acf5-a4ef4f33e33f/context {"name":"OCR"}
2024-10-10 11:43:32.792 [724e8be4][XCUITestDriver@ace7] Calling AppiumDriver.setContext() with args: ["OCR","724e8be4-4614-4763-acf5-a4ef4f33e33f"]
2024-10-10 11:43:32.792 [724e8be4][AppiumDriver@8772] Clearing new command timeout pre-emptively since plugin(s) will handle this command
2024-10-10 11:43:32.793 [724e8be4][AppiumDriver@8772] Plugins which can handle cmd 'setContext': images,ocr
2024-10-10 11:43:32.794 [724e8be4][AppiumDriver@8772] Plugin ocr is now handling cmd 'setContext'
2024-10-10 11:43:32.795 [724e8be4][AppiumDriver@
8772] Plugin images is now handling cmd 'setContext'
2024-10-10 11:43:32.795 [724e8be4][AppiumDriver@8772] Executing default handling behavior for command 'setContext'
2024-10-10 11:43:32.795 [724e8be4][XCUITestDriver@ace7] Executing command 'setContext'
2024-10-10 11:43:32.796 [724e8be4][XCUITestDriver@ace7] Attempting to set context to 'OCR' from 'NATIVE_APP'
2024-10-10 11:43:32.871 [724e8be4][Plugin [ocr]] Default behavior failed handling setContext, ignoring
2024-10-10 11:43:32.871 [724e8be4][Plugin [ocr]] Original error: NoSuchContextError: No such context found.
2024-10-10 11:43:32.872 [724e8be4][XCUITestDriver@ace7] Responding to client with driver.setContext() result: null
2024-10-10 11:43:32.875 [724e8be4][HTTP] <-- POST /wd/hub/session/724e8be4-4614-4763-acf5-a4ef4f33e33f/context 200 83 ms - 14
2024-10-10 11:43:32.931 [724e8be4][HTTP] --> GET /wd/hub/session/724e8be4-4614-4763-acf5-a4ef4f33e33f/context {}
2024-10-10 11:43:32.933 [724e8be4][XCUITestDriver@ace7] Calling AppiumDriver.getCurrentContext() with args: ["724e8be4-4614-4763-acf5-a4ef4f33e33f"]
2024-1
0-10T08:43:32.933617059Z [724e8be4][AppiumDriver@8772] Clearing new command timeout pre-emptively since plugin(s) will handle this command
2024-10-10 11:43:32.934 [724e8be4][AppiumDriver@8772] Plugins which can handle cmd 'getCurrentContext': images,ocr
2024-10-10 11:43:32.934 [724e8be4][AppiumDriver@8772] Plugin ocr is now handling cmd 'getCurrentContext'
2024-10-10 11:43:32.935 [724e8be4][AppiumDriver@8772] Command 'getCurrentContext' was *not* handled by the following behaviours or plugins, even though they were registered to handle it: ["default","images"]. The command *was* handled by these: ["ocr"].
2024-10-10 11:43:32.936 [724e8be4][AppiumDriver@8772] Restarting new command timeout via umbrella driver since plugin did not allow default handler to execute
2024-10-10 11:43:32.937 [724e8be4][XCUITestDriver@ace7] Responding to client with driver.getCurrentContext() result: "OCR"
2024-10-10 11:43:32.938 [724e8be4][HTTP] <-- GET /wd/hub/session/724e8be4-4614-4763-acf5-a4ef4f33e33f/context 200 6 ms - 15
2024-10-10 11:43:32.985 [724e8be4][HTTP] --> GET /wd/hub/session/724e8be4-4614-4763-acf5-a4ef4f33e33f/source {}
2024-10-10 11:43:32.986 [724e8be4][XCUITestDriver@ace7] Calling AppiumDriver.getPageSource() with args: ["724e8be4-4614-4763-acf5-a4ef4f33e33f"]
2024-10-10 11:43:32.986 [724e8be4][AppiumDriver@8772] Clearing new command timeout pre-emptively since plugin(s) will handle this command
2024-10-10 11:43:32.987 [724e8be4][AppiumDriver@8772] Plugins which can handle cmd 'getPageSource': images,ocr
2024-10-10 11:43:32.987 [724e8be4][AppiumDriver@8772] Plugin ocr is now handling cmd 'getPageSource'
2024-10-10 11:43:32.989 [724e8be4][XCUITestDriver@ace7] Taking screenshot with WDA
2024-10-10 11:43:32.991 [724e8be4][XCUITestDriver@ace7] Matched '/screenshot' to command name 'getScreenshot'
2024-10-10 11:43:32.992 [724e8be4][XCUITestDriver@ace7] Proxying [GET /screenshot] to [GET http://127.0.0.1:8100/session/DBF27E45-5ED3-4116-8094-772C815C5EDF/screenshot] with no body
2024-10-10 11:43:33.254 [724e8be4][XCUITestDriver@ace7] Got response with status 200: {"value":"iVBORw0KGgoAAAANSUhEUgAAAzwAAAcACAIAAACaY9F8AAAAAXNSR0IArs4c6QAAAERlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA6ABAAMAAAABAAEAAKACAAQAAAABAAADPKADAAQAAAABAAAHAAAAAAAYjVEBAAAAHGlET1QAAAACAAAAAAAAA4AAAAAoAAADgAAAA4AAAPfcBY8xagAAQABJREFUeAHsvQecHcWV7+99+/bt7vu/9za+sO+tQUQTlHMEjG28Bo0AB5AAY0CAjQmCNWuDhE2SSBIgAUaAAkmJqARKjCSyJIJAgSgkoqRRGEmjHOn/t/rcOVPT3bdvz52gGen05356qqurzzn1656uX5+qOvWdwDZDwBAwBAwBQ8AQMAQMgUaPwHcavYVmoCFgCBgChoAhYAgYAoZAYKTNHgJDwBAwBAwBQ8AQMASaAAJG2prATTITDQFDwBAwBAwBQ8AQMNJmz4AhYAgYAoaAIWAIGAJNAAEjbU3gJpmJhoAhYAgYAoaAIWAIGGmzZ8AQMAQMAUPAEDAEDIEmgICRtiZwk8xEQ8AQMAQMAUPAEDAEjLTZM2AIGAKGgCFgCBgChkATQMBIWxO4SWaiIWAIGAKGgCFgCBgCRtrsGTAEDAFDwBAwBAwBQ6AJIGCkrQncJDPREDAEDAFDwBAwBAwBI232DBgChoAhYAgYAoaAIdAEEDDS1gRukploCBgChoAhYAgYAoaAkTZ7BgwBQ8AQMAQMAUPAEGgCCBhpawI3yUw0BAwBQ8AQMAQMAUPASJs9A4aAIWAIGAKGgCFgCDQBBIy0NYGbZCYaAoaAIWAIGAKGgCFgpM2eAUPAEDAEDAFDwBAwBJoAAkbamsBNMhMNAUPAEDAEDAFDwBAw0mbPgCFgCBgChoAhY
AgYAk0AASNtTeAmmYmGgCFgCBgChoAhYAgYabNnwBAwBAwBQ8AQMAQMgSaAgJG2JnCTzERDwBAwBAwBQ8AQMASMtNk...
2024-10-10 11:43:33.258 [724e8be4][Plugin [ocr]] Using 3.12 as the screenshot-to-screen size ratio
2024-10-10 11:43:33.259 [724e8be4][Plugin [ocr]] Using downsample factor of 1
2024-10-10 11:43:33.261 munmap_chunk(): invalid pointer
2024-10-10 11:43:33.620 Aborted (core dumped)

abaiadse commented 2 weeks ago

- Operating system Info: OS=Linux 3.10.0-1160.49.1.el7.x86_64 # 1 SMP Tue Nov 30 15:51:32 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian

- Appium server version (output of appium --version): 2.11.1

- Appium driver(s) and their version(s) (appium driver list): ✔ Listing available drivers

uiautomator2@3.7.0 [installed (npm)]
xcuitest@7.22.1 [installed (npm)]
espresso@2.44.0 [installed (npm)]
flutter@2.8.0 [installed (npm)]
mac2 [not installed]
windows [not installed]
safari [not installed]
gecko [not installed]
chromium [not installed]

- Appium plugin(s) and their version(s) (appium plugin list): ✔ Listing available plugins

ocr@0.2.0 [installed (npm)]
images@3.0.16 [installed (npm)]
execute-driver [not installed]
relaxed-caps [not installed]
universal-xml [not installed]

- Node.js version (output of node --version): v18.20.3

- npm version (output of npm --version): 10.7.0

- Last component(s) version which did not exhibit the problem: None

- Platform and version under test: iOS, Version 15.0, Model iPhone XR (And reproduces on all iOS devices)

- Real device or emulator/simulator: Real Device

Sharp Version: appium-ocr-plugin@0.2.0 /root/.appium/node_modules/appium-ocr-plugin `-- sharp@0.32.6

Libvips version: 8.14.5

KazuCocoa commented 2 weeks ago

Thank you. So, you run xcuitest driver on Ubuntu. Have you checked UIA2 driver on Ubuntu with ocrDownsampleFactor to go through the path? I wondered if uia2 worked. Then, the screenshot size would be affected. (memory to treat entire screenshot, just my guess)

abaiadse commented 2 weeks ago

Using UIA2 driver with the default value ocrDownsampleFactor=null works fine. setting ocrDownsampleFactor=1 leads to the same error:

[8647239e][AndroidUiautomator2Driver@0354] Responding to client with driver.getCurrentContext() result: "OCR"
[8647239e][HTTP] <-- GET /wd/hub/session/8647239e-4e8b-4afa-b504-7f5b4822af63/context 200 3 ms - 15 
[8647239e][HTTP] --> GET /wd/hub/session/8647239e-4e8b-4afa-b504-7f5b4822af63/source {}
[8647239e][AndroidUiautomator2Driver@0354] Would have proxied command directly, but a plugin exists which might require its value, so will let its value be collected internally and made part of plugin chain
[8647239e][AndroidUiautomator2Driver@0354] Calling AppiumDriver.getPageSource() with args: ["8647239e-4e8b-4afa-b504-7f5b4822af63"]
[8647239e][AppiumDriver@b14e] Clearing new command timeout pre-emptively since plugin(s) will handle this command
[8647239e][AppiumDriver@b14e] Plugins which can handle cmd 'getPageSource': images,ocr
[8647239e][AppiumDriver@b14e] Plugin ocr is now handling cmd 'getPageSource'
[8647239e][AndroidUiautomator2Driver@0354] Matched '/screenshot' to command name 'getScreenshot'
[8647239e][AndroidUiautomator2Driver@0354] Proxying [GET /screenshot] to [GET http://127.0.0.1:8200/session/a0dd4d0b-9471-45a8-8fdd-b0d33ae97780/screenshot] with no body
[8647239e][AndroidUiautomator2Driver@0354] Got response with status 200: {"sessionId":"a0dd4d0b-9471-45a8-8fdd-b0d33ae97780","value":"iVBORw0KGgoAAAANSUhEUgAABDgAAAlgCAYAAABtGofVAAAAAXNSR0IArs4c6QAAAARzQklUCAgICHwIZIgAACAASURBVHic7N15fBT1/T/w1+xukk3IASHcKPcNJiECCkIR8UJQxAuqFa2iYrHiw9raUhV6ifVbBauiVvmpeCAKFgTBg1OgchMggNx3Eo4khBx7v39/ZGeY2Z1NNiHJsvJ6Ph6fB9mdmc985thlPu/9HAoAARERERERERFRFLNEugBERERERERERBeKAQ4iIiIiIiIiinoMcBARERERERFR1GOAg4iIiIiIiIiiHgMcRERERERERBT1GOAgIiIiIiIioqjHAAcRERERERERRT0GOIiIiIiIiIgo6jHAQURERERERERRjwEOIiIiIiIiIop6DHAQERERERERUdRjgIOIiIiIiIiIoh4DHEREREREREQU9RjgICIiIiIiIqKoxwAHEREREREREUU9BjiIiIiIiIiIKOoxwEFEREREREREUY8BDiIiIiIiIiKKegxwEBEREREREVHUY4CDiIiIiIiIiKIeAxxEREREREREFPUY4CAiIiIiIiKiqMcABxERERERERFFPQY4iIiIiIiIiCjqMcBBRERERERERFGPAQ4iIiIiIiIiinoMcBARERERERFR1GOAg4iIiIiIiIiiHgMcRERERERERBT1GOAgIiIiIiIioqjHAAcRERERERERRT0GOIiIiIiIiIgo6jHAQURERERERERRjwEOIiIiIiIiIop6DHAQERERERERUdRjgIOIiIiIiIiIoh4DHEREREREREQU9RjgICIiIiIiIqKoxwAHEREREREREUU9BjiIiIiIiIiIKOoxwEFEREREREREUY8BDiIiIiIiIiKKegxwEBEREREREVHUY4CDiIiIiIiIiKIeAxxEREREREREFPUY4CAiIiIiIiKiqMcABxER...
[8647239e][Plugin [ocr]] Using 1 as the screenshot-to-screen size ratio
[8647239e][Plugin [ocr]] Using downsample factor of 1
munmap_chunk(): invalid pointer
Aborted (core dumped)

KazuCocoa commented 1 week ago

Thank you. Then, maybe this is more environment related 👀

jlipps commented 1 week ago

published in 0.2.1

jlipps / appium-ocr-plugin

Skip resizing image if downsampleFactor is one #14