Can't find or read elements with Unity game

[Shelvak]

Hey first that all, thanks for the amazing tool.

I'm trying to automate a few things with a game (Age of Z) and I can with sleeps and position clicks.

But I can't figure out how to read or detect things in the game. The methods like vc.findViewWithContentDescriptionor findViewWithText don't work...

In case you can guide me to advance with this would be great.

In case this helps:

>>> vc = ViewClient(*ViewClient.connectToDeviceOrExit(verbose=True))
Connecting to a device with serialno=.* with a timeout of 60 secs...
Connected to device with serialno=.*
Actual device serialno=xxxxxxx

>>>
>>>
>>>
>>> [v1, v2] = vc.dump(window=-1)
>>> v1.getText()
''
>>> v2.getText()
''
>>> vc.traverse(transform=ViewClient.TRAVERSE_CIT)

android.widget.FrameLayout
   android.view.View com.camelgames.aoz:id/unitySurfaceView
>>>

[dtmilano]

They might not be Views, that's why you don't have them in the dump.

In those cases, methods like imageToData() could be of great help.

[Shelvak]

It's kind of buggy, because after one shot, the "channel" is closed and I have to instantiate again. but it's better than nothing =P

Other question.. I can't find a way to "shake" (move) the phone... I found 2 solutions that doesn't work:

adb emu sensor set acceleration 100:100:100; sleep 1; adb emu sensor set acceleration 0:0:0
adb shell input keyevent 82

I tried the input keyevent with device.press('KEYCODE_MENU'), both adb command but didn't work.

Tried with a for loop all the KEYEVENT keys...

Any idea? thanks in advance

[dtmilano]

Hi, can you please elaborate on

It's kind of buggy, because after one shot, the "channel" is closed and I have to instantiate again. but it's better than nothing =P

I'm not sure what channel you are referring to.

[Shelvak]

Sorry for that =P I mean that I have to create a new device instance for each snapshot For example:

>>> device, serialno = ViewClient.connectToDeviceOrExit(**kwargs1)
>>> ss = device.takeSnapshot()
>>> device.imageToData(ss.crop((704,1210,704+271,1210+157)))['text']
['', '', '', '', ' ', '', '', '', '{', '“Curacion']
>>> ss = device.takeSnapshot()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/rotsen/.local/lib/python3.10/site-packages/com/dtmilano/android/adb/adbclient.py", line 850, in takeSnapshot
    self.__send('framebuffer:', checkok=True, reconnect=False)
  File "/home/rotsen/.local/lib/python3.10/site-packages/com/dtmilano/android/adb/adbclient.py", line 311, in __send
    self.__checkOk()
  File "/home/rotsen/.local/lib/python3.10/site-packages/com/dtmilano/android/adb/adbclient.py", line 375, in __checkOk
    raise RuntimeError("ERROR: %s %s" % (repr(recv), error))
RuntimeError: ERROR: b'' b''

I instantiate a new device each time that I need to take a snapshot to prevent this =)

[dtmilano]

device.takeSnapshot(reconnect=True)

would do the trick, actually for performance reasons if you are going to take only one screenshot it disconnects, so if you are planning to take more use reconnect=True.

See https://stackoverflow.com/questions/53143331/python-androidviewclient-take-snapshots-in-high-fps for some benchmarks.

Also mentioned in https://github.com/dtmilano/AndroidViewClient/issues/46.

[dtmilano]

Regarding the second part, are you trying to shake a device or emulator?

[Shelvak]

A real device, I'm using culebra with a secondary phone mirrored with scrcpy.