google-research / android_world

AndroidWorld is an environment and benchmark for autonomous agents
Apache License 2.0
141 stars 14 forks source link

Install New Apps in AndroidWorld #118

Closed QiushiSun closed 2 months ago

QiushiSun commented 2 months ago

Thanks for the great work! I would like to ask if the current AndroidWorld environment supports the installation of new apps, such as the applications in AndroidControl. If it does, what is the recommended method or channel for installing them?

Thanks!

crawles commented 2 months ago

Hi! If the apps are OSS and can be acquired from fdroid.org, then you can just use adb_utils.install_apk('my_app.apk, env.controller) (example)

If the apps are not OSS, you will need to download them from the playstore. This is a little trickier and requires you to launch an emulator with Playstore. The default Pixel 6 does not, but you can use another one, e.g., Pixel 6a, which does have Google Play. Then once you do that, you can automate installation using the GUI. I wrote a quick script that you can try in a notebook.

from android_world.env import env_launcher
from android_world.env import json_action, adb_utils, actuation
import time
import os

def _find_adb_directory() -> str:
  """Returns the directory where adb is located."""
  potential_paths = [
      os.path.expanduser('~/Library/Android/sdk/platform-tools/adb'),
      os.path.expanduser('~/Android/Sdk/platform-tools/adb'),
  ]
  for path in potential_paths:
    if os.path.isfile(path):
      return path
  raise EnvironmentError('adb not found') 

env = env_launcher.load_and_setup_env(adb_path=_find_adb_directory())

# First, manually open playstore and login.
apps = ['brave', 'doordash', 'uber', 'firefox']
for app in apps:
    actuation.find_and_click_element('Search for apps & games', env.controller)
    adb_utils.type_text('  ' + app, env.controller)
    env.execute_action(json_action.JSONAction(action_type=json_action.KEYBOARD_ENTER))
    actuation.find_and_click_element('Install', env.controller)
    time.sleep(10)
    actuation.find_and_click_element('Navigate up', env.controller)

It may be a bit brittle, but hopefully this is helpful to get you going.

P.S. You can use this to inspect the UI elements:

from android_world.utils import plotting
state = env.get_state()
plotting.plot_ui_elements(state)
QiushiSun commented 2 months ago

Thank you for your response!!!❀️ I tried switching to Pixel 6a to use Google Play. I followed the exact same setup process as described in the README(for Pixel6), but when I ran python minimal_task_runner.py --task=ContactsAddContact to check if it works, I kept encountering the following errors. Is there any way to resolve this issue?

File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/android_env-1.2.2-py3.11.egg/android_env/components/adb_call_parser.py", line 755, in _handle_generic File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/android_env-1.2.2-py3.11.egg/android_env/components/adb_call_parser.py", line 82, in _execute_command File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/android_env-1.2.2-py3.11.egg/android_env/components/adb_controller.py", line 147, in execute_command android_env.components.errors.AdbControllerError: Error executing adb command: [adb -P 5037 -s emulator-5554 shell date 1015153423.00] Caused by: Command '['/Users/qiushisun/Library/Android/sdk/platform-tools/adb', '-P', '5037', '-s', 'emulator-5554', 'shell', 'date', '1015153423.00']' returned non-zero exit status 1. adb stdout: [b'Sun Oct 15 15:34:00 UTC 2023\ndate: cannot set date: Operation not permitted\n'] adb stderr: [None]

Thanks again for your help.

crawles commented 2 months ago

Hmm, can you try disabling the datetime arg, by doing freeze_datetime = False. This is needed for reproducibility for the AndroidWorld suite. I think the issue is that Google Play store emulator doesn't play as nice with setting the datetime. You could also try running adb root in the terminal, although we are already doing that so it shouldn't be neccessary.

I think since you are installing additional apps outside of the 20 apps in the test suite, it would probably be better to maintain a separate emulator image (in this case Pixel 6a) for installing new apps and testing them, running your experiments, and setting the freeze_datetime=False Then, if you want to run the AndroidWorld test suite (i.e the original 116 tests), just use the original instructions (Pixel 6) and keep that separate. Does this make sense?

QiushiSun commented 2 months ago

Thank you very much for your replyπŸ™. I now understand the logic of app installing / testing and will try further!

QiushiSun commented 3 weeks ago

Thank you very much for your help. πŸ™ I'm currently experimenting with a Pixel 6a (I want to install apps covered by AndroidControl, which aren't all available on F-Droid, so I chose to use the Play Store).

I followed your suggestion and disabled the datetime argument in env_launcher.py:

def setup_env(
    env: interface.AsyncEnv,
    emulator_setup: bool = False,
    freeze_datetime: bool = False, # switch to false
) -> None:
    """Performs environment setup and validation."""
    _increase_file_descriptor_limit()
    if emulator_setup:
        setup.setup_apps(env)
    if freeze_datetime:
        datetime_utils.setup_datetime(env.controller)

and

def load_and_setup_env(
    console_port: int = 5554,
    emulator_setup: bool = False,
    freeze_datetime: bool = False, # switch to false
    adb_path: str = android_world_controller.DEFAULT_ADB_PATH,
    grpc_port: int = 8554,
) -> interface.AsyncEnv:

However, I'm still encountering the error regarding "cannot set date: Operation not permitted" when running python minimal_task_runner.py --task=ContactsAddContact

File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/android_env-1.2.2-py3.11.egg/android_env/components/adb_controller.py", line 147, in execute_command
android_env.components.errors.AdbControllerError: Error executing adb command: [adb -P 5037 -s emulator-5554 shell date 1015153423.00]
Caused by: Command '['/Users/qiushisun/Library/Android/sdk/platform-tools/adb', '-P', '5037', '-s', 'emulator-5554', 'shell', 'date', '1015153423.00']' returned non-zero exit status 1.
adb stdout: [b'Sun Oct 15 15:34:00 HKT 2023\ndate: cannot set date: Operation not permitted\n']
adb stderr: [None]

Could you provide further guidance? Thank you! πŸ™

crawles commented 3 weeks ago

Can you please provide the full stack trace so I can see where it's trying to set the date? Thanks!

QiushiSun commented 3 weeks ago

Thanks a lot for your response! πŸ™ I set both freeze_datetime: bool = False, in env_launcher.py

python minimal_task_runner.py --task=ContactsAddContact

warnings.warn(
/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/google/protobuf/runtime_version.py:112: UserWarning: Protobuf gencode version 5.27.2 is older than the runtime version 5.28.1 at android_env/proto/a11y/android_accessibility_forest.proto. Please avoid checked-in Protobuf gencode that can be obsolete.
  warnings.warn(
/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/google/protobuf/runtime_version.py:112: UserWarning: Protobuf gencode version 5.27.2 is older than the runtime version 5.28.1 at android_env/proto/a11y/rect.proto. Please avoid checked-in Protobuf gencode that can be obsolete.
  warnings.warn(
/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/google/protobuf/runtime_version.py:112: UserWarning: Protobuf gencode version 5.27.2 is older than the runtime version 5.28.1 at android_env/proto/a11y/a11y.proto. Please avoid checked-in Protobuf gencode that can be obsolete.
  warnings.warn(
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1730831269.557623 5294176 trace.cc:90] Unknown tracer: none
W1106 02:28:09.450361 8466665280 android_world_controller.py:98] Could not get a11y tree, retrying.
E1106 02:28:10.973581 8466665280 adb_controller.py:131] Failed to execute ADB command (try 1 of 3): [adb -P 5037 -s emulator-5554 shell date 1015153423.00]
Traceback (most recent call last):
  File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/android_env-1.2.2-py3.11.egg/android_env/components/adb_controller.py", line 122, in execute_command
    cmd_output = subprocess.check_output(
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/qiushisun/Library/Android/sdk/platform-tools/adb', '-P', '5037', '-s', 'emulator-5554', 'shell', 'date', '1015153423.00']' returned non-zero exit status 1.
E1106 02:28:10.975132 8466665280 adb_controller.py:135] **stdout**:
E1106 02:28:10.975165 8466665280 adb_controller.py:137]     b'Sun Oct 15 15:34:00 HKT 2023'
E1106 02:28:10.975341 8466665280 adb_controller.py:137]     b'date: cannot set date: Operation not permitted'
E1106 02:28:16.648721 8466665280 adb_controller.py:131] Failed to execute ADB command (try 2 of 3): [adb -P 5037 -s emulator-5554 shell date 1015153423.00]
Traceback (most recent call last):
  File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/android_env-1.2.2-py3.11.egg/android_env/components/adb_controller.py", line 122, in execute_command
    cmd_output = subprocess.check_output(
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/qiushisun/Library/Android/sdk/platform-tools/adb', '-P', '5037', '-s', 'emulator-5554', 'shell', 'date', '1015153423.00']' returned non-zero exit status 1.
E1106 02:28:16.649288 8466665280 adb_controller.py:135] **stdout**:
E1106 02:28:16.649338 8466665280 adb_controller.py:137]     b'Sun Oct 15 15:34:00 HKT 2023'
E1106 02:28:16.649402 8466665280 adb_controller.py:137]     b'date: cannot set date: Operation not permitted'
Traceback (most recent call last):
  File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/android_env-1.2.2-py3.11.egg/android_env/components/adb_controller.py", line 122, in execute_command
  File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/qiushisun/Library/Android/sdk/platform-tools/adb', '-P', '5037', '-s', 'emulator-5554', 'shell', 'date', '1015153423.00']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/qiushisun/Desktop/OS-Sim Playground/android_env/android_world/minimal_task_runner.py", line 131, in <module>
    app.run(main)
  File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/absl_py-2.1.0-py3.11.egg/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/absl_py-2.1.0-py3.11.egg/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
             ^^^^^^^^^^
  File "/Users/qiushisun/Desktop/OS-Sim Playground/android_env/android_world/minimal_task_runner.py", line 127, in main
    _main()
  File "/Users/qiushisun/Desktop/OS-Sim Playground/android_env/android_world/minimal_task_runner.py", line 107, in _main
    task.initialize_task(env)
  File "/Users/qiushisun/Desktop/OS-Sim Playground/android_env/android_world/android_world/task_evals/common_validators/contacts_validators.py", line 39, in initialize_task
    super().initialize_task(env)
  File "/Users/qiushisun/Desktop/OS-Sim Playground/android_env/android_world/android_world/task_evals/task_eval.py", line 135, in initialize_task
    self.initialize_device_time(env)
  File "/Users/qiushisun/Desktop/OS-Sim Playground/android_env/android_world/android_world/task_evals/task_eval.py", line 129, in initialize_device_time
    datetime_utils.set_datetime(env.controller, self.device_time)
  File "/Users/qiushisun/Desktop/OS-Sim Playground/android_env/android_world/android_world/utils/datetime_utils.py", line 130, in set_datetime
    _set_datetime(env, dt)
  File "/Users/qiushisun/Desktop/OS-Sim Playground/android_env/android_world/android_world/utils/datetime_utils.py", line 201, in _set_datetime
    adb_utils.issue_generic_request(
  File "/Users/qiushisun/Desktop/OS-Sim Playground/android_env/android_world/android_world/env/adb_utils.py", line 528, in issue_generic_request
    response = env.execute_adb_call(
               ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/android_env-1.2.2-py3.11.egg/android_env/wrappers/base_wrapper.py", line 101, in execute_adb_call
  File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/android_env-1.2.2-py3.11.egg/android_env/wrappers/base_wrapper.py", line 101, in execute_adb_call
  File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/android_env-1.2.2-py3.11.egg/android_env/environment.py", line 139, in execute_adb_call
  File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/android_env-1.2.2-py3.11.egg/android_env/components/coordinator.py", line 284, in execute_adb_call
  File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/android_env-1.2.2-py3.11.egg/android_env/components/adb_call_parser.py", line 112, in parse
  File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/android_env-1.2.2-py3.11.egg/android_env/components/adb_call_parser.py", line 755, in _handle_generic
  File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/android_env-1.2.2-py3.11.egg/android_env/components/adb_call_parser.py", line 82, in _execute_command
  File "/Users/qiushisun/opt/miniconda3/envs/android_world/lib/python3.11/site-packages/android_env-1.2.2-py3.11.egg/android_env/components/adb_controller.py", line 147, in execute_command
android_env.components.errors.AdbControllerError: Error executing adb command: [adb -P 5037 -s emulator-5554 shell date 1015153423.00]
Caused by: Command '['/Users/qiushisun/Library/Android/sdk/platform-tools/adb', '-P', '5037', '-s', 'emulator-5554', 'shell', 'date', '1015153423.00']' returned non-zero exit status 1.
adb stdout: [b'Sun Oct 15 15:34:00 HKT 2023\ndate: cannot set date: Operation not permitted\n']
adb stderr: [None]
crawles commented 3 weeks ago

Ah, I see, thanks for that.

For now you should be able to just comment out this line in task_eval.py.

I will think of a fix so this can get disabled if the user desires.

(For posterity, from this error)

File "/Users/qiushisun/Desktop/OS-Sim Playground/android_env/android_world/android_world/task_evals/task_eval.py", line 135, in initialize_task self.initialize_device_time(env)