[Proposal] Alternative action spaces

[ppasupat]

Proposal

Add a way to customize the action space.

Motivation

Previous works on MiniWoB++ use different action spaces:

• Shi17: World of Bits: An Open-Domain Platform for Web-Based Agents
  • Action type: multinomial over {no-op, click, drag, scroll-up, scroll-down, key-press}
  • Cursor position: multinomial over [0, W) × [0, H) grid
  • Keyboard key: multinomial over keys + some allowed shortcuts
• Liu18: Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration, Gur18: Learning to Navigate the Web, and Jia19: DOM-Q-NET: Grounded RL on Structured Language
  • Action type: multinomial over {click, focus-and-type}
  • Element: multinomial over elements
  • Typed text: multinomial over field values
• Humphreys22: A data-driven approach for learning to control computers
  • Action type: multinomial over 11 {no-op, move, click, double-click, press, release, wheel-up, wheel-down, key-press, emit-text}
  • Cursor position: multinomial over 51 bins in each dimension
  • Keyboard key (for key-press): multinomial over keys + some allowed shortcuts
  • Task field (for emit-text): multinomial over field values

Many RL methods work best on discrete action spaces, but it trades off with generality (e.g., binned cursor position vs any cursor position).

Pitch

Add ActionSpaceConfig that allows the user to customize:

• The list of action types to support
• Whether the cursor position should be binned
• The list of allowed keyboard keys
• Whether the emitted text is freeform or selected from a task field

Alternatives

One alternative is to always include all action types in the space (including redundant ones like clicking elements + clicking coordinates). Downsides include:

• The RL agent can no longer define a simple multinomial distribution over the action types (a mask is needed).
• If we add new action types, the action indices will change.

Checklist

• [x] I have checked that there is no similar issue in the repo (required)