lavague-ai / LaVague

Large Action Model framework to develop AI Web Agents
https://docs.lavague.ai/en/latest/
Apache License 2.0
5.45k stars 498 forks source link

Form Filling Automation - Element Not Interactable #600

Open rogaha opened 2 months ago

rogaha commented 2 months ago

Description:

An error occurs when attempting to fill fields in a Google Docs form titled "FORMULÁRIO DE SISTEMA DE COMPENSAÇÃO PARA UNIDADES CONSUMIDORAS". The automation process fails to interact with certain input fields, generating a "Message: element not interactable" error.

Steps to Reproduce:

Navigate to the Google Docs form titled "FORMULÁRIO DE SISTEMA DE COMPENSAÇÃO PARA UNIDADES CONSUMIDORAS". Attempt to fill in the following fields: Unidade Consumidora: '2476760' Nome do Titular: 'ANTONIO CARLOS RUBINI' CPF/CNPJ: '94246424820' Endereço: 'Avenida Roseira, 1543 - Prq. Residencial dos Girassois, CAMPO GRANDE, 79091848' The system attempts to interact with the input field but fails with the error message. Expected Behavior:

The form fields should be filled with the provided data without errors.

Actual Behavior:

The system throws an error: "Message: element not interactable" when trying to click on /html/body/div[2]/div[3]/div[2]/div/div[2]/div/div/div/input.

Error Logs:

Timestamp: 2024-09-05 14:41:52,442
Error Message: Message: element not interactable (Session info: chrome=128.0.6613.115)
Stack Trace:
swift
Copy code
0   chromedriver                        0x000000010456f998 cxxbridge1$str$ptr + 1887096
1   chromedriver                        0x0000000104567e00 cxxbridge1$str$ptr + 1855456
2   chromedriver                        0x000000010416ca48 cxxbridge1$string$len + 89100
3   chromedriver                        0x00000001041b1f3c cxxbridge1$string$len + 372992
4   chromedriver                        0x00000001041a7930 cxxbridge1$string$len + 330484
5   chromedriver                        0x00000001041a73c0 cxxbridge1$string$len + 329092
6   chromedriver                        0x00000001041ea4a4 cxxbridge1$string$len + 603752

Possible Cause:

The targeted input element might not be visible or enabled at the time of interaction, resulting in an "element not interactable" error. Timing or rendering issues within the form interface may be preventing the automation tool from interacting with the field. Suggested Fix:

Implement a check to ensure the element is fully rendered and visible before interacting with it. Introduce waits or retries to handle timing issues, such as WebDriverWait to wait until the element is interactable. Review the form's HTML structure to ensure the XPath /html/body/div[2]/div[3]/div[2]/div/div[2]/div/div/div/input is targeting the correct input element.

Environment:

Browser: Chrome 128.0.6613.115 Automation Tool: ChromeDriver OS: macOS (or relevant operating system)

dhuynh95 commented 2 months ago

@adeprez can you have a look?

adeprez commented 1 month ago

Hello @rogaha, could you share with us the Google Docs Form the agent is trying to interact with?

I guess the element was made uninteractive before the agent could perform the actual action. It could be due to a human interaction on the page, or an unexpected DOM change that made the element non accessible.

Implement a check to ensure the element is fully rendered and visible before interacting with it.

Actually we have this check: agent will only propose actions for elements he can interact with (i.e. rendered, visible, non disabled, non overlayed).