search

web-arena-x / visualwebarena

VisualWebArena is a benchmark for multimodal agents.
https://jykoh.com/vwa
MIT License
211 stars 36 forks source link

The very important element: "sort" can not be found in accessibility_tree_with_captioner. #59

Closed leoozy closed 2 weeks ago

leoozy commented 1 month ago
image

Hello, when I try to run the models with accessibility_tree_with_captioner. I found that sort can not be found in the accessibility_tree_with_captioner. There is only a staticText of sort but no related interactable element.

Tab 0 (current): Toyota » Cars + trucks - Classifieds

[1] RootWebArea 'Toyota » Cars + trucks - Classifieds' focused: True
    [36] link '\"Classifieds, description: The image appears to be a simple graphic or icon representing various media formats. It features a rectangular shape with a border and two smaller circular shapes inside it, which, url: http://127.0.0.1:9980/oc-content/uploads/sigma_logo.png\"'
        [72] img '\"Classifieds, description: The image appears to be a simple graphic or icon representing various media formats. It features a rectangular shape with a border and two smaller circular shapes inside it, which, url: http://127.0.0.1:9980/oc-content/uploads/sigma_logo.png\"'
    [44] link 'My account'
    [45] link 'Logout'
    [46] link 'Publish Ad'
    [53] link 'Classifieds'
    [55] StaticText '> '
    [56] link 'Cars + trucks'
    [60] StaticText 'Search results: Toyota'
    [82] group ''
        [115] heading 'Your search'
        [161] textbox '' required: False
            [248] StaticText 'Toyota'
    [83] group ''
        [117] heading 'City'
        [165] textbox '' required: False
        [166] status '' live: polite atomic: True relevant: additions text
    [84] group ''
        [119] heading 'Show only'
        [169] checkbox '' checked: false
    [85] group ''
        [172] heading 'Price'
        [251] StaticText 'Min.'
        [252] textbox '' required: False
            [477] StaticText '3000'
        [254] StaticText 'Max.'
        [255] textbox '' required: False
            [478] StaticText '6000'
    [122] button 'Apply'
    [89] heading 'Subscribe to this search'
    [97] button 'Subscribe now!'
    [98] heading 'Refine category'
    [175] link 'All categories'
    [176] link 'Cars + trucks'
    [128] heading 'Cars + trucks'
    [178] StaticText '1 - 12 of 149 listings'
    [186] StaticText 'Sort by:'
    [187] StaticText 'Newly listed '
    [198] link '\uf0c9'
    [199] link '\uf84c'
    [67] heading 'Listings'
    [131] link '\"2007 Toyota Yaris, description: The image shows a metallic gold-colored car parked in a driveway. The car has five-spoke alloy wheels and side mirrors that match the body color. The vehicle, url: http://127.0.0.1:9980/oc-content/uploads/7846/7846_thumbnail.png\"'
        [206] img '\"2007 Toyota Yaris, description: The image shows a metallic gold-colored car parked in a driveway. The car has five-spoke alloy wheels and side mirrors that match the body color. The vehicle, url: http://127.0.0.1:9980/oc-content/uploads/7846/7846_thumbnail.png\"'
    [207] LayoutTable ''
        [208] LayoutTableRow ''
            [279] link '2007 Toyota Yaris'
            [285] StaticText '3000.00 
            [479] StaticText 'Cars + trucks'
            [482] StaticText 'Gainesville (Virginia)'
            [291] StaticText '2023/11/16'
            [292] StaticText 'Manual transmission 4 door sedan 4 new tires 200k miles Clean Carfax'
    [133] link '\"2002 toyota camry, description: The image features a silver Toyota Corolla parked on an asphalt surface. The car has a prominent front grille with the Toyota emblem, and it\\'s positioned at a, url: http://127.0.0.1:9980/oc-content/uploads/75756/75756_thumbnail.png\"'
        [209] img '\"2002 toyota camry, description: The image features a silver Toyota Corolla parked on an asphalt surface. The car has a prominent front grille with the Toyota emblem, and it\\'s positioned at a, url: http://127.0.0.1:9980/oc-content/uploads/75756/75756_thumbnail.png\"'
    [210] LayoutTable ''
        [211] LayoutTableRow ''
            [293] link '2002 toyota camry'
            [299] StaticText '4000.00 
            [485] StaticText 'Cars + trucks'
            [488] StaticText 'Town of Middletown (Delaware)'
            [305] StaticText '2023/11/16'
            [306] StaticText "Up for sale is our family's daily driven 2002 "
            [308] StaticText 'toyota'
            [309] StaticText ' camry 2.4l with 149,000 miles and climbing. I purchased this car in august from the Original owner whom was a senior citizen in her 80s. The car was always garage kept and well taken care of. Deale...'
    [135] link '\"2009 Toyota Camry LE sedan, description: The image shows a silver-colored sedan parked on a gravel or cobblestone surface. The car features the logo of a well-known automotive brand, specifically the emblem, url: http://127.0.0.1:9980/oc-content/uploads/63620/63620_thumbnail.png\"'
        [212] img '\"2009 Toyota Camry LE sedan, description: The image shows a silver-colored sedan parked on a gravel or cobblestone surface. The car features the logo of a well-known automotive brand, specifically the emblem, url: http://127.0.0.1:9980/oc-content/uploads/63620/63620_thumbnail.png\"'
    [213] LayoutTable ''
        [214] LayoutTableRow ''
            [310] link '2009 Toyota Camry LE sedan'
            [316] StaticText '5995.00 
            [491] StaticText 'Cars + trucks'
            [494] StaticText 'Borough of South New Castle (Pennsylvania)'
            [322] StaticText '2023/11/16'
            [323] StaticText 'This 2009 '
            [325] StaticText 'Toyota'
            [326] StaticText ' Camry Le sedan in tan has tan cloth interior with a 4 cylinder and a automatic with air power windows and locks tilt and cruise cd rear defrost keyless entry. This car is very very clean and rust free with a clean carfax and has 1730...'
    [137] link '\"2015 Toyota Corolla LE, description: The image shows a black hatchback car, possibly a Toyota Corolla, parked on an asphalt surface with grassy areas alongside it. The car is positioned at, url: http://127.0.0.1:9980/oc-content/uploads/20170/20170_thumbnail.png\"'
        [215] img '\"2015 Toyota Corolla LE, description: The image shows a black hatchback car, possibly a Toyota Corolla, parked on an asphalt surface with grassy areas alongside it. The car is positioned at, url: http://127.0.0.1:9980/oc-content/uploads/20170/20170_thumbnail.png\"'
    [216] LayoutTable ''
        [217] LayoutTableRow ''
            [327] link '2015 Toyota Corolla LE'
            [333] StaticText '5700.00 
            [497] StaticText 'Cars + trucks'
            [500] StaticText 'Delmont (Pennsylvania)'
            [339] StaticText '2023/11/16'
            [340] StaticText 'for sale is a 2015 '
            [342] StaticText 'Toyota'
            [343] StaticText " Corolla LE with 176,000 Mi on it. boost Mobiles are mostly Highway. I have four brand new tires for it and I'm scheduled to put them on. The car will be inspected until next December."
    [139] link '\"2015 Toyota Corolla LE, description: The image appears to be a photograph taken during the day under a clear blue sky with some thin clouds. In the foreground, there\\'s a blue car parked on, url: http://127.0.0.1:9980/oc-content/uploads/38129/38129_thumbnail.png\"'
        [218] img '\"2015 Toyota Corolla LE, description: The image appears to be a photograph taken during the day under a clear blue sky with some thin clouds. In the foreground, there\\'s a blue car parked on, url: http://127.0.0.1:9980/oc-content/uploads/38129/38129_thumbnail.png\"'
    [219] LayoutTable ''
        [220] LayoutTableRow ''
            [344] link '2015 Toyota Corolla LE'
            [350] StaticText '5700.00 
            [503] StaticText 'Cars + trucks'
            [506] StaticText 'Borough of Delmont (Pennsylvania)'
            [356] StaticText '2023/11/16'
            [357] StaticText 'For sale is a 2015 '
            [359] StaticText 'Toyota'
            [360] StaticText ' Corolla LE. 176,000 MI. The miles are mostly Highway. I have four brand new tires for it. I have an appointment to put them on on Tuesday 11/22/2023. I will also be getting the car inspected.'
    [141] link '\"Toyota RAV4 2006 6-CYL Sport 4WD, description: The image shows a black SUV parked on a paved surface. The vehicle has a glossy exterior, indicating it is possibly new or well-maintained. The side, url: http://127.0.0.1:9980/oc-content/uploads/65418/65418_thumbnail.png\"'
        [221] img '\"Toyota RAV4 2006 6-CYL Sport 4WD, description: The image shows a black SUV parked on a paved surface. The vehicle has a glossy exterior, indicating it is possibly new or well-maintained. The side, url: http://127.0.0.1:9980/oc-content/uploads/65418/65418_thumbnail.png\"'
    [222] LayoutTable ''
        [223] LayoutTableRow ''
            [361] link 'Toyota RAV4 2006 6-CYL Sport 4WD'
            [367] StaticText '5900.00 
            [509] StaticText 'Cars + trucks'
            [512] StaticText 'Fairfax Station (Virginia)'
            [373] StaticText '2023/11/16'
            [374] StaticText 'Great reliable transportation!! This RAV4 has only had 2 loving owners. All four tires are new as are the brakes and rotors. Spark plugs, coils, valve gasket, and oxygen sensor are new as well. Additional pictures will be provided upon request.'
kohjingyu commented 1 month ago

This should be available in the SoM format but I don't think the accessibility tree supports this (or other select inputs, for that matter). It's a limitation of this action space in general.

leoozy commented 1 month ago

This should be available in the SoM format but I don't think the accessibility tree supports this (or other select inputs, for that matter). It's a limitation of this action space in general.

I understand the hidden selection is not available to Acc_tree. But The sort by newly listed is visiablle, which still disappears at accessibility tree. It is confusing.

leoozy commented 2 weeks ago

Thanks!