search

lavague-ai / LaVague

Large Action Model framework to develop AI Web Agents
https://docs.lavague.ai/en/latest/
Apache License 2.0
5.49k stars 505 forks source link

lavague-test feature request to be able to specify X number of sites #504

Open lyie28 opened 3 months ago

lyie28 commented 3 months ago

It is a bit limiting to be able to only run one specified site with --site OR all sites in sites.

I would like to see either a --n_tests argument which allows you to say, let's run 5 tests for example. This could be particularly useful if we add many tests into sites as people may or may not want to run all the tests.

While you can specify one site with --site, you can't specify multiple, so it's a bit limiting.

adeprez commented 3 months ago

--site argument can be repeated so you can test multiple sites:

lavague-test -s examples -s hugginface.co -s reddit.com

Having an additional --n_tests to automatically pick sites is feasible too if you think it's relevant. In this case would you expect lavague-test to pick n websites randomly or just the first n in natural order to make it more deterministic?

lyie28 commented 3 months ago

Ah, yes! I forgot to repeat the -s - I think it could still be interesting to have a --n_tests as the number of site examples grows to avoid having to write out X number of tests.

Good question - I think the deterministic approach is preferable for comparisons.