search

web-arena-x / webarena

Code repo for "WebArena: A Realistic Web Environment for Building Autonomous Agents"
https://webarena.dev
Apache License 2.0
700 stars 108 forks source link

model_name's without corresponding Tokenizer's do not work (such as gpt4) #95

Closed tdene closed 6 months ago

tdene commented 8 months ago

It appears that the current code-path tries to find a valid Tokenizer for a given model name, and that this Tokenizer is only used to check whether the user's input is shorter than the context window.

This causes the framework to not work for models that do not have a public Tokenizer.

Most evident is the example of "gpt4". However, more relevant, this prevents users from trying their own custom, private, models that are hosted behind some unknown OpenAI-compatible server somewhere that has not published any information about their models, such as their associated tokenizers.

A better approach would be to simply prompt the OpenAI server with the full input, however long it may be, and then listen for an error. OpenAI-compatible servers typically respond with errors along the lines of "your input exceeds the context window by X tokens, please reformat it and try again".

shuyanzhou commented 6 months ago

gpt-4 and other huggingface models should work with our current implementation.