search

THUDM / AgentBench

A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)
https://llmbench.ai
Apache License 2.0
2.15k stars 150 forks source link

webshop task : JVM exception occured #42

Closed Z-ZHHH closed 1 year ago

Z-ZHHH commented 1 year ago

Faced the below error when I ran the webshop task. It seems the code is running in the docker (as mentioned in issue24 ), can anyone pls help? 

jnius.JavaException: JVM exception occurred: /root/workspace/src/tasks/webshop/web_agent_site/../search_engine/indexes does not exist or is not a directory. java.lang.IllegalArgumentException

I checked in the webshop docker:

(webshop) root@62bdd530bd59:/# cd /root/workspace/src/tasks/webshop
bash: cd: /root/workspace/src/tasks/webshop: No such file or directory

In another folder (root/webshop/search_engine), there are some relevant files:

(webshop) root@62bdd530bd59:~/webshop/search_engine# ls
convert_product_file_format.py  indexes_100   indexes_1k          resources      resources_100k  run_indexing.sh
indexes                         indexes_100k  lucene_searcher.py  resources_100  resources_1k

Here is the error information:

(agentbench) GP-TRT-2:~/AgentBench$ bash .assignments/2023-09-14-10-16-52.sh
Evaluating in docker localhost/task:webshop, Parameters: --task outputs/2023-09-14-10-16-52/llama2-7b/WebShop-dev/task.yaml --agent outputs/2023-09-14-10-16-52/llama2-7b/WebShop-dev/agent.yaml --output outputs/2023-09-14-10-16-52/llama2-7b/WebShop-dev
> [Warning] FastChat agent not available
{'module': 'src.tasks.WebShop', 'parameters': {'end': 280, 'name': 'WebShop-dev', 'num_envs': 3, 'start': 200, 'worker_limit': 3, 'workers': 6}}
{'module': 'src.agents.HTTPAgent', 'parameters': {'body': {'Key1': 'Value1', 'Key2': 'Value2'}, 'headers': {'Content-Type': 'application/json'}, 'name': 'llama2-7b', 'prompter': {'args': {'agent_role': 'assistant'}, 'name': 'role_content_dict'}, 'url': 'http://localhost:8000/v1/chat/completions'}}
[Evaluation] Loading Agent ...
[Evaluation] Successfully loaded Agent.
[Evaluation] Loading Task ...
> [Warning] OSInteraction task not available
> [Warning] ALFWorld task not available
> [Warning] DBBench task not available
Warning: Gym version v0.24.0 has a number of critical issues with `gym.make` such that the `reset` and `step` functions are called before returning the environment. It is recommend to downgrading to v0.23.1 or upgrading to v0.25.1
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
  warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(numpy.__version__) >= "1.19":
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/setuptools/_distutils/version.py:345: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/thefuzz/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
  warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentSiteEnv-v0
  logger.warn(f"Overriding environment {spec.id}")
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentTextEnv-v0
  logger.warn(f"Overriding environment {spec.id}")
> [Warning] Mind2Web task not available
> [Warning] KnowledgeGraph task not available
[Evaluation] Successfully loaded Task.
Evaluating task 'WebShop-dev' ...
Start Predicting All ...
  0%|                                                                                                                   | 0/80 [00:00<?, ?it/s]> [Warning] FastChat agent not available
> [Warning] OSInteraction task not available
> [Warning] FastChat agent not available
> [Warning] ALFWorld task not available
> [Warning] DBBench task not available
> [Warning] FastChat agent not available
> [Warning] OSInteraction task not available
> [Warning] ALFWorld task not available
> [Warning] DBBench task not available
Warning: Gym version v0.24.0 has a number of critical issues with `gym.make` such that the `reset` and `step` functions are called before returning the environment. It is recommend to downgrading to v0.23.1 or upgrading to v0.25.1
> [Warning] OSInteraction task not available
> [Warning] ALFWorld task not available
> [Warning] DBBench task not available
Warning: Gym version v0.24.0 has a number of critical issues with `gym.make` such that the `reset` and `step` functions are called before returning the environment. It is recommend to downgrading to v0.23.1 or upgrading to v0.25.1
Warning: Gym version v0.24.0 has a number of critical issues with `gym.make` such that the `reset` and `step` functions are called before returning the environment. It is recommend to downgrading to v0.23.1 or upgrading to v0.25.1
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
  warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
  warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
  warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(numpy.__version__) >= "1.19":
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/setuptools/_distutils/version.py:345: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(numpy.__version__) >= "1.19":
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/setuptools/_distutils/version.py:345: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(numpy.__version__) >= "1.19":
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/setuptools/_distutils/version.py:345: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/thefuzz/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
  warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/thefuzz/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
  warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/thefuzz/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
  warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentSiteEnv-v0
  logger.warn(f"Overriding environment {spec.id}")
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentTextEnv-v0
  logger.warn(f"Overriding environment {spec.id}")
> [Warning] Mind2Web task not available
> [Warning] KnowledgeGraph task not available
Products loaded.
Keys cleaned.
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentSiteEnv-v0
  logger.warn(f"Overriding environment {spec.id}")
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentTextEnv-v0
  logger.warn(f"Overriding environment {spec.id}")
> [Warning] Mind2Web task not available
> [Warning] KnowledgeGraph task not available
Products loaded.
Keys cleaned.
Attributes loaded.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 70730.25it/s]
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/root/miniconda3/envs/webshop/lib/python3.8/site-packages/multiprocess/process.py", line 315, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/webshop/lib/python3.8/site-packages/multiprocess/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/workspace/src/tasks/webshop_docker/__init__.py", line 38, in predict
    env = WebAgentTextEnv(observation_mode="text", human_goals=True)
  File "/root/workspace/src/tasks/webshop_docker/web_agent_site/envs/web_agent_text_env.py", line 61, in __init__
    self.server = SimServer(
  File "/root/workspace/src/tasks/webshop_docker/web_agent_site/envs/web_agent_text_env.py", line 299, in __init__
    self.search_engine = init_search_engine(num_products=num_products)
  File "/root/workspace/src/tasks/webshop/web_agent_site/engine/engine.py", line 206, in init_search_engine
    search_engine = LuceneSearcher(os.path.join(BASE_DIR, f'../search_engine/{indexes}'))
  File "/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pyserini/search/lucene/_searcher.py", line 51, in __init__
    self.object = JLuceneSearcher(index_dir)
  File "jnius/jnius_export_class.pxi", line 270, in jnius.JavaClass.__init__
  File "jnius/jnius_export_class.pxi", line 384, in jnius.JavaClass.call_constructor
  File "jnius/jnius_utils.pxi", line 79, in jnius.check_exception
jnius.JavaException: JVM exception occurred: /root/workspace/src/tasks/webshop/web_agent_site/../search_engine/indexes does not exist or is not a directory. java.lang.IllegalArgumentException
Z-ZHHH commented 1 year ago

The problem is discussed in (https://github.com/THUDM/AgentBench/issues/24#issue-1863168883) , close the issue