search

ShishirPatil / gorilla

Gorilla: Training and Evaluating LLMs for Function Calls (Tool Calls)
https://gorilla.cs.berkeley.edu/
Apache License 2.0
11.2k stars 933 forks source link

The Urban Dictionary from the RapidAPI is not serving, can't evaluate execution data #563

Open zhangyingerjelly opened 1 month ago

zhangyingerjelly commented 1 month ago

The Urban Dictionary from the RapidAPI is not serving, can't evaluate execution data.

CharlieJCJ commented 1 month ago

Hi @zhangyingerjelly, thanks for raising. I just checked Urban Dictionary API on my end. It looks fine to me? Can you share the error message you're facing?

Source: https://rapidapi.com/community/api/urban-dictionary

❯ python                                                          
Python 3.10.13 (main, Sep 11 2023, 08:16:02) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get(url="https://mashape-community-urban-dictionary.p.rapidapi.com/define", headers={"X-RapidAPI-Key": REPLACE_THIS_WITH_YOUR_KEY, "X-RapidAPI-Host": "mashape-community-urban-dictionary.p.rapidapi.com"}, params={"term": "artwash"}).json()
{'list': [{'definition': 'Buying [art] for museums or galleries in order to improve public [opinion] of you, [your family], or your company.', 'permalink': 'http://artwash.urbanup.com/13751556', 'thumbs_up': 450, 'author': 'Briansue', 'word': 'artwash', 'defid': 13751556, 'current_vote': '', 'written_on': '2019-03-27T12:41:12.095Z', 'example': "The [sackler family] couldn't even artwash away their part in [the north] american opioid [crisis].", 'thumbs_down': 182}]}
BeerTai commented 1 month ago

I get the same error

(BFCL) [myname@node17 eval_checker]$ python eval_runner.py --model THUDM/glm-4-9b-chat --test-category python
🦍 Model: THUDM_glm-4-9b-chat
πŸ” Running test: relevance
βœ… Test completed: relevance. 🎯 Accuracy: 0.875
πŸ” Running test: simple
βœ… Test completed: simple. 🎯 Accuracy: 0.8675
πŸ” Running test: executable_multiple_function
---- Getting real-time execution result from ground truth for executable_multiple_function ----
Getting Executable Expected Output:  26%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Ž                                                                                              | 13/50 [02:10<06:10, 10.00s/it]
Traceback (most recent call last):
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/urllib3/connection.py", line 196, in _new_conn
    sock = connection.create_connection(
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/urllib3/connectionpool.py", line 490, in _make_request
    raise new_e
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/urllib3/connectionpool.py", line 466, in _make_request
    self._validate_conn(conn)
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1095, in _validate_conn
    conn.connect()
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/urllib3/connection.py", line 615, in connect
    self.sock = sock = self._new_conn()
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f79dfa791b0>: Failed to establish a new connection: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/urllib3/connectionpool.py", line 843, in urlopen
    retries = retries.increment(
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.urbandictionary.com', port=443): Max retries exceeded with url: /v0/define?term=lit (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f79dfa791b0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/fl/nlp/myname/gorilla-main/berkeley-function-call-leaderboard/eval_checker/eval_runner.py", line 472, in <module>
    runner(model_names, test_categories, api_sanity_check)
  File "/fl/nlp/myname/gorilla-main/berkeley-function-call-leaderboard/eval_checker/eval_runner.py", line 366, in runner
    get_executable_expected_output(prompt_file)
  File "/fl/nlp/myname/gorilla-main/berkeley-function-call-leaderboard/eval_checker/eval_runner_helper.py", line 766, in get_executable_expected_output
    exec(
  File "<string>", line 2, in <module>
  File "/fl/nlp/myname/gorilla-main/berkeley-function-call-leaderboard/eval_checker/executable_python_function.py", line 418, in find_term_on_urban_dictionary
    response = requests.get(url, headers=headers, params=querystring)
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/requests/sessions.py", line 724, in send
    history = [resp for resp in gen]
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/requests/sessions.py", line 724, in <listcomp>
    history = [resp for resp in gen]
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/requests/sessions.py", line 265, in resolve_redirects
    resp = self.send(
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/myname/anaconda3/envs/BFCL/lib/python3.10/site-packages/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.urbandictionary.com', port=443): Max retries exceeded with url: /v0/define?term=lit (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f79dfa791b0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

however，in https://rapidapi.com/community/api/urban-dictionary/playground，we can get the response: image image

chenhengzh commented 1 month ago

I also encountered this problem. Have you resolved it?