search

KoboldAI / KoboldAI-Client

For GGUF support, see KoboldCPP: https://github.com/LostRuins/koboldcpp
https://koboldai.com
GNU Affero General Public License v3.0
3.46k stars 747 forks source link

Atfer letting my pc online at night to generate kudos.... it crashes in function "g" #336

Closed Blazzycrafter closed 1 year ago

Blazzycrafter commented 1 year ago

yes thats the problem...

ERROR      | __main__:g:611 - An error has been caught in function 'g', process 'MainProcess' (10376), thread 'MainThread' (9768):
Traceback (most recent call last):

  File "B:\python\lib\site-packages\urllib3\connection.py", line 200, in _new_conn
    sock = connection.create_connection(
           │          └ <function create_connection at 0x00000136878E64C0>
           └ <module 'urllib3.util.connection' from 'B:\\python\\lib\\site-packages\\urllib3\\util\\connection.py'>

  File "B:\python\lib\site-packages\urllib3\util\connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
               │      │           │     │     │       │      └ <SocketKind.SOCK_STREAM: 1>
               │      │           │     │     │       └ <module 'eventlet.green._socket_nodns' from 'B:\\python\\lib\\site-packages\\eventlet\\green\\_socket_nodns.py'>
               │      │           │     │     └ <AddressFamily.AF_UNSPEC: 0>
               │      │           │     └ 443
               │      │           └ 'horde.koboldai.net'
               │      └ <function getaddrinfo at 0x00000136844DA040>
               └ <module 'eventlet.green._socket_nodns' from 'B:\\python\\lib\\site-packages\\eventlet\\green\\_socket_nodns.py'>

  File "B:\python\lib\socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               │       │           │     │     │       │     │      └ 0
               │       │           │     │     │       │     └ 0
               │       │           │     │     │       └ <SocketKind.SOCK_STREAM: 1>
               │       │           │     │     └ <AddressFamily.AF_UNSPEC: 0>
               │       │           │     └ 443
               │       │           └ 'horde.koboldai.net'
               │       └ <built-in function getaddrinfo>
               └ <module '_socket' from 'B:\\python\\DLLs\\_socket.pyd'>

socket.gaierror: [Errno 11002] getaddrinfo failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "B:\python\lib\site-packages\urllib3\connectionpool.py", line 790, in urlopen
    response = self._make_request(
               │    └ <function HTTPConnectionPool._make_request at 0x0000013687976670>
               └ <urllib3.connectionpool.HTTPSConnectionPool object at 0x00000136A32F4700>

  File "B:\python\lib\site-packages\urllib3\connectionpool.py", line 491, in _make_request
    raise new_e
          └ NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x00000136A9755820>: Failed to resolve 'horde.koboldai.net...

  File "B:\python\lib\site-packages\urllib3\connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
    │    │              └ <urllib3.connection.HTTPSConnection object at 0x00000136A9755820>
    │    └ <function HTTPSConnectionPool._validate_conn at 0x0000013687976A60>
    └ <urllib3.connectionpool.HTTPSConnectionPool object at 0x00000136A32F4700>

  File "B:\python\lib\site-packages\urllib3\connectionpool.py", line 1092, in _validate_conn
    conn.connect()
    │    └ <function HTTPSConnection.connect at 0x000001368796B040>
    └ <urllib3.connection.HTTPSConnection object at 0x00000136A9755820>

  File "B:\python\lib\site-packages\urllib3\connection.py", line 604, in connect
    self.sock = sock = self._new_conn()
    │    │             │    └ <function HTTPConnection._new_conn at 0x0000013687965820>
    │    │             └ <urllib3.connection.HTTPSConnection object at 0x00000136A9755820>
    │    └ None
    └ <urllib3.connection.HTTPSConnection object at 0x00000136A9755820>

  File "B:\python\lib\site-packages\urllib3\connection.py", line 207, in _new_conn
    raise NameResolutionError(self.host, self, e) from e
          │                   │    │     └ <urllib3.connection.HTTPSConnection object at 0x00000136A9755820>
          │                   │    └ <property object at 0x00000136879674F0>
          │                   └ <urllib3.connection.HTTPSConnection object at 0x00000136A9755820>
          └ <class 'urllib3.exceptions.NameResolutionError'>

urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0x00000136A9755820>: Failed to resolve 'horde.koboldai.net' ([Errno 11002] getaddrinfo failed)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "B:\python\lib\site-packages\requests\adapters.py", line 486, in send
    resp = conn.urlopen(
           │    └ <function HTTPConnectionPool.urlopen at 0x0000013687976820>
           └ <urllib3.connectionpool.HTTPSConnectionPool object at 0x00000136A32F4700>

  File "B:\python\lib\site-packages\urllib3\connectionpool.py", line 844, in urlopen
    retries = retries.increment(
              │       └ <function Retry.increment at 0x00000136879058B0>
              └ Retry(total=0, connect=None, read=False, redirect=None, status=None)

  File "B:\python\lib\site-packages\urllib3\util\retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
          │             │      │    │            └ NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x00000136A9755820>: Failed to resolve 'horde.koboldai.net...
          │             │      │    └ NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x00000136A9755820>: Failed to resolve 'horde.koboldai.net...
          │             │      └ '/api/v2/generate/text/submit'
          │             └ <urllib3.connectionpool.HTTPSConnectionPool object at 0x00000136A32F4700>
          └ <class 'urllib3.exceptions.MaxRetryError'>

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='horde.koboldai.net', port=443): Max retries exceeded with url: /api/v2/generate/text/submit (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x00000136A9755820>: Failed to resolve 'horde.koboldai.net' ([Errno 11002] getaddrinfo failed)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "D:\KoboldAI\KoboldAI-Horde-Bridge\bridge.py", line 213, in bridge
    submit_req = requests.post(cluster + '/api/v2/generate/text/submit', json = submit_dict, headers = headers)
                 │        │    │                                                │                      └ {'apikey': 'BoRt1awIQ4Pb6Bm-tRAkCQ'}
                 │        │    │                                                └ {'id': 'f196277e-86f2-483e-b42f-d645479372fa', 'generation': "Trap Dungeon: You don't understand what you just heard. You wer...
                 │        │    └ 'https://horde.koboldai.net'
                 │        └ <function post at 0x0000013687AA94C0>
                 └ <module 'requests' from 'B:\\python\\lib\\site-packages\\requests\\__init__.py'>

  File "B:\python\lib\site-packages\requests\api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
           │               │         │          │       └ {'headers': {'apikey': 'BoRt1awIQ4Pb6Bm-tRAkCQ'}}
           │               │         │          └ {'id': 'f196277e-86f2-483e-b42f-d645479372fa', 'generation': "Trap Dungeon: You don't understand what you just heard. You wer...
           │               │         └ None
           │               └ 'https://horde.koboldai.net/api/v2/generate/text/submit'
           └ <function request at 0x0000013687A7B9D0>

  File "B:\python\lib\site-packages\requests\api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           │       │              │           │      └ {'data': None, 'json': {'id': 'f196277e-86f2-483e-b42f-d645479372fa', 'generation': "Trap Dungeon: You don't understand what ...
           │       │              │           └ 'https://horde.koboldai.net/api/v2/generate/text/submit'
           │       │              └ 'post'
           │       └ <function Session.request at 0x0000013687AA8A60>
           └ <requests.sessions.Session object at 0x00000136A31472E0>

  File "B:\python\lib\site-packages\requests\sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
           │    │    │       └ {'timeout': None, 'allow_redirects': True, 'proxies': OrderedDict(), 'stream': False, 'verify': True, 'cert': None}
           │    │    └ <PreparedRequest [POST]>
           │    └ <function Session.send at 0x0000013687AA8EE0>
           └ <requests.sessions.Session object at 0x00000136A31472E0>

  File "B:\python\lib\site-packages\requests\sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
        │       │    │          └ {'timeout': None, 'proxies': OrderedDict(), 'stream': False, 'verify': True, 'cert': None}
        │       │    └ <PreparedRequest [POST]>
        │       └ <function HTTPAdapter.send at 0x0000013687AA83A0>
        └ <requests.adapters.HTTPAdapter object at 0x00000136A98594F0>

  File "B:\python\lib\site-packages\requests\adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
          │                          └ <PreparedRequest [POST]>
          └ <class 'requests.exceptions.ConnectionError'>

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='horde.koboldai.net', port=443): Max retries exceeded with url: /api/v2/generate/text/submit (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x00000136A9755820>: Failed to resolve 'horde.koboldai.net' ([Errno 11002] getaddrinfo failed)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "B:\python\lib\site-packages\eventlet\green\thread.py", line 43, in __thread_body
    func(*args, **kwargs)
    │     │       └ {}
    │     └ ()
    └ <bound method Thread._bootstrap of <Thread(Thread-41, started daemon 1334156792064)>>

  File "B:\python\lib\threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
    │    └ <function start_new_thread.<locals>.wrap_bootstrap_inner at 0x00000136A96B6550>
    └ <Thread(Thread-41, started daemon 1334156792064)>

  File "B:\python\lib\site-packages\eventlet\green\thread.py", line 64, in wrap_bootstrap_inner
    bootstrap_inner()
    └ <bound method Thread._bootstrap_inner of <Thread(Thread-41, started daemon 1334156792064)>>

  File "B:\python\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x0000013687BED280>
    └ <Thread(Thread-41, started daemon 1334156792064)>

  File "B:\python\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <Thread(Thread-41, started daemon 1334156792064)>
    │    │        │    └ (<socketio.server.Server object at 0x00000136A19BA9A0>, 'e_LniHLxI8CYrpVyAAAF', '1NN6WQ6i1091yHt_AAAE', ['var_change', {'ID':...
    │    │        └ <Thread(Thread-41, started daemon 1334156792064)>
    │    └ <bound method Server._handle_event_internal of <socketio.server.Server object at 0x00000136A19BA9A0>>
    └ <Thread(Thread-41, started daemon 1334156792064)>

  File "B:\python\lib\site-packages\socketio\server.py", line 731, in _handle_event_internal
    r = server._trigger_event(data[0], namespace, sid, *data[1:])
        │      │              │        │          │     └ ['var_change', {'ID': 'system_horde_share', 'value': True}]
        │      │              │        │          └ 'e_LniHLxI8CYrpVyAAAF'
        │      │              │        └ '/'
        │      │              └ ['var_change', {'ID': 'system_horde_share', 'value': True}]
        │      └ <function Server._trigger_event at 0x00000136A1380EE0>
        └ <socketio.server.Server object at 0x00000136A19BA9A0>

  File "B:\python\lib\site-packages\socketio\server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
           │    │        │          │       └ ('e_LniHLxI8CYrpVyAAAF', {'ID': 'system_horde_share', 'value': True})
           │    │        │          └ 'var_change'
           │    │        └ '/'
           │    └ {'/': {'get_model_info': <function get_model_info at 0x00000136A1B024C0>, 'OAI_Key_Update': <function get_oai_models at 0x000...
           └ <socketio.server.Server object at 0x00000136A19BA9A0>

  File "B:\python\lib\site-packages\flask_socketio\__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
           │    │             │        │        │          └ 'e_LniHLxI8CYrpVyAAAF'
           │    │             │        │        └ '/'
           │    │             │        └ 'var_change'
           │    │             └ <function UI_2_var_change at 0x00000136A1B25550>
           │    └ <function SocketIO._handle_event at 0x00000136A17965E0>
           └ <flask_socketio.SocketIO object at 0x00000136A19BAA90>

  File "B:\python\lib\site-packages\flask_socketio\__init__.py", line 828, in _handle_event
    ret = handler(*args)
          │        └ ({'ID': 'system_horde_share', 'value': True},)
          └ <function UI_2_var_change at 0x00000136A1B25550>

> File "aiserver.py", line 611, in g
    return f(*a, **k)
           │  │    └ {}
           │  └ ({'ID': 'system_horde_share', 'value': True},)
           └ <function UI_2_var_change at 0x00000136A1B25280>

  File "aiserver.py", line 6183, in UI_2_var_change
    setattr(koboldai_vars, name, value)
            │              │     └ True
            │              └ 'horde_share'
            └ <koboldai_settings.koboldai_vars object at 0x00000136A1A25F40>

  File "D:\KoboldAI\koboldai_settings.py", line 554, in __setattr__
    setattr(self._system_settings, name, value)
            │    │                 │     └ True
            │    │                 └ 'horde_share'
            │    └ <koboldai_settings.system_settings object at 0x00000136A1A2F7F0>
            └ <koboldai_settings.koboldai_vars object at 0x00000136A1A25F40>

  File "D:\KoboldAI\koboldai_settings.py", line 1371, in __setattr__
    threading.Thread(target=self._horde_pid.bridge, args=(1, api_key, kai_name, kai_url, cluster_url, priority_usernames)).run()
    │         │             │    │          │                │        │         │        │            └ []
    │         │             │    │          │                │        │         │        └ 'https://horde.koboldai.net'
    │         │             │    │          │                │        │         └ 'http://127.0.0.1:8020'
    │         │             │    │          │                │        └ 'BLZ'
    │         │             │    │          │                └ 'BoRt1awIQ4Pb6Bm-tRAkCQ'
    │         │             │    │          └ <function kai_bridge.bridge at 0x00000136A96B6310>
    │         │             │    └ <KoboldAI-Horde-Bridge.bridge.kai_bridge object at 0x00000136B8E00F70>
    │         │             └ <koboldai_settings.system_settings object at 0x00000136A1A2F7F0>
    │         └ <class 'threading.Thread'>
    └ <module 'threading' from 'B:\\python\\lib\\threading.py'>

  File "B:\python\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
    │             │             └ <Thread(Thread-2, initial)>
    │             └ <Thread(Thread-2, initial)>
    └ <Thread(Thread-2, initial)>

  File "D:\KoboldAI\KoboldAI-Horde-Bridge\bridge.py", line 234, in bridge
    error.warning(f"Server {cluster} unavailable during submit. Waiting 10 seconds...")

NameError: name 'error' is not defined
lee-b commented 1 year ago

The crash was in horde-bridge, rather than this project. That project should handle this more gracefully without crashing, but FYI, from the "getaddrinfo...MaxRetryError" part of your tracebacks, the root cause here was probably that you lost network connection, or (less likely) KoboldAI Horde did.

lee-b commented 1 year ago

This appears to be fixed in the upstream koboldai-horde. The line is now logger.error(...) rather than error(...):

https://github.com/db0/KoboldAI-Horde-Bridge/blob/master/bridge.py#L239

This is the 2023-05-15 commit specifically to fix that: https://github.com/db0/KoboldAI-Horde-Bridge/commit/57f7dac43e35244875f09619c8867287794d6f27

This doesn't appear to be included in KoboldAI yet, though.

henk717 commented 1 year ago

Newest horde bridge is approved

Blazzycrafter commented 1 year ago

i dont know if its fixed but you guey worked on it so.... For now I close it ;D