search

ItzCrazyKns / Perplexica

Perplexica is an AI-powered search engine. It is an Open source alternative to Perplexity AI
MIT License
10.97k stars 972 forks source link

FileNotFoundError for missing 'framalibre.py' in searxng engine initialization #18

Open shengdade opened 2 months ago

shengdade commented 2 months ago

Describe the bug The application is experiencing multiple startup errors during the initialization of specific search engine modules, particularly "framalibre". The log files also indicate missing configuration files, namely limiter.toml for bot detection settings which may lead to further issues if not addressed.

To Reproduce

docker compose up -d

Expected behavior The expected behavior includes successful initialization of all search engines configured for the service, including "framalibre". There should not be errors pertaining to missing modules or configuration files. The system should ideally handle missing configurations gracefully or prompt the user/administrator to rectify missing data.

Additional context The errors regarding the "framalibre" search engine suggest either an issue with its configuration pointing to a non-existent module or perhaps the module was set to be included without being properly implemented or deployed. Additionally, the recurring warnings about the missing limiter.toml could imply a missing or incorrect path setting or failed configurations, which could affect the bot detection efficiency and may allow harmful traffic that can potentially cause service disruptions.

                    perplexica-backend-1   | yarn run v1.22.19
                    perplexica-backend-1   | $ node dist/app.js
                    perplexica-backend-1   | WebSocket server started on port 3001
                    perplexica-backend-1   | (node:28) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
                    perplexica-backend-1   | (Use `node --trace-deprecation ...` to show where the warning was created)
                    perplexica-backend-1   | API server started on port 3001
                    perplexica-backend-1   | Connection closed
                    perplexica-backend-1   | Connection closed
                    perplexica-frontend-1  | yarn run v1.22.19
                    perplexica-frontend-1  | $ next start
                    perplexica-frontend-1  |    ▲ Next.js 14.1.4
                    perplexica-frontend-1  |    - Local:        http://localhost:3000
                    perplexica-frontend-1  | 
                    perplexica-frontend-1  |  ✓ Ready in 434ms
                    searxng-1              | SearXNG version 2024.4.28+e6c345f89
                    searxng-1              | Create /etc/searxng/uwsgi.ini
                    searxng-1              | Use existing /etc/searxng/settings.yml
                    searxng-1              | Listen on 0.0.0.0:8080
                    searxng-1              | [uWSGI] getting INI configuration from /etc/searxng/uwsgi.ini
                    searxng-1              | [uwsgi-static] added mapping for /static => /usr/local/searxng/searx/static
                    searxng-1              | *** Starting uWSGI 2.0.23 (64bit) on [Sun Apr 28 12:33:06 2024] ***
                    searxng-1              | compiled with version: 13.2.1 20231014 on 30 November 2023 14:34:33
                    searxng-1              | os: Linux-6.6.22-linuxkit #1 SMP Fri Mar 29 12:21:27 UTC 2024
                    searxng-1              | nodename: ed72579ef066
                    searxng-1              | machine: aarch64
                    searxng-1              | clock source: unix
                    searxng-1              | pcre jit disabled
                    searxng-1              | detected number of CPU cores: 12
                    searxng-1              | current working directory: /usr/local/searxng
                    searxng-1              | detected binary path: /usr/sbin/uwsgi
                    searxng-1              | chdir() to /usr/local/searxng/searx/
                    searxng-1              | your memory page size is 4096 bytes
                    searxng-1              | detected max file descriptor number: 1048576
                    searxng-1              | building mime-types dictionary from file /etc/mime.types...1390 entry found
                    searxng-1              | lock engine: pthread robust mutexes
                    searxng-1              | thunder lock: disabled (you can enable it with --thunder-lock)
                    searxng-1              | uwsgi socket 0 bound to TCP address 0.0.0.0:8080 fd 3
                    searxng-1              | Python version: 3.11.9 (main, Apr 14 2024, 13:40:00) [GCC 13.2.1 20231014]
                    searxng-1              | Python main interpreter initialized at 0xffffaaa10ac0
                    searxng-1              | python threads support enabled
                    searxng-1              | your server socket listen backlog is limited to 100 connections
                    searxng-1              | your mercy for graceful operations on workers is 60 seconds
                    searxng-1              | mapped 1568736 bytes (1531 KB) for 48 cores
                    searxng-1              | *** Operational MODE: preforking+threaded ***
                    searxng-1              | added /usr/local/searxng/ to pythonpath.
                    searxng-1              | spawned uWSGI master process (pid: 7)
                    searxng-1              | spawned uWSGI worker 1 (pid: 12, cores: 4)
                    searxng-1              | spawned uWSGI worker 2 (pid: 14, cores: 4)
                    searxng-1              | spawned 4 offload threads for uWSGI worker 1
                    searxng-1              | spawned uWSGI worker 3 (pid: 19, cores: 4)
                    searxng-1              | spawned 4 offload threads for uWSGI worker 2
                    searxng-1              | spawned uWSGI worker 4 (pid: 23, cores: 4)
                    searxng-1              | spawned 4 offload threads for uWSGI worker 3
                    searxng-1              | spawned uWSGI worker 5 (pid: 28, cores: 4)
                    searxng-1              | spawned uWSGI worker 6 (pid: 29, cores: 4)
                    searxng-1              | spawned uWSGI worker 7 (pid: 35, cores: 4)
                    searxng-1              | spawned 4 offload threads for uWSGI worker 4
                    searxng-1              | spawned 4 offload threads for uWSGI worker 5
                    searxng-1              | spawned uWSGI worker 8 (pid: 42, cores: 4)
                    searxng-1              | spawned uWSGI worker 9 (pid: 46, cores: 4)
                    searxng-1              | spawned 4 offload threads for uWSGI worker 7
                    searxng-1              | spawned 4 offload threads for uWSGI worker 6
                    searxng-1              | spawned uWSGI worker 10 (pid: 54, cores: 4)
                    searxng-1              | spawned 4 offload threads for uWSGI worker 8
                    searxng-1              | spawned 4 offload threads for uWSGI worker 9
                    searxng-1              | spawned uWSGI worker 11 (pid: 58, cores: 4)
                    searxng-1              | spawned uWSGI worker 12 (pid: 59, cores: 4)
                    searxng-1              | spawned 4 offload threads for uWSGI worker 10
                    searxng-1              | spawned 4 offload threads for uWSGI worker 11
                    searxng-1              | spawned 4 offload threads for uWSGI worker 12
                    searxng-1              | 2024-04-28 12:33:07,066 ERROR:searx.engines: Cannot load engine "framalibre"
                    searxng-1              | Traceback (most recent call last):
                    searxng-1              |   File "/usr/local/searxng/searx/engines/__init__.py", line 122, in load_engine
                    searxng-1              |     engine = load_module(module_name + '.py', ENGINE_DIR)
                    searxng-1              |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    searxng-1              |   File "/usr/local/searxng/searx/utils.py", line 428, in load_module
                    searxng-1              |     spec.loader.exec_module(module)
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 936, in exec_module
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1073, in get_code
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1130, in get_data
                    searxng-1              | FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/searxng/searx/engines/framalibre.py'
                    searxng-1              | 2024-04-28 12:33:07,066 ERROR:searx.engines: Cannot load engine "framalibre"
                    searxng-1              | Traceback (most recent call last):
                    searxng-1              |   File "/usr/local/searxng/searx/engines/__init__.py", line 122, in load_engine
                    searxng-1              |     engine = load_module(module_name + '.py', ENGINE_DIR)
                    searxng-1              |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    searxng-1              |   File "/usr/local/searxng/searx/utils.py", line 428, in load_module
                    searxng-1              |     spec.loader.exec_module(module)
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 936, in exec_module
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1073, in get_code
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1130, in get_data
                    searxng-1              | FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/searxng/searx/engines/framalibre.py'
                    searxng-1              | 2024-04-28 12:33:07,067 ERROR:searx.engines: Cannot load engine "framalibre"
                    searxng-1              | Traceback (most recent call last):
                    searxng-1              |   File "/usr/local/searxng/searx/engines/__init__.py", line 122, in load_engine
                    searxng-1              |     engine = load_module(module_name + '.py', ENGINE_DIR)
                    searxng-1              |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    searxng-1              |   File "/usr/local/searxng/searx/utils.py", line 428, in load_module
                    searxng-1              |     spec.loader.exec_module(module)
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 936, in exec_module
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1073, in get_code
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1130, in get_data
                    searxng-1              | FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/searxng/searx/engines/framalibre.py'
                    searxng-1              | 2024-04-28 12:33:07,066 ERROR:searx.engines: Cannot load engine "framalibre"
                    searxng-1              | Traceback (most recent call last):
                    searxng-1              |   File "/usr/local/searxng/searx/engines/__init__.py", line 122, in load_engine
                    searxng-1              |     engine = load_module(module_name + '.py', ENGINE_DIR)
                    searxng-1              |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    searxng-1              |   File "/usr/local/searxng/searx/utils.py", line 428, in load_module
                    searxng-1              |     spec.loader.exec_module(module)
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 936, in exec_module
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1073, in get_code
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1130, in get_data
                    searxng-1              | FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/searxng/searx/engines/framalibre.py'
                    searxng-1              | 2024-04-28 12:33:07,067 ERROR:searx.engines: Cannot load engine "framalibre"
                    searxng-1              | Traceback (most recent call last):
                    searxng-1              |   File "/usr/local/searxng/searx/engines/__init__.py", line 122, in load_engine
                    searxng-1              |     engine = load_module(module_name + '.py', ENGINE_DIR)
                    searxng-1              |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    searxng-1              |   File "/usr/local/searxng/searx/utils.py", line 428, in load_module
                    searxng-1              |     spec.loader.exec_module(module)
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 936, in exec_module
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1073, in get_code
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1130, in get_data
                    searxng-1              | FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/searxng/searx/engines/framalibre.py'
                    searxng-1              | 2024-04-28 12:33:07,067 ERROR:searx.engines: Cannot load engine "framalibre"
                    searxng-1              | Traceback (most recent call last):
                    searxng-1              |   File "/usr/local/searxng/searx/engines/__init__.py", line 122, in load_engine
                    searxng-1              |     engine = load_module(module_name + '.py', ENGINE_DIR)
                    searxng-1              |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    searxng-1              |   File "/usr/local/searxng/searx/utils.py", line 428, in load_module
                    searxng-1              |     spec.loader.exec_module(module)
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 936, in exec_module
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1073, in get_code
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1130, in get_data
                    searxng-1              | FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/searxng/searx/engines/framalibre.py'
                    searxng-1              | 2024-04-28 12:33:07,067 ERROR:searx.engines: Cannot load engine "framalibre"
                    searxng-1              | Traceback (most recent call last):
                    searxng-1              |   File "/usr/local/searxng/searx/engines/__init__.py", line 122, in load_engine
                    searxng-1              |     engine = load_module(module_name + '.py', ENGINE_DIR)
                    searxng-1              |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    searxng-1              |   File "/usr/local/searxng/searx/utils.py", line 428, in load_module
                    searxng-1              |     spec.loader.exec_module(module)
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 936, in exec_module
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1073, in get_code
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1130, in get_data
                    searxng-1              | FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/searxng/searx/engines/framalibre.py'
                    searxng-1              | 2024-04-28 12:33:07,067 ERROR:searx.engines: Cannot load engine "framalibre"
                    searxng-1              | Traceback (most recent call last):
                    searxng-1              |   File "/usr/local/searxng/searx/engines/__init__.py", line 122, in load_engine
                    searxng-1              |     engine = load_module(module_name + '.py', ENGINE_DIR)
                    searxng-1              |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    searxng-1              |   File "/usr/local/searxng/searx/utils.py", line 428, in load_module
                    searxng-1              |     spec.loader.exec_module(module)
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 936, in exec_module
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1073, in get_code
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1130, in get_data
                    searxng-1              | FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/searxng/searx/engines/framalibre.py'
                    searxng-1              | 2024-04-28 12:33:07,067 ERROR:searx.engines: Cannot load engine "framalibre"
                    searxng-1              | Traceback (most recent call last):
                    searxng-1              |   File "/usr/local/searxng/searx/engines/__init__.py", line 122, in load_engine
                    searxng-1              |     engine = load_module(module_name + '.py', ENGINE_DIR)
                    searxng-1              |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    searxng-1              |   File "/usr/local/searxng/searx/utils.py", line 428, in load_module
                    searxng-1              |     spec.loader.exec_module(module)
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 936, in exec_module
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1073, in get_code
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1130, in get_data
                    searxng-1              | FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/searxng/searx/engines/framalibre.py'
                    searxng-1              | 2024-04-28 12:33:07,067 ERROR:searx.engines: Cannot load engine "framalibre"
                    searxng-1              | Traceback (most recent call last):
                    searxng-1              |   File "/usr/local/searxng/searx/engines/__init__.py", line 122, in load_engine
                    searxng-1              |     engine = load_module(module_name + '.py', ENGINE_DIR)
                    searxng-1              |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    searxng-1              |   File "/usr/local/searxng/searx/utils.py", line 428, in load_module
                    searxng-1              |     spec.loader.exec_module(module)
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 936, in exec_module
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1073, in get_code
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1130, in get_data
                    searxng-1              | FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/searxng/searx/engines/framalibre.py'
                    searxng-1              | 2024-04-28 12:33:07,068 ERROR:searx.engines: Cannot load engine "framalibre"
                    searxng-1              | Traceback (most recent call last):
                    searxng-1              |   File "/usr/local/searxng/searx/engines/__init__.py", line 122, in load_engine
                    searxng-1              |     engine = load_module(module_name + '.py', ENGINE_DIR)
                    searxng-1              |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    searxng-1              |   File "/usr/local/searxng/searx/utils.py", line 428, in load_module
                    searxng-1              |     spec.loader.exec_module(module)
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 936, in exec_module
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1073, in get_code
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1130, in get_data
                    searxng-1              | FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/searxng/searx/engines/framalibre.py'
                    searxng-1              | 2024-04-28 12:33:07,066 ERROR:searx.engines: Cannot load engine "framalibre"
                    searxng-1              | Traceback (most recent call last):
                    searxng-1              |   File "/usr/local/searxng/searx/engines/__init__.py", line 122, in load_engine
                    searxng-1              |     engine = load_module(module_name + '.py', ENGINE_DIR)
                    searxng-1              |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    searxng-1              |   File "/usr/local/searxng/searx/utils.py", line 428, in load_module
                    searxng-1              |     spec.loader.exec_module(module)
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 936, in exec_module
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1073, in get_code
                    searxng-1              |   File "<frozen importlib._bootstrap_external>", line 1130, in get_data
                    searxng-1              | FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/searxng/searx/engines/framalibre.py'
                    searxng-1              | 2024-04-28 12:33:07,102 WARNING:searx.botdetection.config: missing config file: /etc/searxng/limiter.toml
                    searxng-1              | 2024-04-28 12:33:07,103 WARNING:searx.botdetection.config: missing config file: /etc/searxng/limiter.toml
                    searxng-1              | WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0xffffaaa10ac0 pid: 29 (default app)
                    searxng-1              | WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0xffffaaa10ac0 pid: 14 (default app)
                    searxng-1              | 2024-04-28 12:33:07,105 WARNING:searx.botdetection.config: missing config file: /etc/searxng/limiter.toml
                    searxng-1              | WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0xffffaaa10ac0 pid: 42 (default app)
                    searxng-1              | 2024-04-28 12:33:07,105 WARNING:searx.botdetection.config: missing config file: /etc/searxng/limiter.toml
                    searxng-1              | 2024-04-28 12:33:07,105 WARNING:searx.botdetection.config: missing config file: /etc/searxng/limiter.toml
                    searxng-1              | WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0xffffaaa10ac0 pid: 59 (default app)
                    searxng-1              | WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0xffffaaa10ac0 pid: 12 (default app)
                    searxng-1              | 2024-04-28 12:33:07,112 WARNING:searx.botdetection.config: missing config file: /etc/searxng/limiter.toml
                    searxng-1              | WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0xffffaaa10ac0 pid: 28 (default app)
                    searxng-1              | 2024-04-28 12:33:07,113 WARNING:searx.botdetection.config: missing config file: /etc/searxng/limiter.toml
                    searxng-1              | WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0xffffaaa10ac0 pid: 35 (default app)
                    searxng-1              | 2024-04-28 12:33:07,114 WARNING:searx.botdetection.config: missing config file: /etc/searxng/limiter.toml
                    searxng-1              | 2024-04-28 12:33:07,113 WARNING:searx.botdetection.config: missing config file: /etc/searxng/limiter.toml
                    searxng-1              | 2024-04-28 12:33:07,114 WARNING:searx.botdetection.config: missing config file: /etc/searxng/limiter.toml
                    searxng-1              | WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0xffffaaa10ac0 pid: 46 (default app)
                    searxng-1              | WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0xffffaaa10ac0 pid: 23 (default app)
                    searxng-1              | WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0xffffaaa10ac0 pid: 19 (default app)
                    searxng-1              | 2024-04-28 12:33:07,114 WARNING:searx.botdetection.config: missing config file: /etc/searxng/limiter.toml
                    searxng-1              | WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0xffffaaa10ac0 pid: 54 (default app)
                    searxng-1              | 2024-04-28 12:33:07,118 WARNING:searx.botdetection.config: missing config file: /etc/searxng/limiter.toml
                    searxng-1              | WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0xffffaaa10ac0 pid: 58 (default app)
ItzCrazyKns commented 2 months ago

Seems like SearxNG removed the framalibre engine, it is present in our SearxNG settings file so that might be causing an issue.

ItzCrazyKns commented 2 months ago

I reproduced the issue and I see that SearxNG works completely fine, it is not affected by that error. Please ignore any errors (except for something major) from SearxNG. If you face any issues, feel free to re-open this issue. Thanks

shengdade commented 2 months ago

It might not be related to SearxNG. I can open http://localhost:3000/, but when I search for something, it gets stuck at loading. The browser console states, "WebSocket is already in CLOSING or CLOSED state."

Below are the Docker logs for perplexica-perplexica-backend-1:

2024-04-28 09:26:32 yarn run v1.22.19
2024-04-28 09:26:32 $ node dist/app.js
2024-04-28 09:26:33 WebSocket server started on port 3001
2024-04-28 09:26:33 API server started on port 3001
2024-04-28 09:26:37 Connection closed
2024-04-28 09:26:33 (node:28) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
2024-04-28 09:26:33 (Use `node --trace-deprecation ...` to show where the warning was created)
2024-04-28 09:28:01 Connection closed

It might be related to port 3001 on my Apple M3.

ItzCrazyKns commented 2 months ago

Try refreshing your page. If that doesn't work provide me with more context so I can help.

ItzCrazyKns commented 2 months ago

Additionally, it can be related to Ollama, if you don't use the right model name. You can update the model to use from the settings dialog.