search

benbusby / whoogle-search

A self-hosted, ad-free, privacy-respecting metasearch engine
https://pypi.org/project/whoogle-search/
MIT License
9.13k stars 914 forks source link

[BUG] frequent 504 behind NPM #1163

Open working-name opened 2 weeks ago

working-name commented 2 weeks ago

Describe the bug For some reason I find myself doing docker restart the more I use search throughout the day. The container is just sitting pretty there, no load. Nothing in NPM's logs other than timeout connecting to whoogle container. Private instance, I see nothing in NPM logs to make me think someone anyone else is hitting it outside of me.

2024/06/26 13:24:21 [error] 878044#878044: *597529 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 1.1.1.1, server: whoogle.domain, request: "GET /search?q=Lumen HTTP/2.0", upstream: "http://172.17.0.1:18084/search?q=Lumen", host: "whoogle.domain"

Container sitting pretty, not choked, all the while ignoring incoming requests. Normally PIDS are about 8 when idle, so there's a few hanging for sure.

CONTAINER ID   NAME             CPU %     MEM USAGE / LIMIT   MEM %     NET I/O        BLOCK I/O       PIDS
4be21cef5885   whoogle-search   0.15%     128.6MiB / 1GiB     12.56%    102MB / 75MB   108MB / 168kB   15

To Reproduce Steps to reproduce the behavior:

  1. Run whoogle via docker, place it behind NPM
  2. Use as you would normally
  3. Bump into 504 errors

Deployment Method

Version of Whoogle Search

Desktop (please complete the following information):

Smartphone (please complete the following information): NOT applicable.

Additional context The container either recovers by itself after a while (is it single threaded and it waits for something to time out?), or it never does and I have to manually restart it.

It's not NPM, I checked logs, it passes the request along. I SSHed into the box, and ran a curl against whoogle container's locally exposed port, and it times out. It's 100% whoogle.

There's no useful errors logs in docker logs except this repeating issue with simple-theme.css:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1348, in handle_user_exception
    return self.ensure_sync(handler)(e)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/whoogle/app/routes.py", line 624, in internal_error
    return render_template(
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/templating.py", line 151, in render_template
    return _render(app, template, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/templating.py", line 132, in _render
    rv = template.render(context)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/whoogle/app/templates/error.html", line 8, in top-level template code
    <link rel="stylesheet" href="{{ cb_url(config.theme + '-theme.css') }}"/>
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/whoogle/app/__init__.py", line 179, in <lambda>
    cb_url=lambda f: app.config['CACHE_BUSTING_MAP'][f])
                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
KeyError: 'simple-theme.css'
working-name commented 2 weeks ago

Found it dead again. This time the container says unhealthy. I just issued a killall -9 curl to kill all the health check processes that hung. Here's what's running on it:

$ docker compose exec whoogle-search ps aux
PID   USER     TIME  COMMAND
    1 whoogle   0:00 {run} /bin/sh ./run
    7 whoogle   0:00 {start-tor.sh} /bin/sh misc/tor/start-tor.sh
   11 whoogle   0:12 tor -f /etc/tor/torrc
   15 whoogle   0:59 python3 -um app --host 0.0.0.0 --port 5000
12776 whoogle   0:00 ps aux

Trying to curl from within the container results in the same behavior: timeout. Does this have to do with Tor? I don't remember enabling it.

$ docker compose exec whoogle-search curl localhost:5000 -vvv -m 2
* Host localhost:5000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:5000...
* Immediate connect fail for ::1: Address not available
*   Trying 127.0.0.1:5000...
* Connected to localhost (127.0.0.1) port 5000
> GET / HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/8.5.0
> Accept: */*
>
* Operation timed out after 2002 milliseconds with 0 bytes received
* Closing connection
curl: (28) Operation timed out after 2002 milliseconds with 0 bytes received

Okay, let's rebuild the container.

$ docker compose exec whoogle-search curl localhost:5000 -vvv -m 2; docker compose up -d --force-recreate && sleep 10 && docker compose exec whoogle-search curl localhost:5000 -vvv -m 2
* Host localhost:5000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:5000...
* Immediate connect fail for ::1: Address not available
*   Trying 127.0.0.1:5000...
* Connected to localhost (127.0.0.1) port 5000
> GET / HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/8.5.0
> Accept: */*
>
* Operation timed out after 2002 milliseconds with 0 bytes received
* Closing connection
curl: (28) Operation timed out after 2002 milliseconds with 0 bytes received
[+] Running 1/1
 ✔ Container whoogle-search  Started                                                                                                                                            20.2s
* Host localhost:5000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:5000...
* Immediate connect fail for ::1: Address not available
*   Trying 127.0.0.1:5000...
* Connected to localhost (127.0.0.1) port 5000
> GET / HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/8.5.0
> Accept: */*
>
* Operation timed out after 2002 milliseconds with 0 bytes received
* Closing connection
curl: (28) Operation timed out after 2002 milliseconds with 0 bytes received
### 10 seconds was not enough for the process to be ready...
containme@geek23:~/apps/whoogle-search$ docker compose exec whoogle-search curl localhost:5000 -vvv -m 2
* Host localhost:5000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:5000...
* Immediate connect fail for ::1: Address not available
*   Trying 127.0.0.1:5000...
* Connected to localhost (127.0.0.1) port 5000
> GET / HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Cache-Control: max-age=86400
< Content-Length: 12765
< Content-Type: text/html; charset=utf-8
< Date: Thu, 27 Jun 2024 03:21:07 GMT
< Server: waitress
< Set-Cookie: __Secure-session=.eJw1zMsKgkAUANBfibt2wEc-ElrmokDQRoU2cZ1H2ugU5kAl_nuzaXdWZ4HrU0wjaqFnSOfJCAfQzN3f7KFlf4N0WR1Q4mMBmxZSyN2waLMBc3_omrv6nvXwLrxOtqouK3rpyoDneKhPdcPCagxfNDhONMtURd092MuYntuGyZh7AhOCPjKy3blIEtm6JGERjzGIrD1Yf9UWNKw.Znzaow.JVGZZBm2hG4RsYpAPSdyjLnUzgU; Expires=Fri, 27 Jun 2025 03:21:07 GMT; Secure; HttpOnly; Path=/; SameSite=Lax
< Vary: Cookie
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
<
<html style="background: #000;">
<head>
... and so on. Normal response.