docker-setup with Multi-Tree support fails to start the web container

strombringer commented 5 months ago

I'm trying to setup a Multi-Tree installation with Docker and followed the instructions at https://www.grampsweb.org/Deployment/, but starting the grampsweb container fails with this error message:

/usr/local/lib/python3.11/dist-packages/gramps/plugins/tool/check.py:51: PyGIWarning: Gtk was imported without specifying a version first. Use gi.require_version('Gtk', '3.0') before import to ensure that the right version gets loaded.
  from gi.repository import Gtk
(__main__.py:9): Gtk-CRITICAL **: 16:35:05.519: gtk_icon_theme_get_for_screen: assertion 'GDK_IS_SCREEN (screen)' failed
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/__main__.py", line 234, in <module>
    cli(
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1685, in invoke
    super().invoke(ctx)
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/__main__.py", line 128, in search
    raise ValueError("`tree` is required when multi-tree support is enabled.")
ValueError: `tree` is required when multi-tree support is enabled.

I'm using the docker-compose file linked in the documentation, with minor changes: Port, SECRET_KEY, TREE and MEDIA_PREFIX_TREE are changed

version: "3.7"

services:
  grampsweb: &grampsweb
    image: ghcr.io/gramps-project/grampsweb:latest
    restart: unless-stopped
    ports:
      - "13000:5000"  # host:docker
    environment:
      GRAMPSWEB_SECRET_KEY: "mysecret"
      GRAMPSWEB_TREE: "*"  # will create a new tree if not exists
      GRAMPSWEB_MEDIA_PREFIX_TREE: "True"
      GRAMPSWEB_CELERY_CONFIG__broker_url: "redis://grampsweb_redis:6379/0"
      GRAMPSWEB_CELERY_CONFIG__result_backend: "redis://grampsweb_redis:6379/0"
      GRAMPSWEB_RATELIMIT_STORAGE_URI: redis://grampsweb_redis:6379/1
...

My understanding from the documentation is,

that this configuration should enable the 3 containers to start and
then to be able to open https://myhost:13000 to setup the server administrator account

Is my understanding wrong and I overlooked something? Where should I have setup the tree, that's mentioned in the error message?

Thanks in advance for the support!

DavidMStraub commented 5 months ago

Ok, I think that's a problem with our compose examples for multi-tree. Please try adding entrypoint: [] to the grampsweb service.

strombringer commented 5 months ago

Thank you! With the entrypoint change I'm able to start the containers.

I'm still a bit lost on how to continue from here. Opening the service in the browser gives me the login form, but I don't have a user yet. The user registration doesn't work and from the documentation I know that it can't work in multi-tree mode and I should disable it. The "first-run wizard" that's mentioned in the documentation is not shown.

https://www.grampsweb.org/multi-tree/ says that users are tied to trees. If I want to create a tree, it is recommended to do that via the web api. But to do that, I need a JWT, which I get by authorizing as a user. That's a chicken and egg issue for me :D

DavidMStraub commented 5 months ago

It should show the first run wizard. Please check the container logs.

strombringer commented 5 months ago

These are the logs of a fresh gramps instance that I started and opened in the browser:

[2024-04-21 20:13:10 +0000] [7] [INFO] Starting gunicorn 21.2.0
[2024-04-21 20:13:10 +0000] [7] [INFO] Listening at: http://0.0.0.0:5000 (7)
[2024-04-21 20:13:10 +0000] [7] [INFO] Using worker: sync
[2024-04-21 20:13:10 +0000] [8] [INFO] Booting worker with pid: 8
[2024-04-21 20:13:10 +0000] [9] [INFO] Booting worker with pid: 9
[2024-04-21 20:13:11 +0000] [10] [INFO] Booting worker with pid: 10
[2024-04-21 20:13:11 +0000] [11] [INFO] Booting worker with pid: 11
[2024-04-21 20:13:11 +0000] [12] [INFO] Booting worker with pid: 12
[2024-04-21 20:13:11 +0000] [13] [INFO] Booting worker with pid: 13
[2024-04-21 20:13:11 +0000] [14] [INFO] Booting worker with pid: 14
[2024-04-21 20:13:11 +0000] [15] [INFO] Booting worker with pid: 15
/usr/local/lib/python3.11/dist-packages/gramps/plugins/tool/check.py:51: PyGIWarning: Gtk was imported without specifying a version first. Use gi.require_version('Gtk', '3.0') before import to ensure that the right version gets loaded.
  from gi.repository import Gtk
/usr/local/lib/python3.11/dist-packages/gramps/plugins/tool/check.py:51: PyGIWarning: Gtk was imported without specifying a version first. Use gi.require_version('Gtk', '3.0') before import to ensure that the right version gets loaded.
  from gi.repository import Gtk
/usr/local/lib/python3.11/dist-packages/gramps/plugins/tool/check.py:51: PyGIWarning: Gtk was imported without specifying a version first. Use gi.require_version('Gtk', '3.0') before import to ensure that the right version gets loaded.
  from gi.repository import Gtk
/usr/local/lib/python3.11/dist-packages/gramps/plugins/tool/check.py:51: PyGIWarning: Gtk was imported without specifying a version first. Use gi.require_version('Gtk', '3.0') before import to ensure that the right version gets loaded.
  from gi.repository import Gtk
/usr/local/lib/python3.11/dist-packages/gramps/plugins/tool/check.py:51: PyGIWarning: Gtk was imported without specifying a version first. Use gi.require_version('Gtk', '3.0') before import to ensure that the right version gets loaded.
  from gi.repository import Gtk
(gunicorn:9): Gtk-CRITICAL **: 20:13:19.926: gtk_icon_theme_get_for_screen: assertion 'GDK_IS_SCREEN (screen)' failed
/usr/local/lib/python3.11/dist-packages/gramps/plugins/tool/check.py:51: PyGIWarning: Gtk was imported without specifying a version first. Use gi.require_version('Gtk', '3.0') before import to ensure that the right version gets loaded.
  from gi.repository import Gtk
(gunicorn:12): Gtk-CRITICAL **: 20:13:19.928: gtk_icon_theme_get_for_screen: assertion 'GDK_IS_SCREEN (screen)' failed
/usr/local/lib/python3.11/dist-packages/gramps/plugins/tool/check.py:51: PyGIWarning: Gtk was imported without specifying a version first. Use gi.require_version('Gtk', '3.0') before import to ensure that the right version gets loaded.
  from gi.repository import Gtk
(gunicorn:14): Gtk-CRITICAL **: 20:13:19.941: gtk_icon_theme_get_for_screen: assertion 'GDK_IS_SCREEN (screen)' failed
(gunicorn:10): Gtk-CRITICAL **: 20:13:19.954: gtk_icon_theme_get_for_screen: assertion 'GDK_IS_SCREEN (screen)' failed
/usr/local/lib/python3.11/dist-packages/gramps/plugins/tool/check.py:51: PyGIWarning: Gtk was imported without specifying a version first. Use gi.require_version('Gtk', '3.0') before import to ensure that the right version gets loaded.
  from gi.repository import Gtk
(gunicorn:15): Gtk-CRITICAL **: 20:13:19.961: gtk_icon_theme_get_for_screen: assertion 'GDK_IS_SCREEN (screen)' failed
(gunicorn:8): Gtk-CRITICAL **: 20:13:19.988: gtk_icon_theme_get_for_screen: assertion 'GDK_IS_SCREEN (screen)' failed
(gunicorn:13): Gtk-CRITICAL **: 20:13:19.994: gtk_icon_theme_get_for_screen: assertion 'GDK_IS_SCREEN (screen)' failed
(gunicorn:11): Gtk-CRITICAL **: 20:13:20.039: gtk_icon_theme_get_for_screen: assertion 'GDK_IS_SCREEN (screen)' failed
[2024-04-21 20:14:10 +0000] [11] [ERROR] Error handling request /api/token/create_owner/
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/engine/base.py", line 1970, in _exec_single_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/engine/default.py", line 924, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: users
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/usr/local/lib/python3.11/dist-packages/gunicorn/workers/sync.py", line 178, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1488, in __call__
    return self.wsgi_app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1466, in wsgi_app
    response = self.handle_exception(e)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1463, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 872, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 870, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 855, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/views.py", line 110, in view
    return current_app.ensure_sync(self.dispatch_request)(**kwargs)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/views.py", line 191, in dispatch_request
    return current_app.ensure_sync(meth)(**kwargs)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask_limiter/extension.py", line 1299, in __inner
    return cast(R, flask.current_app.ensure_sync(obj)(*a, **k))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/webargs/core.py", line 649, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/token.py", line 161, in post
    if get_all_user_details(
       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/auth/__init__.py", line 231, in get_all_user_details
    users = query.all()
            ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/orm/query.py", line 2673, in all
    return self._iter().all()  # type: ignore
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/orm/query.py", line 2827, in _iter
    result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
                                                  ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/orm/session.py", line 2306, in execute
    return self._execute_internal(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/orm/session.py", line 2191, in _execute_internal
    result: Result[Any] = compile_state_cls.orm_execute_statement(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement
    result = conn.execute(
             ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/engine/base.py", line 1421, in execute
    return meth(
           ^^^^^
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/sql/elements.py", line 514, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/engine/base.py", line 1643, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/engine/base.py", line 1849, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/engine/base.py", line 1989, in _exec_single_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/engine/base.py", line 2356, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/engine/base.py", line 1970, in _exec_single_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/engine/default.py", line 924, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: users
[SQL: SELECT users.id AS users_id, users.name AS users_name, users.email AS users_email, users.fullname AS users_fullname, users.pwhash AS users_pwhash, users.role AS users_role, users.tree AS users_tree 
FROM users]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

So it seems like the database is not created correctly.

DavidMStraub commented 5 months ago

Did you specify USER_DB_URI? If not, please try that. If yes, please run

docker-compose run grampsweb python3 -m gramps_webapi --config /app/config/config.cfg user migrate

strombringer commented 5 months ago

I had not specified USER_DB_URI, as it was not needed for the single-tree installation.

I now added this to the compose.yml: GRAMPSWEB_USER_DB_URI: "sqlite:////app/users/users.sqlite" This is the default setting anyway, right? At least it did not seem to have an effect.

I then ran docker-compose run grampsweb python3 -m gramps_webapi --config /app/config/config.cfg user migrate and got this:

[+] Running 1/0
 ⠿ Container grampsweb_redis  Running                                                                                                                                  0.0s
/usr/local/lib/python3.11/dist-packages/gramps/plugins/tool/check.py:51: PyGIWarning: Gtk was imported without specifying a version first. Use gi.require_version('Gtk', '3.0') before import to ensure that the right version gets loaded.
  from gi.repository import Gtk

(__main__.py:1): Gtk-CRITICAL **: 15:30:13.940: gtk_icon_theme_get_for_screen: assertion 'GDK_IS_SCREEN (screen)' failed
  FAILED: No config file 'alembic.ini' found, or file has no '[alembic]' section
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/__main__.py", line 234, in <module>
    cli(
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/__main__.py", line 118, in migrate_db
    subprocess.run(cmd, env=env, check=True)
  File "/usr/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'alembic', 'upgrade', 'head']' returned non-zero exit status 255.

I tried the command with and without the GRAMPSWEB_USER_DB_URI setting. both returned the same result. I also cleared the volumes between each try, just to avoid any strange side effects.

DavidMStraub commented 5 months ago

Next try:

docker-compose run --entrypoint="" --rm -w /app/src grampsweb python3 -m gramps_webapi --config /app/config/config.cfg user migrate

Thanks for bearing with me :sweat_smile:

strombringer commented 5 months ago

I'm happy to work through this with your help 😄

The last command worked and the first run wizard is shown in the browser. But when I enter my data and submit, I get a "Wrong username or password" error message: grafik SMTP data left empty, no tree to import.

I don't see any error messages in either of the three containers logs. When I try to submit again, I get the message "Users already exists" (I assume it should be "User"). Username is in the format "firstname-lastname" and password is 48 characters [a-zA-Z0-9], in case it makes a difference.

Going back to the root in the browser, I'm now presented with the login form again. But when trying to login, it says "Error: Wrong username or password"

DavidMStraub commented 5 months ago

Ha, that's weird because it means creating the user worked, but fetching a token didn't work, and the Javascript functions for both operations use the same variables for the username and password.

If you can start over and monitor the API calls in the browser dev console, that would be useful. Perhaps it's some weird special character issue, but I wouldn't see why...

strombringer commented 5 months ago

These are requests for loading the page, entering the user information and sending it (with placeholder data). Everything on a completely fresh installation. grafik

I saved the requests as a HAR file. Github doesn't allow attaching it here, so I uploaded it to Dropbox: (removed, because no longer needed)

Maybe that's helpful for you 😄

For completeness sake:

This is the part that's changed from the default compose.yml file:

    environment:
      GRAMPSWEB_SECRET_KEY: "mysecret"
      GRAMPSWEB_TREE: "*"
      GRAMPSWEB_MEDIA_PREFIX_TREE: "True"
      ...
    entrypoint: []

And I executed this command of yours:

docker-compose run --entrypoint="" --rm -w /app/src grampsweb python3 -m gramps_webapi --config /app/config/config.cfg user migrate

strombringer commented 5 months ago

I just realized I didn't mention this before: I tried the default compose.yml for a single-tree setup and there everything worked out of the box, with the same username and password that I tried for the multi-tree setup. So I don't think it's a special character issue.

DavidMStraub commented 5 months ago

Ok I think I know what's happening.

In the multi-tree setup, the first run wizard creates a site admin, not a tree owner. But the site admin does not have a tree. You would first have to create a tree, or to assign the admin to a specific tree. In a multi-tenant setup, the site admin might anyway just be a service account that is used to create new trees and manage trees, without being itself associated with a tree.

Of course, the UI for that doesn't make sense right now and needs to be changed - I'll open an issue in the frontend repo.

Your next step is to create a new tree via the API by posting to /api/trees/. It will return a tree ID. You can then go to https://yourgrampsweb/firstrun/<treeid> to get the first run wizard for the tree owner.

DavidMStraub commented 5 months ago

(Thanks by the way for the detailed logs, very helpful!)

strombringer commented 5 months ago

For posting to /api/trees, I need a JWT. Otherwise I get a 401 error (as it should be). But when I try POSTing to /api/token with the username and password from the first run wizard, I get a 403 error.

Do I need to copy the token from the web session, for creating the first tree? I expected that the call to api/token should work in any case.

DavidMStraub commented 5 months ago

Right :man_facepalming: I take that back:

Ok I think I know what's happening.

Your logs showed you got a 403 permission denied when trying to fetch a token with username/password, which should work regardless of tree. I will try to reproduce this later, there must be a bug somewhere.

Corepex commented 3 months ago

@strombringer did you manage to create a tree via the api endpoint?

strombringer commented 3 months ago

@Corepex no, I didn't. After the last test in April, I switched to single-tree mode and the setup wizard worked for that. I'll try again when a bugfix is available, though it's not high priority for me.

DavidMStraub commented 3 months ago

FYI, I am running a multi tree setup and creating new trees via the API all the time, so there could be a bug, but it's not obvious to me what it is as I'm not affected by it.

gramps-project / gramps-web-api

docker-setup with Multi-Tree support fails to start the web container #510