Python version

[Ala321]

Hi I want to run build_latest.sh but i have error below

pleas update /indico-containers/indico/worker/Dockerfile to FROM python:3.12-bullseye

after that build is ok but i something is wrong

[ThiefMaster]

For Indico v3.3 you need to use the code from the #28 branch

[Ala321]

@ThiefMaster thx that is working now, but i have problem with db. I want to migrate db form 2.3.4 version and i trying to do this using pgdump from old db and run on container like this cat /home/indico_04-06-24.sql | docker exec -i indico-prod-indico-postgres-1 psql -U indico -d indico after that i try to run on indico-prod-indico-web-1 but i still have no events and users form old db. what i'm missing ?

[ThiefMaster]

Did you do docker compose down by any chance? If yes, that may have deleted your database volume, since it's an anonymous volume and not a named one...

Edit: NVM I see you restored a dump. So that should be fine. Was there any interesting output when restoring the dump?

[Ala321]

yes many errors , i need to investigate that. I checked with 3.2.8 the same errors after login i can see

[ThiefMaster]

Please check indico.log (or possibly the output of the indico web container) for the full traceback of that error.

[Ala321]

2024-06-14 18:45:52,674  ERROR    f727bf04a01a4147  -       indico.flask              'NoneType' object has no attribute '_title'
2024-06-05 17:12:47,655  ERROR    a00b61a6bd9f40a3  268     indico.flask              (psycopg2.errors.UndefinedColumn) column persons_1.affiliation_id does not exist
2024-06-14 19:19:39,767  ERROR    0000000000000000  -       indico.plugins            Plugin payment_manual does not exist
2024-06-14 19:21:49,418  ERROR    0000000000000000  -       indico.plugins            Plugin previewer_code does not exist
2024-06-14 18:50:25,814  ERROR    0000000000000000  -       indico.plugins            Plugin vc_zoom does not exist

[ThiefMaster]

You need to remove those plugins from your indico.conf's PLUGINS entry (or install them).

[Ala321]

I want to ask about this migration steps , after set up all this container i need to drop indico db from postgres befor i restore them form my production backup form version 2.3 or not ?

[ThiefMaster]

If your postgres container keeps the data that's ideal - no need for the extra hassle of dumping and restoring the DB. You just need to execute indico db upgrade inside the indico container to perform the database structure upgrades.

Nonetheless, if this is not just a test system make sure that you have a proper dump of your database (outside the containers!)

[Ala321]

i have old indico on one server - version 2.3. Im trying to run a new indico on diffrent machine using docker. After creating containers I dropped database created by scritps from v 3.3.2 and I restored my backup version 2.3 and after that I ran docker exec -i indico-prod-indico-web-1 /opt/indico/.venv/bin/indico db upgrade

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade e787389ca868 -> 26985db8ed12, Add attach_ical to reminders
INFO  [alembic.runtime.migration] Running upgrade 26985db8ed12 -> f26c201c8254, Add attach_ical to RegistrationForm
INFO  [alembic.runtime.migration] Running upgrade f26c201c8254 -> 3782de7970da, Rename oauth default_scopes
INFO  [alembic.runtime.migration] Running upgrade 3782de7970da -> da06d8f50342, Separate authorized scopes from tokens
INFO  [alembic.runtime.migration] Running upgrade da06d8f50342 -> c36abe1c23c7, Make OAuth PKCE flow configurable
INFO  [alembic.runtime.migration] Running upgrade c36abe1c23c7 -> d354278c6d95, Store tokens as hashes
INFO  [alembic.runtime.migration] Running upgrade d354278c6d95 -> ecc7088914e7, Use cascading FKs for oauth
INFO  [alembic.runtime.migration] Running upgrade ecc7088914e7 -> 26806768cd3f, Remove Flower oauth app
INFO  [alembic.runtime.migration] Running upgrade 26806768cd3f -> d89585afaf2e, Add index on user merged_into_id
INFO  [alembic.runtime.migration] Running upgrade d89585afaf2e -> cf9e1b4e2f5f, Restrict event creation if cat has subcats
INFO  [alembic.runtime.migration] Running upgrade cf9e1b4e2f5f -> 178d297eae7e, Disallow seconds in durations
INFO  [alembic.runtime.migration] Running upgrade 178d297eae7e -> 735dc4e8d2f3, Update merged users in registrations
INFO  [alembic.runtime.migration] Running upgrade 735dc4e8d2f3 -> fda76e047e87, Make sure contributions have titles
INFO  [alembic.runtime.migration] Running upgrade fda76e047e87 -> 79e770865675, Add more FTS indexes
INFO  [alembic.runtime.migration] Running upgrade 79e770865675 -> 90384b9b3d22, Make room division non-nullable
INFO  [alembic.runtime.migration] Running upgrade 90384b9b3d22 -> 356b8985ae7c, Add anonymous_submissions table for surveys
INFO  [alembic.runtime.migration] Running upgrade 356b8985ae7c -> 1f6738730753, Add personal access tokens
INFO  [alembic.runtime.migration] Running upgrade 1f6738730753 -> 1b7e98f581bc, Make token scopes not nullable
INFO  [alembic.runtime.migration] Running upgrade 1b7e98f581bc -> 5cbb0eb12eb3, Add last_used_ip and use_count to tokens
INFO  [alembic.runtime.migration] Running upgrade 5cbb0eb12eb3 -> 420195768776, Add reminder send_to_speakers column
INFO  [alembic.runtime.migration] Running upgrade 420195768776 -> cd3fef2095b4, Add is_flat_view_enabled setting for categories
INFO  [alembic.runtime.migration] Running upgrade cd3fef2095b4 -> 1cec32e42f65, Add reviewed_dt to paper revisions
INFO  [alembic.runtime.migration] Running upgrade 1cec32e42f65 -> 4b097412a8d9, Add category logs
INFO  [alembic.runtime.migration] Running upgrade 4b097412a8d9 -> 9d00917b2fa8, Add event category request
INFO  [alembic.runtime.migration] Running upgrade 9d00917b2fa8 -> dc53d6e8c576, Make event category nullable
INFO  [alembic.runtime.migration] Running upgrade dc53d6e8c576 -> b36825c7869e, Add registration tags
INFO  [alembic.runtime.migration] Running upgrade b36825c7869e -> 8993132179d3, Ensure single_choice fields have item_type
INFO  [alembic.runtime.migration] Running upgrade 8993132179d3 -> 84874a17eaf3, Cleanup registration fields is_billable
INFO  [alembic.runtime.migration] Running upgrade 84874a17eaf3 -> 3dafee32ba7d, Remove is_billable from registration fields
INFO  [alembic.runtime.migration] Running upgrade 3dafee32ba7d -> 82fb6c6ac6db, Add publish registrations with consent
INFO  [alembic.runtime.migration] Running upgrade 82fb6c6ac6db -> ef7a8b2e6737, Add access column to menu entries
INFO  [alembic.runtime.migration] Running upgrade ef7a8b2e6737 -> 5123f24eb41e, Add retention period to RegistrationFormItem
INFO  [alembic.runtime.migration] Running upgrade 5123f24eb41e -> 57696d76f9b0, Add registration visibility override
INFO  [alembic.runtime.migration] Running upgrade 57696d76f9b0 -> a61ce4bd7549, Add publish registrations duration to registration forms
INFO  [alembic.runtime.migration] Running upgrade a61ce4bd7549 -> 88eb87ee0d3e, Add retention period to RegistrationForm
INFO  [alembic.runtime.migration] Running upgrade 88eb87ee0d3e -> a707753d16e2, Add 'is_purged' flag to registration forms
INFO  [alembic.runtime.migration] Running upgrade a707753d16e2 -> 812aa90a3660, Add favorite events table
INFO  [alembic.runtime.migration] Running upgrade 812aa90a3660 -> c39db219f85a, Add predefined affiliations
INFO  [alembic.runtime.migration] Running upgrade c39db219f85a -> 1950e5d12ab5, Add predefined affiliations to persons
INFO  [alembic.runtime.migration] Running upgrade 1950e5d12ab5 -> 0c4bb2973536, Move affiliation into users table
INFO  [alembic.runtime.migration] Running upgrade 0c4bb2973536 -> 33c3ab67d729, Add registration created_by_manager column
INFO  [alembic.runtime.migration] Running upgrade 33c3ab67d729 -> b60f5c45acf7, Add require_captcha to regforms
INFO  [alembic.runtime.migration] Running upgrade b60f5c45acf7 -> b45847c0e62f, Add event series title pattern
INFO  [alembic.runtime.migration] Running upgrade b45847c0e62f -> 7551bd141960, Fix purged accommodation fields
INFO  [alembic.runtime.migration] Running upgrade 7551bd141960 -> 5d05eda06776, Add undone state to editing revisions
INFO  [alembic.runtime.migration] Running upgrade 5d05eda06776 -> 9b3fc740b722, Fix user name_format values
INFO  [alembic.runtime.migration] Running upgrade 9b3fc740b722 -> aba7935f9226, Add is_event_not_happening to EventLabel
INFO  [alembic.runtime.migration] Running upgrade aba7935f9226 -> e47fc6634291, Add menu entry ACLs
INFO  [alembic.runtime.migration] Running upgrade e47fc6634291 -> a59688f9ba40, Add require_privacy_policy_agreement to regforms
INFO  [alembic.runtime.migration] Running upgrade a59688f9ba40 -> 0af8f63aa603, Refactor editing revisions
INFO  [alembic.runtime.migration] Running upgrade 0af8f63aa603 -> cb46beecbb93, Add Editable.is_deleted
INFO  [alembic.runtime.migration] Running upgrade cb46beecbb93 -> cd7038a37646, Add internal_note column to room bookings
INFO  [alembic.runtime.migration] Running upgrade cd7038a37646 -> 0c44046dc1be, Add generate accompanying person tickets
INFO  [alembic.runtime.migration] Running upgrade 0c44046dc1be -> d8562ad31e90, Add created_dt on User model
INFO  [alembic.runtime.migration] Running upgrade d8562ad31e90 -> 13d8ba15a83b, Add recurrence_weekdays to reservations
INFO  [alembic.runtime.migration] Running upgrade 13d8ba15a83b -> 155cfc134f0c, Setting for future events threshold
INFO  [alembic.runtime.migration] Running upgrade 155cfc134f0c -> 31b699664893, Add modified date to editing revisions
INFO  [alembic.runtime.migration] Running upgrade 31b699664893 -> 0acf26d68434, Add subcontrib_speakers_can_submit to events table
INFO  [alembic.runtime.migration] Running upgrade 0acf26d68434 -> 252d61f890a0, Add receipt data structures
INFO  [alembic.runtime.migration] Running upgrade 252d61f890a0 -> fb0ca1440185, Add a redirect URI for the new Check-in app
INFO  [alembic.runtime.migration] Running upgrade fb0ca1440185 -> e2b69fe5155d, Add data_export_requests table
INFO  [alembic.runtime.migration] Running upgrade e2b69fe5155d -> 8e08236a529f, Add skip_access_check to invitations
INFO  [alembic.runtime.migration] Running upgrade 8e08236a529f -> 5bb555dd91eb, Update merged users in abstract email logs
Traceback (most recent call last):
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.DuplicateTable: relation "merge_map" already exists

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/indico/.venv/bin/indico", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/indico/cli/util.py", line 108, in invoke
    return self._impl.invoke(ctx)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/flask/cli.py", line 386, in decorator
    return ctx.invoke(f, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/indico/cli/database.py", line 139, in _call_with_plugins
    func(*args, **kwargs)
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/flask/cli.py", line 386, in decorator
    return ctx.invoke(f, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/flask_migrate/cli.py", line 154, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/opt/indico/.venv/lib/python3.12/site-packages/flask_migrate/__init__.py", line 111, in wrapped
    f(*args, **kwargs)
  File "/opt/indico/.venv/lib/python3.12/site-packages/flask_migrate/__init__.py", line 200, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/opt/indico/.venv/lib/python3.12/site-packages/alembic/command.py", line 403, in upgrade
    script.run_env()
  File "/opt/indico/.venv/lib/python3.12/site-packages/alembic/script/base.py", line 583, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/opt/indico/.venv/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
    module = load_module_py(module_id, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/opt/indico/.venv/lib/python3.12/site-packages/indico/migrations/env.py", line 91, in <module>
    run_migrations_online()
  File "/opt/indico/.venv/lib/python3.12/site-packages/indico/migrations/env.py", line 83, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/opt/indico/.venv/lib/python3.12/site-packages/alembic/runtime/environment.py", line 948, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/opt/indico/.venv/lib/python3.12/site-packages/alembic/runtime/migration.py", line 627, in run_migrations
    step.migration_fn(**kw)
  File "/opt/indico/.venv/lib/python3.12/site-packages/indico/migrations/versions/20240206_1223_5bb555dd91eb_update_merged_users_in_abstract_email_logs.py", line 19, in upgrade
    op.execute('''
  File "<string>", line 8, in execute
  File "<string>", line 3, in execute
  File "/opt/indico/.venv/lib/python3.12/site-packages/alembic/operations/ops.py", line 2537, in execute
    return operations.invoke(op)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/alembic/operations/base.py", line 445, in invoke
    return fn(self, operation)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/alembic/operations/toimpl.py", line 224, in execute_sql
    operations.migration_context.impl.execute(
  File "/opt/indico/.venv/lib/python3.12/site-packages/alembic/ddl/impl.py", line 214, in execute
    self._exec(sql, execution_options)
  File "/opt/indico/.venv/lib/python3.12/site-packages/alembic/ddl/impl.py", line 207, in _exec
    return conn.execute(construct, multiparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1385, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
    self._handle_dbapi_exception(
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
    util.raise_(
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.DuplicateTable) relation "merge_map" already exists

[SQL:
        CREATE TEMPORARY TABLE merge_map AS
        WITH RECURSIVE merge_map (user_id, current_user_id) AS (
            SELECT id, id FROM users.users WHERE merged_into_id IS NULL
            UNION ALL
            SELECT u.id, mm.current_user_id FROM users.users u, merge_map mm WHERE u.merged_into_id = mm.user_id
        ) SELECT * FROM merge_map WHERE user_id != current_user_id;

        CREATE INDEX ix_merge_map_current_user_id ON merge_map USING btree (current_user_id);

        UPDATE event_abstracts.email_logs eml
        SET user_id = mm.current_user_id
        FROM merge_map mm
        WHERE mm.user_id = eml.user_id AND mm.current_user_id != eml.user_id;
    ]
(Background on this error at: https://sqlalche.me/e/14/f405)

[ThiefMaster]

Use Indico v3.3.3 (released earlier this week) which has a fix for this.

Alternatively, run indico db upgrade 735dc4e8d2f3 first and then indico db upgrade.

[Ala321]

Thank you, This solves the issue indico db upgrade 735dc4e8d2f3. What does it mean 735dc4e8d2f3?

[ThiefMaster]

it's the db revision to upgrade to. by running it in two steps there's a db commit in between