The migrations Job fails. The pod goes to the Error state and gets started again until it runs out of retries. This is reproducible by doing "kubectl apply -f migrations.yml". The logs for the pods contain:
psycopg2.errors.UniqueViolation: could not create unique index "uq_tenantconfig_tenant_id"
DETAIL: Key (tenant_id)=(*) is duplicated.
Full logs:
2023-06-05 18:27:49,534 INFO: returning a logger set to level: INFO for module: tapisservice.tenants [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43]
2023-06-05 18:27:54,087 INFO: returning a logger set to level: INFO for module: tapisservice.auth [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43]
2023-06-05 18:27:54,087 INFO: returning a logger set to level: INFO for module: service [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43]
Got exception trying to unpickle and create spec for spec_path: "files"; exception: [Errno 2] No such file or directory: '/usr/local/lib/python3.7/site-packages/tapipy/specs/tapis-project-tapis-files-dev-api-src-main-resources-openapi.pickle'
Falling back to prod spec for "files" resource
2023-06-05 18:28:00,864 INFO: returning a logger set to level: INFO for module: tapisservice [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43]
2023-06-05 18:28:06,046 INFO: returning a logger set to level: INFO for module: tapisservice.tapisflask.utils [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43]
2023-06-05 18:28:06,204 INFO: returning a logger set to level: INFO for module: tapisservice.tapisflask.resources [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43]
2023-06-05 18:28:06,342 INFO: returning a logger set to level: INFO for module: service.models [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43]
2023-06-05 18:28:06,345 INFO: returning a logger set to level: INFO for module: service.models [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43]
2023-06-05 18:28:06,475 INFO: returning a logger set to level: INFO for module: service.ldap [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43]
2023-06-05 18:28:06,475 INFO: returning a logger set to level: INFO for module: service.auth [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43]
2023-06-05 18:28:06,513 INFO: returning a logger set to level: INFO for module: service.oauth2ext [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43]
2023-06-05 18:28:06,515 INFO: returning a logger set to level: INFO for module: service.mfa [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43]
2023-06-05 18:28:06,515 INFO: returning a logger set to level: INFO for module: service.controllers [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43]
2023-06-05 18:28:06,515 INFO: returning a logger set to level: INFO for module: service.api [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43]
2023-06-05 18:28:06,529 INFO: Authenticator ready [in /home/tapis/service/api.py:99]
/usr/local/lib/python3.7/site-packages/flask_sqlalchemy/init.py:835: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future. Set it to True or False to suppress this warning.
'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '
/usr/local/lib/python3.7/site-packages/flask_sqlalchemy/init.py:835: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future. Set it to True or False to suppress this warning.
'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [ca442ccae0ddpy] returning a logger set to level: INFO for module: ca442ccae0ddpy
INFO [migrations134py] returning a logger set to level: INFO for module: migrations134py
INFO [alembic.runtime.migration] Running upgrade acf5fe06e3c5 -> ca442ccae0dd, empty message
INFO [ca442ccae0dd__py] Starting ca442ccae0dd upgrade
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.UniqueViolation: could not create unique index "uq_tenantconfig_tenant_id"
DETAIL: Key (tenant_id)=(*) is duplicated.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/flask", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 967, in main
cli.main(args=sys.argv[1:], prog_name="python -m flask" if as_module else None)
File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 586, in main
return super(FlaskGroup, self).main(args, kwargs)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, ctx.params)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(args, kwargs)
File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, *kwargs)
File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 426, in decorator
return __ctx.invoke(f, args, kwargs)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, *kwargs)
File "/home/tapis/.local/lib/python3.7/site-packages/flask_migrate/cli.py", line 150, in upgrade
_upgrade(directory, revision, sql, tag, x_arg)
File "/home/tapis/.local/lib/python3.7/site-packages/flask_migrate/init.py", line 111, in wrapped
f(args, kwargs)
File "/home/tapis/.local/lib/python3.7/site-packages/flask_migrate/init.py", line 200, in upgrade
command.upgrade(config, revision, sql=sql, tag=tag)
File "/home/tapis/.local/lib/python3.7/site-packages/alembic/command.py", line 378, in upgrade
script.run_env()
File "/home/tapis/.local/lib/python3.7/site-packages/alembic/script/base.py", line 576, in run_env
util.load_python_file(self.dir, "env.py")
File "/home/tapis/.local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
module = load_module_py(module_id, path)
File "/home/tapis/.local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "migrations/env.py", line 96, in
run_migrations_online()
File "migrations/env.py", line 90, in run_migrations_online
context.run_migrations()
File "", line 8, in run_migrations
File "/home/tapis/.local/lib/python3.7/site-packages/alembic/runtime/environment.py", line 868, in run_migrations
self.get_context().run_migrations(kw)
File "/home/tapis/.local/lib/python3.7/site-packages/alembic/runtime/migration.py", line 622, in run_migrations
step.migrationfn(**kw)
File "/home/tapis/migrations/versions/ca442ccae0dd.py", line 61, in upgrade
conn.execute("alter table public.tenantconfig add constraint uq_tenantconfig_tenant_id unique (tenant_id)")
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1370, in execute
future=False,
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1676, in _exec_driver_sql
distilled_parameters,
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1944, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2125, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=excinfo[2], from=e
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
raise exception
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) could not create unique index "uq_tenantconfig_tenant_id"
DETAIL: Key (tenant_id)=(*) is duplicated.
[SQL: alter table public.tenantconfig add constraint uq_tenantconfig_tenant_id unique (tenant_id)]
(Background on this error at: https://sqlalche.me/e/14/gkpj)
The migrations Job fails. The pod goes to the Error state and gets started again until it runs out of retries. This is reproducible by doing "kubectl apply -f migrations.yml". The logs for the pods contain: psycopg2.errors.UniqueViolation: could not create unique index "uq_tenantconfig_tenant_id" DETAIL: Key (tenant_id)=(*) is duplicated.
Full logs: 2023-06-05 18:27:49,534 INFO: returning a logger set to level: INFO for module: tapisservice.tenants [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43] 2023-06-05 18:27:54,087 INFO: returning a logger set to level: INFO for module: tapisservice.auth [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43] 2023-06-05 18:27:54,087 INFO: returning a logger set to level: INFO for module: service [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43] Got exception trying to unpickle and create spec for spec_path: "files"; exception: [Errno 2] No such file or directory: '/usr/local/lib/python3.7/site-packages/tapipy/specs/tapis-project-tapis-files-dev-api-src-main-resources-openapi.pickle' Falling back to prod spec for "files" resource 2023-06-05 18:28:00,864 INFO: returning a logger set to level: INFO for module: tapisservice [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43] 2023-06-05 18:28:06,046 INFO: returning a logger set to level: INFO for module: tapisservice.tapisflask.utils [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43] 2023-06-05 18:28:06,204 INFO: returning a logger set to level: INFO for module: tapisservice.tapisflask.resources [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43] 2023-06-05 18:28:06,342 INFO: returning a logger set to level: INFO for module: service.models [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43] 2023-06-05 18:28:06,345 INFO: returning a logger set to level: INFO for module: service.models [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43] 2023-06-05 18:28:06,475 INFO: returning a logger set to level: INFO for module: service.ldap [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43] 2023-06-05 18:28:06,475 INFO: returning a logger set to level: INFO for module: service.auth [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43] 2023-06-05 18:28:06,513 INFO: returning a logger set to level: INFO for module: service.oauth2ext [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43] 2023-06-05 18:28:06,515 INFO: returning a logger set to level: INFO for module: service.mfa [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43] 2023-06-05 18:28:06,515 INFO: returning a logger set to level: INFO for module: service.controllers [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43] 2023-06-05 18:28:06,515 INFO: returning a logger set to level: INFO for module: service.api [in /usr/local/lib/python3.7/site-packages/tapisservice/logs.py:43] 2023-06-05 18:28:06,529 INFO: Authenticator ready [in /home/tapis/service/api.py:99] /usr/local/lib/python3.7/site-packages/flask_sqlalchemy/init.py:835: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future. Set it to True or False to suppress this warning. 'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and ' /usr/local/lib/python3.7/site-packages/flask_sqlalchemy/init.py:835: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future. Set it to True or False to suppress this warning. 'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and ' INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. INFO [ca442ccae0ddpy] returning a logger set to level: INFO for module: ca442ccae0ddpy INFO [migrations134py] returning a logger set to level: INFO for module: migrations134py INFO [alembic.runtime.migration] Running upgrade acf5fe06e3c5 -> ca442ccae0dd, empty message INFO [ca442ccae0dd__py] Starting ca442ccae0dd upgrade Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context cursor, statement, parameters, context File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute cursor.execute(statement, parameters) psycopg2.errors.UniqueViolation: could not create unique index "uq_tenantconfig_tenant_id" DETAIL: Key (tenant_id)=(*) is duplicated.
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/usr/local/bin/flask", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 967, in main
cli.main(args=sys.argv[1:], prog_name="python -m flask" if as_module else None)
File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 586, in main
return super(FlaskGroup, self).main(args, kwargs)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, ctx.params)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(args, kwargs)
File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, *kwargs)
File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 426, in decorator
return __ctx.invoke(f, args, kwargs)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, *kwargs)
File "/home/tapis/.local/lib/python3.7/site-packages/flask_migrate/cli.py", line 150, in upgrade
_upgrade(directory, revision, sql, tag, x_arg)
File "/home/tapis/.local/lib/python3.7/site-packages/flask_migrate/init.py", line 111, in wrapped
f(args, kwargs)
File "/home/tapis/.local/lib/python3.7/site-packages/flask_migrate/init.py", line 200, in upgrade
command.upgrade(config, revision, sql=sql, tag=tag)
File "/home/tapis/.local/lib/python3.7/site-packages/alembic/command.py", line 378, in upgrade
script.run_env()
File "/home/tapis/.local/lib/python3.7/site-packages/alembic/script/base.py", line 576, in run_env
util.load_python_file(self.dir, "env.py")
File "/home/tapis/.local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
module = load_module_py(module_id, path)
File "/home/tapis/.local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "migrations/env.py", line 96, in
run_migrations_online()
File "migrations/env.py", line 90, in run_migrations_online
context.run_migrations()
File "", line 8, in run_migrations
File "/home/tapis/.local/lib/python3.7/site-packages/alembic/runtime/environment.py", line 868, in run_migrations
self.get_context().run_migrations( kw)
File "/home/tapis/.local/lib/python3.7/site-packages/alembic/runtime/migration.py", line 622, in run_migrations
step.migrationfn(**kw)
File "/home/tapis/migrations/versions/ca442ccae0dd.py", line 61, in upgrade
conn.execute("alter table public.tenantconfig add constraint uq_tenantconfig_tenant_id unique (tenant_id)")
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1370, in execute
future=False,
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1676, in _exec_driver_sql
distilled_parameters,
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1944, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2125, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=excinfo[2], from=e
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
raise exception
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) could not create unique index "uq_tenantconfig_tenant_id"
DETAIL: Key (tenant_id)=(*) is duplicated.
[SQL: alter table public.tenantconfig add constraint uq_tenantconfig_tenant_id unique (tenant_id)] (Background on this error at: https://sqlalche.me/e/14/gkpj)