search

tortoise / aerich

A database migrations tool for TortoiseORM, ready to production.
https://github.com/tortoise/aerich
Apache License 2.0
804 stars 90 forks source link

aerich init-db with sqlite3 freezes #302

Open massiccio opened 1 year ago

massiccio commented 1 year ago

Hello,

I have been observing the same behavior reported at here while using aerich with sqlite I have no problem when using the same models with postgres.

aerich init -s src/ -t tci.sqlite.database.TORTOISE_ORM
Success create migrate location ./migrations
Success write config to pyproject.toml
aerich init-db
Success create app migrate location migrations/models
Success generate schema for app "models"
^CException ignored in: <module 'threading' from '/usr/lib/python3.10/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1567, in _shutdown
    lock.acquire()
KeyboardInterrupt:

Test with no changes, working as expected

aerich init-db
Success create app migrate location migrations/models
Success generate schema for app "models"
^CException ignored in: <module 'threading' from '/usr/lib/python3.10/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1567, in _shutdown
    lock.acquire()
KeyboardInterrupt:

Removing the "acknowledged" column from the model

aerich migrate --name drop_column
Traceback (most recent call last):
  File "my_dir/bin/aerich", line 8, in <module>
    sys.exit(main())
  File "my_dir/lib/python3.10/site-packages/aerich/cli.py", line 258, in main
    cli()
  File "my_dir/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "my_dir/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "my_dir/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "my_dir/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "my_dir/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "my_dir/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "my_dir/lib/python3.10/site-packages/aerich/cli.py", line 31, in wrapper
    loop.run_until_complete(f(*args, **kwargs))
  File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "my_dir/lib/python3.10/site-packages/aerich/cli.py", line 86, in migrate
    ret = await command.migrate(name)
  File "my_dir/lib/python3.10/site-packages/aerich/__init__.py", line 121, in migrate
    return await Migrate.migrate(name)
  File "my_dir/lib/python3.10/site-packages/aerich/migrate.py", line 140, in migrate
    cls.diff_models(cls._last_version_content, new_version_content)
  File "my_dir/lib/python3.10/site-packages/aerich/migrate.py", line 388, in diff_models
    cls._drop_index(
  File "my_dir/lib/python3.10/site-packages/aerich/migrate.py", line 527, in _drop_index
    fields_name = cls._resolve_fk_fields_name(model, fields_name)
  File "my_dir/lib/python3.10/site-packages/aerich/migrate.py", line 512, in _resolve_fk_fields_name
    field = model._meta.fields_map[field_name]
KeyError: 'acknowledged'
pip list |grep aerich
aerich                    0.7.1
massiccio commented 1 year ago

I have tried the workaround described here, i.e., downgrading to version 0.6.3. That fixed the init-db, however the upgrade still fails.

Installing collected packages: aerich
  Attempting uninstall: aerich
    Found existing installation: aerich 0.7.0
    Uninstalling aerich-0.7.0:
      Successfully uninstalled aerich-0.7.0
Successfully installed aerich-0.6.3
aerich init -s src/ -t tci.sqlite.database.TORTOISE_ORM
Success create migrate location ./migrations
Success write config to pyproject.toml
aerich init-db
Success create app migrate location migrations/models
Success generate schema for app "models"
aerich migrate --name opsgenie_comment_column_dropped
Success migrate 5_20230518173359_opsgenie_comment_column_dropped.sql
(opsgenie_to_jira) mic@U333633:my_dir$ aerich upgrade
Traceback (most recent call last):
  File "my_dir/lib/python3.10/site-packages/tortoise/backends/sqlite/client.py", line 34, in translate_exceptions_
    return await func(self, query, *args)
  File "my_dir/lib/python3.10/site-packages/tortoise/backends/sqlite/client.py", line 155, in execute_script
    await connection.executescript(query)
  File "my_dir/lib/python3.10/site-packages/aiosqlite/core.py", line 216, in executescript
    cursor = await self._execute(self._conn.executescript, sql_script)
  File "my_dir/lib/python3.10/site-packages/aiosqlite/core.py", line 129, in _execute
    return await future
  File "my_dir/lib/python3.10/site-packages/aiosqlite/core.py", line 102, in run
    result = function()
sqlite3.OperationalError: error in index idx_alert_acknowl_8a515a after drop column: no such column: acknowledged

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "my_dir/bin/aerich", line 8, in <module>
    sys.exit(main())
  File "my_dir/lib/python3.10/site-packages/aerich/cli.py", line 257, in main
    cli()
  File "my_dir/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "my_dir/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "my_dir/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "my_dir/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "my_dir/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "my_dir/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "my_dir/lib/python3.10/site-packages/aerich/cli.py", line 31, in wrapper
    loop.run_until_complete(f(*args, **kwargs))
  File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "my_dir/lib/python3.10/site-packages/aerich/cli.py", line 97, in upgrade
    migrated = await command.upgrade()
  File "my_dir/lib/python3.10/site-packages/aerich/__init__.py", line 55, in upgrade
    await conn.execute_script(upgrade_query)
  File "my_dir/lib/python3.10/site-packages/tortoise/backends/sqlite/client.py", line 36, in translate_exceptions_
    raise OperationalError(exc)
tortoise.exceptions.OperationalError: error in index idx_alert_acknowl_8a515a after drop column: no such column: acknowledged

I have also tried not to have that column as an index, and in spite of the migration throwing an error, the upgrade succeeded

aerich migrate --name opsgenie_comment_column_dropped
Traceback (most recent call last):
  File "my_dir/bin/aerich", line 8, in <module>
    sys.exit(main())
  File "my_dir/lib/python3.10/site-packages/aerich/cli.py", line 257, in main
    cli()
  File "my_dir/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "my_dir/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "my_dir/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "my_dir/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "my_dir/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "my_dir/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "my_dir/lib/python3.10/site-packages/aerich/cli.py", line 31, in wrapper
    loop.run_until_complete(f(*args, **kwargs))
  File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "my_dir/lib/python3.10/site-packages/aerich/cli.py", line 86, in migrate
    ret = await command.migrate(name)
  File "my_dir/lib/python3.10/site-packages/aerich/__init__.py", line 126, in migrate
    return await Migrate.migrate(name)
  File "my_dir/lib/python3.10/site-packages/aerich/migrate.py", line 132, in migrate
    cls.diff_models(cls._last_version_content, new_version_content)
  File "my_dir/lib/python3.10/site-packages/aerich/migrate.py", line 184, in diff_models
    old_models.pop(_aerich, None)
AttributeError: 'NoneType' object has no attribute 'pop'
aerich upgrade
Success upgrade 5_20230518173819_None.sql
long2ice commented 1 year ago

Try remove migratetions/models and aerich table and rerun migrate

massiccio commented 1 year ago

I tried that, but to no avail.

Just in case you wish to reproduce the problem, here is a simple model.

class Alert(models.Model):
    id = fields.IntField(pk=True)
    opsgenie_id = fields.CharField(max_length=50, index=True, unique=True, null=False)
    priority = fields.SmallIntField(null=False, index=True)
    message = fields.TextField(null=False)
    team = fields.CharField(max_length=20, null=False)
    site = fields.CharField(max_length=10, null=False)
    stage = fields.CharField(max_length=3, null=False)
    created_at = fields.DatetimeField(null=True)
    updated_at = fields.DatetimeField(null=True)
    jira_id = fields.CharField(max_length=30)
    acknowledged = fields.BooleanField(default=False, index=True, null=False)  # try to remove this field
orm_config = {
        "connections": {
            "default": {
                "engine": "tortoise.backends.sqlite",
                "credentials": {"file_path": "db.sqlite3"},
            }
        },
        "apps": {
            "models": {
                "models": ["tci.sqlite.models", "aerich.models"],
                "default_connection": "default",
            }
        },
    }
yotaro-shimose commented 1 year ago

Same happened to me. It'd be great if someone can provide any sort of workaround.