Open Erazx opened 3 years ago
What's the version?
I'm seeing the same issue and i'm using version aerich==0.5.3.
Output is this:
Traceback (most recent call last):
File "/srv/SimpleService/venv/bin/aerich", line 8, in <module>
sys.exit(main())
File "/srv/SimpleService/venv/lib/python3.9/site-packages/aerich/cli.py", line 298, in main
cli()
File "/srv/SimpleService/venv/lib/python3.9/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/srv/SimpleService/venv/lib/python3.9/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/srv/SimpleService/venv/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/srv/SimpleService/venv/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/srv/SimpleService/venv/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/srv/SimpleService/venv/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/srv/SimpleService/venv/lib/python3.9/site-packages/aerich/cli.py", line 41, in wrapper
loop.run_until_complete(f(*args, **kwargs))
File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/srv/SimpleService/venv/lib/python3.9/site-packages/aerich/cli.py", line 95, in migrate
ret = await Migrate.migrate(name)
File "/srv/SimpleService/venv/lib/python3.9/site-packages/aerich/migrate.py", line 130, in migrate
cls.diff_models(cls._last_version_content, new_version_content)
File "/srv/SimpleService/venv/lib/python3.9/site-packages/aerich/migrate.py", line 208, in diff_models
table = change[0][1].get("through")
AttributeError: 'str' object has no attribute 'get'
I join, faced with the same problem, version 0.5.3 I have to constantly do init-db
@Aquinary I'm getting the same issue and init-db
isn't resolving it for me instead of resolving the error it's giving me another error as AttributeError: 'NoneType' object has no attribute 'pop'
on doing aerich migrate
@Aquinary I'm getting the same issue and
init-db
isn't resolving it for me instead of resolving the error it's giving me another error asAttributeError: 'NoneType' object has no attribute 'pop'
on doingaerich migrate
Yes. Need to delete all the tables that are affected by the migration (including those that are linked), then init-db is worker normally. This is acceptable in dev, but it can't be a solution in production, so I switched to sqlalchemy.
I'm also about to switch to sqlalchemy
On Tue, Jun 8, 2021 at 8:19 PM Aquinary @.***> wrote:
@Aquinary https://github.com/Aquinary I'm getting the same issue and init-db isn't resolving it for me instead of resolving the error it's giving me another error as AttributeError: 'NoneType' object has no attribute 'pop' on doing aerich migrate
Yes. Need to delete all the tables that are affected by the migration (including those that are linked), then init-db is worker normally. This is acceptable in dev, but it can't be a solution in production, so I switched to sqlalchemy.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tortoise/aerich/issues/150#issuecomment-856834709, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANXV22XVWPB3N2P6IKK6TMTTRYUXZANCNFSM43Q7NJKA .
Are you feeling comfortable with SQLAlchemy now?
On Tue, Jun 8, 2021 at 8:33 PM FIRDOUS BHAT @.***> wrote:
I'm also about to switch to sqlalchemy
On Tue, Jun 8, 2021 at 8:19 PM Aquinary @.***> wrote:
@Aquinary https://github.com/Aquinary I'm getting the same issue and init-db isn't resolving it for me instead of resolving the error it's giving me another error as AttributeError: 'NoneType' object has no attribute 'pop' on doing aerich migrate
Yes. Need to delete all the tables that are affected by the migration (including those that are linked), then init-db is worker normally. This is acceptable in dev, but it can't be a solution in production, so I switched to sqlalchemy.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tortoise/aerich/issues/150#issuecomment-856834709, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANXV22XVWPB3N2P6IKK6TMTTRYUXZANCNFSM43Q7NJKA .
Are you feeling comfortable with SQLAlchemy now? … On Tue, Jun 8, 2021 at 8:33 PM FIRDOUS BHAT @.> wrote: I'm also about to switch to sqlalchemy On Tue, Jun 8, 2021 at 8:19 PM Aquinary @.> wrote: > @Aquinary https://github.com/Aquinary I'm getting the same issue and > init-db isn't resolving it for me instead of resolving the error it's > giving me another error as AttributeError: 'NoneType' object has no > attribute 'pop' on doing aerich migrate > > Yes. Need to delete all the tables that are affected by the migration > (including those that are linked), then init-db is worker normally. > This is acceptable in dev, but it can't be a solution in production, so I > switched to sqlalchemy. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#150 (comment)>, > or unsubscribe > https://github.com/notifications/unsubscribe-auth/ANXV22XVWPB3N2P6IKK6TMTTRYUXZANCNFSM43Q7NJKA > . >
It is a little more complex than TORM in the initial setup and is slightly different from TORM in terms of use, but if you look into docs, you can use it quite well. It has its own nuances, which you have to get used to. Something like "more flexibility, but less convenience" + sqlalchemy it's a very good technology to have on resume Regarding flexibility, for example, in TORM, this code will automatically create the FK fields user_from_id, user_to_id. And you can access the associated fields by userfrom _ username.
class Friend(models.Model):
id = fields.IntField(pk=True)
friend_from = fields.ForeignKeyField('models.User', related_name='friend_from', description='Отправитель заявки')
friend_to = fields.ForeignKeyField('models.User', related_name='friend_to', description='Получатель заявки')
status = fields.BooleanField(default=False, description='Статус подтверждения')
In SQLAlchemy, you have to write like this and define the necessary related fields in advance
class Friend(Base):
__tablename__ = 'friends'
user_from_id = Column(Integer, ForeignKey('users.id'), primary_key=True)
user_from = relationship('User', foreign_keys=[user_from_id])
user_to_id = Column(Integer, ForeignKey('users.id'), primary_key=True)
user_to = relationship('User', foreign_keys=[user_to_id])
status = Column(Boolean, default=False)
There is no syntax like user_from__username in SQLAlchemy. Instead, you should filter the request yourself:
request = FriendRequest(user_from__username=username, user_to__username=request_username) # TORM
request = FriendRequest(user_from=db.query(User).filter_by(username=username).one(),
user_to=db.query(User).filter_by(username=request_username).one()) # SQLAlchemy
But if you get used to it, then the work goes like clockwork and there are no more problems.
Okay, Thank you for explaining it to me.
On Tue, Jun 8, 2021 at 9:24 PM Aquinary @.***> wrote:
Are you feeling comfortable with SQLAlchemy now? … <#m1531964176004411680> On Tue, Jun 8, 2021 at 8:33 PM FIRDOUS BHAT @.> wrote: I'm also about to switch to sqlalchemy On Tue, Jun 8, 2021 at 8:19 PM Aquinary @.> wrote: > @Aquinary https://github.com/Aquinary https://github.com/Aquinary I'm getting the same issue and > init-db isn't resolving it for me instead of resolving the error it's > giving me another error as AttributeError: 'NoneType' object has no > attribute 'pop' on doing aerich migrate > > Yes. Need to delete all the tables that are affected by the migration > (including those that are linked), then init-db is worker normally. > This is acceptable in dev, but it can't be a solution in production, so I > switched to sqlalchemy. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#150 (comment) https://github.com/tortoise/aerich/issues/150#issuecomment-856834709>,
or unsubscribe > https://github.com/notifications/unsubscribe-auth/ANXV22XVWPB3N2P6IKK6TMTTRYUXZANCNFSM43Q7NJKA . >
It is a little more complex than TORM in the initial setup and is slightly different from TORM in terms of use, but if you look into docs, you can use it quite well. It has its own nuances, which you have to get used to. Something like "more flexibility, but less convenience" + sqlalchemy it's a very good technology to have on resume Regarding flexibility, for example, in TORM, this code will automatically create the FK fields user_from_id, user_to_id. And you can access the associated fields by userfrom _ username. class Friend(models.Model): id = fields.IntField(pk=True) friend_from = fields.ForeignKeyField('models.User', related_name='friend_from', description='Отправитель заявки') friend_to = fields.ForeignKeyField('models.User', related_name='friend_to', description='Получатель заявки') status = fields.BooleanField(default=False, description='Статус подтверждения')
In SQLAlchemy, you have to write like this and define the necessary related fields in advance
class Friend(Base): tablename = 'friends'
user_from_id = Column(Integer, ForeignKey('users.id'), primary_key=True)
user_from = relationship('User', foreign_keys=[user_from_id])
user_to_id = Column(Integer, ForeignKey('users.id'), primary_key=True)
user_to = relationship('User', foreign_keys=[user_to_id])
status = Column(Boolean, default=False)
There is no syntax like user_from__username in SQLAlchemy. Instead, you should filter the request yourself:
request = FriendRequest(user_from__username=username, user_to__username=request_username) # TORM request = FriendRequest(user_from=db.query(User).filter_by(username=username).one(), user_to=db.query(User).filter_by(username=request_username).one()) # SQLAlchemy
But if you get used to it, then the work goes like clockwork and there are no more problems.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tortoise/aerich/issues/150#issuecomment-856891460, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANXV22R43WFQZ65JZEX6J23TRY4MFANCNFSM43Q7NJKA .
I'm having the same issue with aerich 0.5.5
.
I'm however adding a JSON field. It seems it's enough if there is a m2m field.
I took a bit of a closer look at this and it seems that there's a whole type of action missing in the migration methods. When a key is change
(i.e not add
or remove
) the script will always break since that case isn't treated at all. If you skip all "changes" from the diff, it doesn't crash, but also misses the additions or removes from m2m related entries of the model.
I didn't have time to test more in depth, but it seemed to me the issue isn't with the differ but with the serialization of the models since the differ isn't able to discriminate the difference between a change in some m2m field and an addition of a new m2m field.
But anyway, bumping this thread, it is a big breaking issue.
It's still not fixed, ffs...
Join to issue.
aerich==0.5.8 tortoise-orm==0.17.7
Ok. I debugged it.
Error in usage of method from dictdiffer https://github.com/inveniosoftware/dictdiffer/blob/d2f84b7dbe5e2ea871c25f7cb013d36e3be221e8/dictdiffer/__init__.py#L148
When it comparing lists of dicts, it's ignore that first list can contains dict from second but at another order. So, it show diff between dict of existing m2m and new m2m.
My opinion it's partially mistake of dictdiffer. It should find levenshtein distance for sequences. But it is not solution to fix bug.
EXAMPLE: 1) My m2m on migrate command. You can see, that i have created m2m storages_address_auth_user_group and didn't change m2m auth_user_auth_user_group
2) Got diff
3) Comparing first intersection. it will compare first (index=0) items of lists.
So, we try to find difference between different m2m's
Related code as below:
class Sys_Role(MyAbstractBaseModel): """角色表""" role_code = CharField(max_length=128, description="角色代码", null=False, unique=True) role_name = CharField(max_length=128, description="角色名称", null=False, unique=True) status = SmallIntField(description="角色状态", null=False, default=0) users: ManyToManyRelation[Sys_User] apis: ManyToManyRelation[Sys_Api] ## menus: ManyToManyRelation["Sys_Menu"] class Meta: table = "sys_role" table_description = "角色表" def __str__(self): return f'{self.name}({self.code})' class Sys_Menu(MyAbstractBaseModel): """菜单组件表""" path = CharField(max_length=128, description="路由路径", null=False, unique=True) name = CharField(max_length=128, description="路由名称", null=False, unique=True) parent_id = BigIntField(description="父级菜单ID", null=False, default=0) full_path = CharField(max_length=256, description="路由全路径", null=False) sort = SmallIntField(description="排序", null=False, default=0) menu_type = SmallIntField(description="菜单类型", null=False, default=0) hidden = BooleanField(description="是否隐藏", null=False, default=False) permission = CharField(max_length=128, description="路由权限", null=False, default="", unique=True) icon = CharField(max_length=128, description="菜单图标", null=False, default="") title = CharField(max_length=128, description="菜单标题", null=False) ## roles: ManyToManyRelation[Sys_Role] = ManyToManyField('system.Sys_Role', related_name="menus", through="sys_role_menu") class Meta: table = "sys_menu" table_description = "菜单组件表" def __str__(self): return f'{self.full_path}[{self.name}]'
After I added new ManyToManyField (sys_role_menu, uncomment those lines),and run
aerich migrate
,it throws an AttributeError Exception:Traceback (most recent call last): File "/data/py-nwci/venv/bin/aerich", line 8, in <module> sys.exit(main()) File "/data/py-nwci/venv/lib/python3.7/site-packages/aerich/cli.py", line 298, in main cli() File "/data/py-nwci/venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/data/py-nwci/venv/lib/python3.7/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/data/py-nwci/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/data/py-nwci/venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/data/py-nwci/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/data/py-nwci/venv/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func return f(get_current_context(), *args, **kwargs) File "/data/py-nwci/venv/lib/python3.7/site-packages/aerich/cli.py", line 41, in wrapper loop.run_until_complete(f(*args, **kwargs)) File "/usr/local/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete return future.result() File "/data/py-nwci/venv/lib/python3.7/site-packages/aerich/cli.py", line 95, in migrate ret = await Migrate.migrate(name) File "/data/py-nwci/venv/lib/python3.7/site-packages/aerich/migrate.py", line 130, in migrate cls.diff_models(cls._last_version_content, new_version_content) File "/data/py-nwci/venv/lib/python3.7/site-packages/aerich/migrate.py", line 208, in diff_models table = change[0][1].get("through") AttributeError: 'str' object has no attribute 'get'
Just came across this now....
Had to hack my way around it, by manually changing
The following lines in aerich/migrate.py @ diff_models
, to:
Line 232:
Original: if change[0][0] == "db_constraint":
New: if isinstance(change[0], bool) or change[0][0] == "db_constraint":
Line 235:
Original: table = change[0][1].get("through")
New:
if isinstance(change[0][1], str):
for new_m2m_field in new_m2m_fields:
if new_m2m_field['name'] == change[0][1]:
table = new_m2m_field.get('through')
break
else:
table = change[0][1].get("through")
This then allows you to actually migrate M2M fields...
We shouldn't really have to hack this together, though, for it to work.
Another small issue I've found, is if you're using the through
kwarg for a M2M, Aerich correctly uses the right table name when doing initial migrations (the one specified in through
), but then seems to ignore the table specified in through
later on when you try create new migrations.
@long2ice - just checking you've seen this issue?
Still not fixed. aerich==0.6.2.
If someone looks for more production ready workaround here it is:
# overload in Tortoise model, note: aerich may blow up at model which has M2M field declared
# OR at destination model(!!)
@classmethod
def describe(cls, serializable: bool = True) -> dict:
result = super().describe(serializable)
m2m_order = ('location', 'slaves', 'devicegroups') # << here put your M2M fields names
assert set(m2m_order) == set(cls._meta.m2m_fields)
result['m2m_fields'] = [
cls._meta.fields_map[name].describe(serializable)
for name in m2m_order
]
return result
Workaround explanation
As @dstlny said, the issue is in m2m fields compare. Depending on many things the order of M2M fields may change between migrations (but not have to, I hit this issue after adding third M2M relation to the same model). So to make it working without hacking aerich code or migrations data in aerich database table, I tried to make sure that order of M2M fields will be always the same and newer fields always appears after existing ones.
After some time of debugging I found how to achieve this. Aerich is taking model state using describe()
method from Tortoise model, so to make sure that differ is always getting same order this method must return M2M fields in the same order. It can be done manually by hardcoding the order. I've add an assert line to make sure that nothing wrong happens when someone adds new relation to my model (by adding M2M field in the model or relation in another model to the model).
Generic fix proposal
At the aerich level I think there should be some code which orders new_m2m_fields
list basing on the order in old_m2m_fields
(somewhere in diff_models
method in migrate.py
). Matching by field names should solve most of issues. It would be great if as a fallback aerich try to match missing old list fields by relation destination and if it's found on new list consider this as relation field name change (like Django do).
Reordered new_m2m_fields
list can be passed to differ along with current old_m2m_fields
list.
@dstlny thanks, it worked for me
What worked for me was just commenting out the many to many relation. Then first migrate to create the new table and remove the old many_to_many. And then add back the many_to_many for the new table (indeed alembic for sqlalcehemy is a bit more robust here). Still I'm not ditching TORM just yet for this single flaw ;)
I'm having this issue without even making a change to a M2M field on the model, just adding a new CharField. I was able to resolve quickly by patching aerich/migrate.py:257 locally with
get_op = getattr(change[0][1], "get", None) if callable(get_op): table = change[0][1].get("through")
It seems like the case where action = 'change' isn't covered, and in my case just making sure the dict get is failing quietly, the rest of the migration logic still works.
0.7.1 has bug too. Using dictdiffer is a problem.
Is there any solution?
Is there any solution?
Change 244 line in migrate.py
table = change[0][1].get("through")
to
if isinstance(change[0][1], str):
for new_m2m_field in new_m2m_fields:
if new_m2m_field['name'] == change[0][1]:
table = new_m2m_field.get('through')
break
else:
table = change[0][1].get("through")
Change 244 line in migrate.py
table = change[0][1].get("through")
to
if isinstance(change[0][1], str): for new_m2m_field in new_m2m_fields: if new_m2m_field['name'] == change[0][1]: table = new_m2m_field.get('through') break else: table = change[0][1].get("through")
Hi, could you make a PR?
Change 244 line in migrate.py
table = change[0][1].get("through")
to
if isinstance(change[0][1], str): for new_m2m_field in new_m2m_fields: if new_m2m_field['name'] == change[0][1]: table = new_m2m_field.get('through') break else: table = change[0][1].get("through")
Hi, could you make a PR?
Look like permission denied((
Change 244 line in migrate.py
table = change[0][1].get("through")
to
if isinstance(change[0][1], str): for new_m2m_field in new_m2m_fields: if new_m2m_field['name'] == change[0][1]: table = new_m2m_field.get('through') break else: table = change[0][1].get("through")
Hi, could you make a PR?
Look like permission denied((
Why? just fork and make pull request
Change 244 line in migrate.py
table = change[0][1].get("through")
to
if isinstance(change[0][1], str): for new_m2m_field in new_m2m_fields: if new_m2m_field['name'] == change[0][1]: table = new_m2m_field.get('through') break else: table = change[0][1].get("through")
Hi, could you make a PR?
Look like permission denied((
Why? just fork and make pull request
Done
I got the same problem when added 'backward_key' and 'forward_key' to existing ManyToManyField. aerich==0.7.2
Fl0kse's solution has not been released yet
@long2ice Is this bug fixed, or fixable ? 3.5+ years passed
I believe this is a major flaw, makes tortoise orm and aerich unacceptable choice for any projects requires m2m relations
Any non-trivial projects probably have some m2m relations
Related code as below:
After I added new ManyToManyField (sys_role_menu, uncomment those lines),and run
aerich migrate
,it throws an AttributeError Exception: