Closed gnat closed 2 years ago
Same results on both:
And latest development versions
Have you identified the offending query? I imagine it's just taking really a long time rather than hanging indefinitely? Are you saying the query is unneeded and can be skipped?
Simple 3 node cluster. Completely fresh from cockroach init
.
Added to django_cockroachdb/base.py
:
#BigAutoField='DEFAULT unique_rowid()',
BigAutoField='USING HASH DEFAULT unique_rowid()',
Initial model.
class Listing(models.Model):
title = models.CharField(max_length=150)
price = models.IntegerField()
First migration.
Operations to perform:
Apply all migrations: admin, auth, contenttypes, products, sessions
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying auth.0012_alter_user_first_name_max_length... OK
Applying products.0001_initial... OK
Applying sessions.0001_initial... OK
SQL after initial migration:
root@:26257/test_django> show create products_listing;
table_name | create_statement
-------------------+---------------------------------------------------------------------------------------------------------------------------------
products_listing | CREATE TABLE public.products_listing (
| crdb_internal_id_shard_16 INT8 NOT VISIBLE NOT NULL AS (mod(fnv32(crdb_internal.datums_to_bytes(id)), 16:::INT8)) VIRTUAL,
| id INT8 NOT NULL DEFAULT unique_rowid(),
| title VARCHAR(150) NOT NULL,
| price INT8 NOT NULL,
| CONSTRAINT products_listing_pkey PRIMARY KEY (id ASC) USING HASH WITH (bucket_count=16)
| )
(1 row)
Time: 35ms total (execution 34ms / network 0ms)
Adding some data:
INSERT INTO products_listing (title, price) VALUES ('Test Product', '55');
INSERT INTO products_listing (title, price) VALUES ('Test Product B', '60');
root@:26257/test_django> select * from products_listing;
id | title | price
---------------------+----------------+--------
790911495589101569 | Test Product | 55
790911510319955969 | Test Product B | 60
(2 rows)
Time: 4ms total (execution 4ms / network 0ms)
Updated model.
# Create your models here.
class Listing(models.Model):
title = models.CharField(max_length=150)
price = models.IntegerField()
description = models.TextField(default='')
Second migration.
Operations to perform:
Apply all migrations: admin, auth, contenttypes, products, sessions
Running migrations:
Applying products.0002_listing_description...
:warning: Hangs indefinitely here! :warning:
The query that indefinitely is in REVERTING, RETRYING:
ALTER TABLE test_django.public.products_listing ADD COLUMN description STRING NOT NULL DEFAULT ''
Operations to perform:
Apply all migrations: admin, auth, contenttypes, products, sessions
Running migrations:
Applying products.0002_listing_description...Traceback (most recent call last):
File "/home/gnat/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 87, in _execute
return self.cursor.execute(sql)
File "/home/gnat/.local/lib/python3.10/site-packages/psycopg/cursor.py", line 560, in execute
raise ex.with_traceback(None)
psycopg.OperationalError: consuming input failed: EOF detected
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/gnat/Desktop/test_django/manage.py", line 22, in <module>
main()
File "/home/gnat/Desktop/test_django/manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/home/gnat/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
utility.execute()
File "/home/gnat/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/gnat/.local/lib/python3.10/site-packages/django/core/management/base.py", line 402, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/gnat/.local/lib/python3.10/site-packages/django/core/management/base.py", line 448, in execute
output = self.handle(*args, **options)
File "/home/gnat/.local/lib/python3.10/site-packages/django/core/management/base.py", line 96, in wrapper
res = handle_func(*args, **kwargs)
File "/home/gnat/.local/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 349, in handle
post_migrate_state = executor.migrate(
File "/home/gnat/.local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 135, in migrate
state = self._migrate_all_forwards(
File "/home/gnat/.local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
state = self.apply_migration(
File "/home/gnat/.local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
state = migration.apply(state, schema_editor)
File "/home/gnat/.local/lib/python3.10/site-packages/django/db/migrations/migration.py", line 130, in apply
operation.database_forwards(
File "/home/gnat/.local/lib/python3.10/site-packages/django/db/migrations/operations/fields.py", line 108, in database_forwards
schema_editor.add_field(
File "/home/gnat/.local/lib/python3.10/site-packages/django/db/backends/base/schema.py", line 680, in add_field
self.execute(sql, params)
File "/home/gnat/.local/lib/python3.10/site-packages/django/db/backends/postgresql/schema.py", line 53, in execute
return super().execute(sql, None)
File "/home/gnat/.local/lib/python3.10/site-packages/django/db/backends/base/schema.py", line 199, in execute
cursor.execute(sql, params)
File "/home/gnat/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 103, in execute
return super().execute(sql, params)
File "/home/gnat/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
File "/home/gnat/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/home/gnat/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
with self.db.wrap_database_errors:
File "/home/gnat/.local/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/home/gnat/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 87, in _execute
return self.cursor.execute(sql)
File "/home/gnat/.local/lib/python3.10/site-packages/psycopg/cursor.py", line 560, in execute
raise ex.with_traceback(None)
django.db.utils.OperationalError: consuming input failed: EOF detected
null=True
# Create your models here.
class Listing(models.Model):
title = models.CharField(max_length=150)
price = models.IntegerField()
description = models.TextField(null=True)
Operations to perform:
Apply all migrations: admin, auth, contenttypes, products, sessions
Running migrations:
Applying products.0002_listing_description... OK
default=
again.# Create your models here.
class Listing(models.Model):
title = models.CharField(max_length=150)
price = models.IntegerField()
description = models.TextField(null=True)
description2 = models.TextField(default='a')
Operations to perform:
Apply all migrations: admin, auth, contenttypes, products, sessions
Running migrations:
Applying products.0003_listing_description2...
:warning: :cry: Hangs indefinitely here... again! :warning:
Query, again indefinitely REVERTING, RETRYING:
ALTER TABLE test_django2.public.products_listing ADD COLUMN description2 STRING NOT NULL DEFAULT 'a'
Can you reproduce the problem outside of Django, i.e. by executing the SQL statements that Django generates using cockroachdb sql
? It's not clear to me that this isn't a problem with CockroachDB itself.
Oh wow, you're correct @timgraham It's a Cockroach DB issue! This is a bad bug. :open_mouth:
Thanks for this report! Could you share the way of reproducing the bug using only cockroach sql
? And file an issue at https://github.com/cockroachdb/cockroach (or post here, and I can file it for you).
Closing, this bug has been triaged.
Great, thanks. Closing as "not completed" since it's not applicable to this repo.
What works.
First time table creation, and most other migrations work fine.
What does not work.
Migrations hang when
default=
is set on a new field, whileUSING HASH
in the table.Issue Details
This issue is mentioned here: https://github.com/django/django/blob/7e3c9c3205e9646261cea5e5a7af7ec0e806690a/docs/ref/migration-operations.txt#L163
I believe, when the migration has to perform a full rewrite of the table, Django cannot understand
crdb_internal_id_shard_16
(or how it fits into the model) whenUSING HASH
is specified.Possible resolution
Is there a way to ignore or hide fields which CockroachDB generates automatically when using hash sharded index?
If we can get Django to simply ignore anything of
crdb_internal_*
we could be okay.Related: 78049 https://github.com/cockroachdb/django-cockroachdb/issues/264