search

citusdata / citus

Distributed PostgreSQL as an extension
https://www.citusdata.com
GNU Affero General Public License v3.0
10.59k stars 670 forks source link

distributed deadlock when dropping distributed table referencing a reference table #3526

Open onurctirtir opened 4 years ago

onurctirtir commented 4 years ago

To reproduce it on master: 2c51057013e102f7e5364a7242d66a65ecd524bc :

SET citus.shard_replication_factor TO 1; 
set log_min_messages to DEBUG4;
set client_min_messages to DEBUG4;
SET citus.log_remote_commands TO ON;

CREATE TABLE ref_table (a int primary key); 
SELECT create_reference_table('ref_table'); 

CREATE TABLE dist_table(a int); 
SELECT create_distributed_table('dist_table', 'a'); 

ALTER TABLE dist_table ADD CONSTRAINT fkey FOREIGN KEY(a) REFERENCES ref_table(a);

, and, finally running the below:

-- to reproduce the bug easily 
set citus.force_max_query_parallelization to on;

BEGIN; 
  SELECT COUNT(*) FROM dist_table; 
  DROP TABLE dist_table; 
ROLLBACK;

, we have the below error:

postgres=#   DROP TABLE dist_table;
DEBUG:  drop auto-cascades to type dist_table
DEBUG:  drop auto-cascades to type dist_table[]
DEBUG:  drop auto-cascades to trigger truncate_trigger_45324 on table dist_table
DEBUG:  drop auto-cascades to constraint fkey on table dist_table
DEBUG:  drop auto-cascades to trigger RI_ConstraintTrigger_a_46850 on table ref_table
DEBUG:  drop auto-cascades to trigger RI_ConstraintTrigger_a_46851 on table ref_table
DEBUG:  drop auto-cascades to trigger RI_ConstraintTrigger_c_46852 on table dist_table
DEBUG:  drop auto-cascades to trigger RI_ConstraintTrigger_c_46853 on table dist_table
DEBUG:  EventTriggerInvoke 16450
ERROR:  canceling the transaction since it was involved in a distributed deadlock
CONTEXT:  SQL statement "SELECT master_drop_all_shards(v_obj.objid, v_obj.schema_name, v_obj.object_name)"
PL/pgSQL function citus_drop_trigger() line 19 at PERFORM
onurctirtir commented 4 years ago

or, from a more verbose aspect:

onurctirtir=# BEGIN;
onurctirtir=#   SELECT COUNT(*) FROM dist_table;
DEBUG:  Router planner cannot handle multi-shard select queries
DEBUG:  generated sql query for task 1
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102009 dist_table WHERE true"
DEBUG:  generated sql query for task 2
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102010 dist_table WHERE true"
DEBUG:  generated sql query for task 3
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102011 dist_table WHERE true"
DEBUG:  generated sql query for task 4
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102012 dist_table WHERE true"
DEBUG:  generated sql query for task 5
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102013 dist_table WHERE true"
DEBUG:  generated sql query for task 6
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102014 dist_table WHERE true"
CK;DEBUG:  generated sql query for task 7
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102015 dist_table WHERE true"
DEBUG:  generated sql query for task 8
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102016 dist_table WHERE true"
DEBUG:  generated sql query for task 9
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102017 dist_table WHERE true"
DEBUG:  generated sql query for task 10
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102018 dist_table WHERE true"
DEBUG:  generated sql query for task 11
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102019 dist_table WHERE true"
DEBUG:  generated sql query for task 12
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102020 dist_table WHERE true"
DEBUG:  generated sql query for task 13
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102021 dist_table WHERE true"
DEBUG:  generated sql query for task 14
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102022 dist_table WHERE true"
DEBUG:  generated sql query for task 15
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102023 dist_table WHERE true"
DEBUG:  generated sql query for task 16
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102024 dist_table WHERE true"
DEBUG:  generated sql query for task 17
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102025 dist_table WHERE true"
DEBUG:  generated sql query for task 18
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102026 dist_table WHERE true"
DEBUG:  generated sql query for task 19
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102027 dist_table WHERE true"
DEBUG:  generated sql query for task 20
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102028 dist_table WHERE true"
DEBUG:  generated sql query for task 21
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102029 dist_table WHERE true"
DEBUG:  generated sql query for task 22
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102030 dist_table WHERE true"
DEBUG:  generated sql query for task 23
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102031 dist_table WHERE true"
DEBUG:  generated sql query for task 24
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102032 dist_table WHERE true"
DEBUG:  generated sql query for task 25
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102033 dist_table WHERE true"
DEBUG:  generated sql query for task 26
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102034 dist_table WHERE true"
DEBUG:  generated sql query for task 27
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102035 dist_table WHERE true"
DEBUG:  generated sql query for task 28
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102036 dist_table WHERE true"
DEBUG:  generated sql query for task 29
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102037 dist_table WHERE true"
DEBUG:  generated sql query for task 30
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102038 dist_table WHERE true"
DEBUG:  generated sql query for task 31
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102039 dist_table WHERE true"
DEBUG:  generated sql query for task 32
DETAIL:  query string: "SELECT count(*) AS count FROM dist_table_102040 dist_table WHERE true"
DEBUG:  assigned task 1 to node localhost:9701
DEBUG:  assigned task 2 to node localhost:9702
DEBUG:  assigned task 3 to node localhost:9701
DEBUG:  assigned task 4 to node localhost:9702
DEBUG:  assigned task 5 to node localhost:9701
DEBUG:  assigned task 6 to node localhost:9702
DEBUG:  assigned task 7 to node localhost:9701
DEBUG:  assigned task 8 to node localhost:9702
DEBUG:  assigned task 9 to node localhost:9701
DEBUG:  assigned task 10 to node localhost:9702
DEBUG:  assigned task 11 to node localhost:9701
DEBUG:  assigned task 12 to node localhost:9702
DEBUG:  assigned task 13 to node localhost:9701
DEBUG:  assigned task 14 to node localhost:9702
DEBUG:  assigned task 15 to node localhost:9701
DEBUG:  assigned task 16 to node localhost:9702
DEBUG:  assigned task 17 to node localhost:9701
DEBUG:  assigned task 18 to node localhost:9702
DEBUG:  assigned task 19 to node localhost:9701
DEBUG:  assigned task 20 to node localhost:9702
DEBUG:  assigned task 21 to node localhost:9701
DEBUG:  assigned task 22 to node localhost:9702
DEBUG:  assigned task 23 to node localhost:9701
DEBUG:  assigned task 24 to node localhost:9702
DEBUG:  assigned task 25 to node localhost:9701
DEBUG:  assigned task 26 to node localhost:9702
DEBUG:  assigned task 27 to node localhost:9701
DEBUG:  assigned task 28 to node localhost:9702
DEBUG:  assigned task 29 to node localhost:9701
DEBUG:  assigned task 30 to node localhost:9702
DEBUG:  assigned task 31 to node localhost:9701
DEBUG:  assigned task 32 to node localhost:9702
DEBUG:  opening 16 new connections to localhost:9701
DEBUG:  established connection to localhost:9701 for session 7
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 3
DEBUG:  opening 16 new connections to localhost:9702
DEBUG:  established connection to localhost:9702 for session 23
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 4
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102009 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 3
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102010 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 4
DEBUG:  established connection to localhost:9701 for session 8
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 5
DEBUG:  established connection to localhost:9701 for session 9
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 6
DEBUG:  established connection to localhost:9701 for session 10
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 7
DEBUG:  established connection to localhost:9701 for session 11
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 8
DEBUG:  established connection to localhost:9701 for session 12
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 9
DEBUG:  established connection to localhost:9701 for session 15
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 12
DEBUG:  established connection to localhost:9702 for session 29
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 25
DEBUG:  established connection to localhost:9701 for session 22
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 19
DEBUG:  established connection to localhost:9701 for session 16
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 13
DEBUG:  established connection to localhost:9701 for session 13
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 10
DEBUG:  established connection to localhost:9702 for session 33
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 29
DEBUG:  established connection to localhost:9701 for session 19
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 16
DEBUG:  established connection to localhost:9701 for session 18
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 15
DEBUG:  established connection to localhost:9702 for session 28
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 24
DEBUG:  established connection to localhost:9702 for session 25
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 21
DEBUG:  established connection to localhost:9702 for session 32
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 28
DEBUG:  established connection to localhost:9702 for session 30
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 26
DEBUG:  established connection to localhost:9701 for session 20
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 17
DEBUG:  established connection to localhost:9702 for session 26
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 22
DEBUG:  established connection to localhost:9701 for session 17
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 14
DEBUG:  established connection to localhost:9702 for session 35
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 31
DEBUG:  established connection to localhost:9702 for session 37
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 33
DEBUG:  established connection to localhost:9702 for session 38
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 34
DEBUG:  established connection to localhost:9702 for session 24
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 20
DEBUG:  established connection to localhost:9701 for session 14
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 11
DEBUG:  established connection to localhost:9702 for session 34
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 30
DEBUG:  established connection to localhost:9702 for session 36
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 32
DEBUG:  established connection to localhost:9702 for session 31
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 27
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102011 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 12
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102013 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 5
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102015 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 8
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102017 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 9
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102019 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 7
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102021 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 6
DEBUG:  established connection to localhost:9702 for session 27
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 23
DEBUG:  established connection to localhost:9701 for session 21
NOTICE:  issuing BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 6, '2020-02-21 11:04:55.932792+03');
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 18
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102023 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 13
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102025 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 10
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102012 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 29
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102014 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 34
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102016 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 25
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102027 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 19
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102018 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 22
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102020 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 26
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102029 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 15
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102022 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 23
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102024 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 24
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102026 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 21
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102031 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 17
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102033 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 11
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102028 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 27
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102030 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 28
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102032 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 33
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102035 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 16
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102037 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 18
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102034 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 32
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102036 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 31
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102038 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 20
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102039 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 14
NOTICE:  issuing SELECT count(*) AS count FROM dist_table_102040 dist_table WHERE true
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 30
DEBUG:  Total number of commands sent over the session 7: 1
DEBUG:  Total number of commands sent over the session 8: 1
DEBUG:  Total number of commands sent over the session 9: 1
DEBUG:  Total number of commands sent over the session 10: 1
DEBUG:  Total number of commands sent over the session 11: 1
DEBUG:  Total number of commands sent over the session 12: 1
DEBUG:  Total number of commands sent over the session 13: 1
DEBUG:  Total number of commands sent over the session 14: 1
DEBUG:  Total number of commands sent over the session 15: 1
DEBUG:  Total number of commands sent over the session 16: 1
DEBUG:  Total number of commands sent over the session 17: 1
DEBUG:  Total number of commands sent over the session 18: 1
DEBUG:  Total number of commands sent over the session 19: 1
DEBUG:  Total number of commands sent over the session 20: 1
DEBUG:  Total number of commands sent over the session 21: 1
DEBUG:  Total number of commands sent over the session 22: 1
DEBUG:  Total number of commands sent over the session 23: 1
DEBUG:  Total number of commands sent over the session 24: 1
DEBUG:  Total number of commands sent over the session 25: 1
DEBUG:  Total number of commands sent over the session 26: 1
DEBUG:  Total number of commands sent over the session 27: 1
DEBUG:  Total number of commands sent over the session 28: 1
DEBUG:  Total number of commands sent over the session 29: 1
DEBUG:  Total number of commands sent over the session 30: 1
DEBUG:  Total number of commands sent over the session 31: 1
DEBUG:  Total number of commands sent over the session 32: 1
DEBUG:  Total number of commands sent over the session 33: 1
DEBUG:  Total number of commands sent over the session 34: 1
DEBUG:  Total number of commands sent over the session 35: 1
DEBUG:  Total number of commands sent over the session 36: 1
DEBUG:  Total number of commands sent over the session 37: 1
DEBUG:  Total number of commands sent over the session 38: 1
 count
-------
     0
(1 row)

onurctirtir=#   DROP TABLE dist_table;
DEBUG:  drop auto-cascades to type dist_table
DEBUG:  drop auto-cascades to type dist_table[]
DEBUG:  drop auto-cascades to trigger truncate_trigger_16818 on table dist_table
DEBUG:  drop auto-cascades to constraint fkey on table dist_table
DEBUG:  drop auto-cascades to trigger RI_ConstraintTrigger_a_16820 on table ref_table
DEBUG:  drop auto-cascades to trigger RI_ConstraintTrigger_a_16821 on table ref_table
DEBUG:  drop auto-cascades to trigger RI_ConstraintTrigger_c_16822 on table dist_table
DEBUG:  drop auto-cascades to trigger RI_ConstraintTrigger_c_16823 on table dist_table
DEBUG:  EventTriggerInvoke 16450
NOTICE:  issuing DROP TABLE IF EXISTS public.dist_table_102009 CASCADE
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 3
NOTICE:  issuing DROP TABLE IF EXISTS public.dist_table_102010 CASCADE
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 4
NOTICE:  issuing DROP TABLE IF EXISTS public.dist_table_102011 CASCADE
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 12
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 3
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 4
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 5
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 6
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 7
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 8
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 9
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 25
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 19
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 13
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 10
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 29
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 16
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 15
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 24
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 21
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 28
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 26
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 17
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 22
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 14
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 31
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 33
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 34
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 20
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 11
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 30
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 32
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 27
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9702 connectionId: 23
NOTICE:  issuing ROLLBACK
DETAIL:  on server onurctirtir@localhost:9701 connectionId: 18
ERROR:  canceling the transaction since it was involved in a distributed deadlock
CONTEXT:  SQL statement "SELECT master_drop_all_shards(v_obj.objid, v_obj.schema_name, v_obj.object_name)"
PL/pgSQL function citus_drop_trigger() line 19 at PERFORM
onurctirtir commented 4 years ago

I guess, this bug needs a backport label :/ (you can remove it if it does not)

onderkalaci commented 4 years ago

I guess, this bug needs a backport label :/ (you can remove it if it does not)

This is probably a bug that exists from 7.5 or so. So, probably nothing urgent to backport, but if we have a release, we can cherry-pick it.

onderkalaci commented 4 years ago

We should also double check what is the behavior for TRUCATE / other DDLs etc

onurctirtir commented 4 years ago

same behavior is observed for TRUNCATE as well

onderkalaci commented 4 years ago

same behavior is observed for TRUNCATE as well

Are you sure? I wasn't able to reproduce that with the steps you've shared above, replacing DROP with TRUNCATE.

Well, if TRUNCATE also leads to issue, this is becoming a more important.

onurctirtir commented 4 years ago

Sorry for inconvenience You are right, not applicable for TRUNCATE