tempesta-tech / mariadb

MariaDB System Versioning

GNU General Public License v2.0

6 stars 5 forks source link

VTMD column names tracking (CMMD) (4w) #205

Closed midenok closed 4 years ago

midenok commented 7 years ago

2 operations of column changes should be tracked without data copying:

name change;
order change.

When column type is changed, data should be copied.

create or replace table t1 (x int, y int) with system versioning engine innodb;
insert into t1 values (1, 2);
set @p0= now(6);
alter table t1 change y z int;
set @p1= now(6);
alter table t1 change x x int after z;
set @p2= now(6);

Related to #126. Depends on #127.

TODO

[ ] CMMD (Column Mapping Metadata) table (2w);
[ ] Track column name, order changes (1w);
[ ] Testing, fixes (1w).

midenok commented 7 years ago

VTMD ctor on rename

#0  VTMD_table::VTMD_table (this=0x7fffe5718b60, _about=...) at /home/midenok/src/mariadb/midenok/src/sql/vtmd.h:76
#1  0x0000555555b8cb25 in VTMD_exists::VTMD_exists (this=0x7fffe5718b60, _about=...) at /home/midenok/src/mariadb/midenok/src/sql/vtmd.h:104
#2  0x0000555555b8cb7b in VTMD_rename::VTMD_rename (this=0x7fffe5718b60, _about=...) at /home/midenok/src/mariadb/midenok/src/sql/vtmd.h:116
#3  0x0000555555c39b51 in mysql_alter_table (thd=0x7fff8c000b00, new_db=0x7fff8c011700 "t205", new_name=0x0, create_info=0x7fffe571a4d0, table_list=0x7fff8c0110d0, alter_info=0x7fffe571a420, order_num=0, order=0x0, ignore=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:9625
#4  0x0000555555cbb752 in Sql_cmd_alter_table::execute (this=0x7fff8c011800, thd=0x7fff8c000b00) at /home/midenok/src/mariadb/midenok/src/sql/sql_alter.cc:318
#5  0x0000555555b67ad7 in mysql_execute_command (thd=0x7fff8c000b00) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:6224
#6  0x0000555555b6bdd8 in mysql_parse (thd=0x7fff8c000b00, rawbuf=0x7fff8c010fe8 "alter table t1 change z y int", length=29, parser_state=0x7fffe571b6b0, is_com_multi=false, is_next_command=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:7902

midenok commented 7 years ago

How it works now

1. detect if we need archive table (`vers_data_mod` flag).

  bool vers_data_mod= versioned &&
    thd->variables.vers_alter_history == VERS_ALTER_HISTORY_SURVIVE &&
    alter_info->vers_data_modifying();

2. set backup table name to archive table

  char backup_name[FN_LEN];
  if (vers_data_mod)
    VTMD_table::archive_name(thd, alter_ctx.table_name, backup_name,
                         sizeof(backup_name));
  else
    my_snprintf(backup_name, sizeof(backup_name), "%s2-%lx-%lx",
                tmp_file_prefix, current_pid, thd->thread_id);

3. put info into VTMD

  if (vers_data_mod && new_versioned)
  {
    DBUG_ASSERT(alter_info && table_list);
    VTMD_rename vtmd(*table_list);
    bool rc= alter_info->flags & Alter_info::ALTER_RENAME ?
      vtmd.try_rename(thd, alter_ctx.new_db, alter_ctx.new_alias, backup_name) :
      vtmd.update(thd, backup_name);
    if (rc)
      goto err_after_rename;
  }

4. skip deleting backup table

  // ALTER TABLE succeeded, delete the backup of the old table.
  if (!(vers_data_mod && new_versioned) &&
      quick_rm_table(thd, old_db_type, alter_ctx.db, backup_name, FN_IS_TMP))
  {

midenok commented 7 years ago

We need to split flag vers_data_mod into two flags:

vers_save_archive that will store archive table;
vers_update_vtmd that will update VTMD accordingly.

Need to detect on ALTER_CHANGE_COLUMN that it is only renamed or data type changed.

midenok commented 7 years ago

vers_save_archive may be modified by copy_data_between_tables(). Here it fills array of modified fields:

  for (Field **ptr=to->field ; *ptr ; ptr++)
  {
    def=it++;
    if (def->field)
    {
     ...
      (copy_end++)->set(*ptr,def->field,0);
    }

class Copy_field :public Sql_alloc {
public:
  uchar *from_ptr,*to_ptr;
  uchar *from_null_ptr,*to_null_ptr;
  bool *null_row;
  uint  from_bit,to_bit;
  /**
    Number of bytes in the fields pointed to by 'from_ptr' and
    'to_ptr'. Usually this is the number of bytes that are copied from
    'from_ptr' to 'to_ptr'.

    For variable-length fields (VARCHAR), the first byte(s) describe
    the actual length of the text. For VARCHARs with length 
       < 256 there is 1 length byte 
       >= 256 there is 2 length bytes
    Thus, if from_field is VARCHAR(10), from_length (and in most cases
    to_length) is 11. For VARCHAR(1024), the length is 1026. @see
    Field_varstring::length_bytes

    Note that for VARCHARs, do_copy() will be do_varstring*() which
    only copies the length-bytes (1 or 2) + the actual length of the
    text instead of from/to_length bytes.
  */
  uint from_length,to_length;
  Field *from_field,*to_field;

midenok commented 7 years ago

`vtmd.test` fails

#0  my_error (nr=4088, MyFlags=0) at /home/midenok/src/mariadb/midenok/src/mysys/my_error.c:109
#1  0x0000555555dbb2b2 in Local_da::finish (this=0x7ffff0a85220) at /home/midenok/src/mariadb/midenok/src/sql/vers_utils.h:60
#2  0x0000555555dbb1dc in Local_da::~Local_da (this=0x7ffff0a85220, __in_chrg=<optimized out>) at /home/midenok/src/mariadb/midenok/src/sql/vers_utils.h:53
#3  0x0000555555db94c3 in VTMD_table::update (this=0x7ffff0a85c20, thd=0x7fffa8000b00, archive_name=0x0) at /home/midenok/src/mariadb/midenok/src/sql/vtmd.cc:243
#4  0x0000555555c2da9c in mysql_create_table (thd=0x7fffa8000b00, create_table=0x7fffa8012668, create_info=0x7ffff0a85f80, alter_info=0x7ffff0a85ed0) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:5083
#5  0x0000555555b61b11 in mysql_execute_command (thd=0x7fffa8000b00) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:4084
#6  0x0000555555b6bdd6 in mysql_parse (thd=0x7fffa8000b00, rawbuf=0x7fffa8012548 "create or replace table t0 (x int) with system versioning", length=57, parser_state=0x7ffff0a86660, is_com_multi=false, is_next_command=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:7902

(gdb) p sql_error
$1 = 4088
(gdb) p message()
$2 = 0x7ffff0a8523b "`test.t0_vtmd` exists and not empty!"

#0  VTMD_table::update (this=0x7ffff0a85c20, thd=0x7fffa0000b00, archive_name=0x0) at /home/midenok/src/mariadb/midenok/src/sql/vtmd.cc:170
#1  0x0000555555c2da9c in mysql_create_table (thd=0x7fffa0000b00, create_table=0x7fffa0012668, create_info=0x7ffff0a85f80, alter_info=0x7ffff0a85ed0) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:5083
#2  0x0000555555b61b11 in mysql_execute_command (thd=0x7fffa0000b00) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:4084
#3  0x0000555555b6bdd6 in mysql_parse (thd=0x7fffa0000b00, rawbuf=0x7fffa0012548 "create or replace table t0 (x int) with system versioning", length=57, parser_state=0x7ffff0a86660, is_com_multi=false, is_next_command=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:7902

midenok commented 7 years ago

In case of any VTMD error we don't want to continue ALTER operation. We want to return to original state, because otherwise we are losing survival sense of feature.

midenok commented 7 years ago

#0  THD::parse_error (this=0x7fffa8000b00, err_text=0x555557beaaef "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use", yytext=0x0) at /home/midenok/src/mariadb/midenok/src/sql/sql_class.h:3935
#1  0x0000555555d4b785 in MYSQLerror (thd=0x7fffa8000b00, s=0x555557beaaef "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use") at /home/midenok/src/mariadb/midenok/src/sql/sql_yacc.yy:150
#2  0x0000555555d79585 in MYSQLparse (thd=0x7fffa8000b00) at /home/midenok/src/mariadb/midenok/build/sql/sql_yacc.cc:45498
#3  0x0000555555b70c39 in parse_sql (thd=0x7fffa8000b00, parser_state=0x7ffff0a84730, creation_ctx=0x0, do_pfs_digest=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:9932
#4  0x0000555555b879f2 in Prepared_statement::prepare (this=0x7fffa80e92f0, packet=0x7ffff0a84940 "drop table #sql2-1bf3-8", packet_len=23) at /home/midenok/src/mariadb/midenok/src/sql/sql_prepare.cc:3830
#5  0x0000555555b84f42 in mysql_sql_stmt_prepare (thd=0x7fffa8000b00) at /home/midenok/src/mariadb/midenok/src/sql/sql_prepare.cc:2823
#6  0x0000555555b6079a in mysql_execute_command (thd=0x7fffa8000b00) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:3585
#7  0x0000555555f874fe in sp_instr_stmt::exec_core (this=0x7fffa80d0d68, thd=0x7fffa8000b00, nextp=0x7ffff0a853d4) at /home/midenok/src/mariadb/midenok/src/sql/sp_head.cc:3344
#8  0x0000555555f86a91 in sp_lex_keeper::reset_lex_and_exec_core (this=0x7fffa80d0db0, thd=0x7fffa8000b00, nextp=0x7ffff0a853d4, open_tables=false, instr=0x7fffa80d0d68) at /home/midenok/src/mariadb/midenok/src/sql/sp_head.cc:3088
#9  0x0000555555f87195 in sp_instr_stmt::execute (this=0x7fffa80d0d68, thd=0x7fffa8000b00, nextp=0x7ffff0a853d4) at /home/midenok/src/mariadb/midenok/src/sql/sp_head.cc:3260
#10 0x0000555555f81591 in sp_head::execute (this=0x7fffa80bd608, thd=0x7fffa8000b00, merge_da_on_success=true) at /home/midenok/src/mariadb/midenok/src/sql/sp_head.cc:1233
#11 0x0000555555f837ef in sp_head::execute_procedure (this=0x7fffa80bd608, thd=0x7fffa8000b00, args=0x7fffa8005430) at /home/midenok/src/mariadb/midenok/src/sql/sp_head.cc:2059
#12 0x0000555555b5eb56 in do_execute_sp (thd=0x7fffa8000b00, sp=0x7fffa80bd608) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:2882
#13 0x0000555555b66783 in mysql_execute_command (thd=0x7fffa8000b00) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:5810
#14 0x0000555555b6bdd6 in mysql_parse (thd=0x7fffa8000b00, rawbuf=0x7fffa8012548 "call drop_archives('t0_vtmd')", length=29, parser_state=0x7ffff0a86660, is_com_multi=false, is_next_command=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:7902

Failed statement:

drop table #sql2-1bf3-8

Temporary table name in VTMD.

midenok commented 7 years ago

alter table t1 change column z y int;
# or
alter table t1 modify column z int;

Flags set: ALTER_CHANGE_COLUMN

alter table t1 change column z z int after x;
# or
alter table t1 modify column z int after x;

Flags set: ALTER_CHANGE_COLUMN | ALTER_COLUMN_ORDER

midenok commented 7 years ago

In case of use_inplace how we can revert back altered table in case of VTMD error?

    if (use_inplace)
    {
      table->s->frm_image= &frm;
      int res= mysql_inplace_alter_table(thd, table_list, table, altered_table,
                                         &ha_alter_info, inplace_supported,
                                         &target_mdl_request, &alter_ctx);
      my_free(const_cast<uchar*>(frm.str));

      if (res)
        DBUG_RETURN(true);

      goto end_inplace;
    }

midenok commented 7 years ago

Wrong field name on SELECT

#0  my_error (nr=1054, MyFlags=0) at /home/midenok/src/mariadb/midenok/src/mysys/my_error.c:109
#1  0x0000555555aead50 in find_field_in_tables (thd=0x7fff8c000b00, item=0x7fff8c011088, first_table=0x7fff8c0111c8, last_table=0x0, ref=0x7fff8c011188, report_error=REPORT_ALL_ERRORS, check_privileges=true, register_tree_change=true) at /home/midenok/src/mariadb/midenok/src/sql/sql_base.cc:6010
#2  0x0000555555e27498 in Item_field::fix_outer_field (this=0x7fff8c011088, thd=0x7fff8c000b00, from_field=0x7fffe571a348, reference=0x7fff8c011188) at /home/midenok/src/mariadb/midenok/src/sql/item.cc:5472
#3  0x0000555555e27f5d in Item_field::fix_fields (this=0x7fff8c011088, thd=0x7fff8c000b00, reference=0x7fff8c011188) at /home/midenok/src/mariadb/midenok/src/sql/item.cc:5709
#4  0x0000555555aed90e in setup_fields (thd=0x7fff8c000b00, ref_pointer_array=..., fields=..., mark_used_columns=MARK_COLUMNS_READ, sum_func_list=0x7fff8c011c00, allow_sum_func=true) at /home/midenok/src/mariadb/midenok/src/sql/sql_base.cc:7124
#5  0x0000555555ba0a0f in JOIN::prepare (this=0x7fff8c0118e8, tables_init=0x7fff8c0111c8, wild_num=0, conds_init=0x0, og_num=0, order_init=0x0, skip_order_by=false, group_init=0x0, having_init=0x0, proc_param_init=0x0, select_lex_arg=0x7fff8c004e48, unit_arg=0x7fff8c0046e8) at /home/midenok/src/mariadb/midenok/src/sql/sql_select.cc:1168
#6  0x0000555555bab2fd in mysql_select (thd=0x7fff8c000b00, tables=0x7fff8c0111c8, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748608, result=0x7fff8c0118c8, unit=0x7fff8c0046e8, select_lex=0x7fff8c004e48) at /home/midenok/src/mariadb/midenok/src/sql/sql_select.cc:4055
#7  0x0000555555b9d7c8 in handle_select (thd=0x7fff8c000b00, lex=0x7fff8c004620, result=0x7fff8c0118c8, setup_tables_done_option=0) at /home/midenok/src/mariadb/midenok/src/sql/sql_select.cc:363
#8  0x0000555555b68ad5 in execute_sqlcom_select (thd=0x7fff8c000b00, all_tables=0x7fff8c0111c8) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:6455
#9  0x0000555555b607a9 in mysql_execute_command (thd=0x7fff8c000b00) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:3574
#10 0x0000555555b6be16 in mysql_parse (thd=0x7fff8c000b00, rawbuf=0x7fff8c010fe8 "select r from t1", length=16, parser_state=0x7fffe571b6b0, is_com_multi=false, is_next_command=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:7902

midenok commented 7 years ago

Shadow table: archive table that has data of older archive table. Shadow table can have some fields renamed and/or moved.

Shadow table tasks:

[ ] init TABLE_SHARE from original archive table;
[ ] modify s->name_hash, table->field array, etc.
[ ] keep original field array and map it from table->field

midenok commented 7 years ago

Tables opened at connect

#0  TABLE_SHARE::init_from_binary_frm_image (this=0x7fff8c0e9ae8, thd=0x7fff8c000b00, write=false, frm_image=0x7fff8c0ea1e0 "\376\001\n\f\033", frm_length=2092) at /home/midenok/src/mariadb/midenok/src/sql/table.cc:1171
#1  0x0000555555c61cc6 in open_table_def (thd=0x7fff8c000b00, share=0x7fff8c0e9ae8, flags=11) at /home/midenok/src/mariadb/midenok/src/sql/table.cc:674
#2  0x0000555555d450dd in tdc_acquire_share (thd=0x7fff8c000b00, tl=0x7fffe571b6b0, flags=3, out_table=0x7fffe571ad58) at /home/midenok/src/mariadb/midenok/src/sql/table_cache.cc:825
#3  0x0000555555ae1dc3 in open_table (thd=0x7fff8c000b00, table_list=0x7fffe571b6b0, ot_ctx=0x7fffe571b360) at /home/midenok/src/mariadb/midenok/src/sql/sql_base.cc:1742
#4  0x0000555555ae4ecf in open_and_process_table (thd=0x7fff8c000b00, lex=0x7fff8c004620, tables=0x7fffe571b6b0, counter=0x7fffe571b424, flags=1024, prelocking_strategy=0x7fffe571b428, has_prelocking_list=false, ot_ctx=0x7fffe571b360) at /home/midenok/src/mariadb/midenok/src/sql/sql_base.cc:3414
#5  0x0000555555ae60c8 in open_tables (thd=0x7fff8c000b00, options=..., start=0x7fffe571b410, counter=0x7fffe571b424, flags=1024, prelocking_strategy=0x7fffe571b428) at /home/midenok/src/mariadb/midenok/src/sql/sql_base.cc:3932
#6  0x0000555555adeb26 in open_tables (thd=0x7fff8c000b00, tables=0x7fffe571b410, counter=0x7fffe571b424, flags=1024, prelocking_strategy=0x7fffe571b428) at /home/midenok/src/mariadb/midenok/src/sql/sql_base.h:236
#7  0x0000555555ae799e in open_normal_and_derived_tables (thd=0x7fff8c000b00, tables=0x7fffe571b6b0, flags=1024, dt_phases=34) at /home/midenok/src/mariadb/midenok/src/sql/sql_base.cc:4750
#8  0x0000555555bf4470 in mysqld_list_fields (thd=0x7fff8c000b00, table_list=0x7fffe571b6b0, wild=0x7fff8c011018 "") at /home/midenok/src/mariadb/midenok/src/sql/sql_show.cc:1458
#9  0x0000555555b5c892 in dispatch_command (command=COM_FIELD_LIST, thd=0x7fff8c000b00, packet=0x7fff8c008809 "", packet_length=8, is_com_multi=false, is_next_command=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:2008
#10 0x0000555555b5adf2 in do_command (thd=0x7fff8c000b00) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:1376
#11 0x0000555555cb668e in do_handle_one_connection (connect=0x555558368bd0) at /home/midenok/src/mariadb/midenok/src/sql/sql_connect.cc:1354
#12 0x0000555555cb640d in handle_one_connection (arg=0x555558368bd0) at /home/midenok/src/mariadb/midenok/src/sql/sql_connect.cc:1260
#13 0x00007ffff72f76da in start_thread (arg=0x7fffe571c700) at pthread_create.c:456
#14 0x00007ffff6585d7f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105

$24 = {
  str = 0x7fff8c0ea145 "t1_vtmd", 
  length = 7
}

Table opened at `CREATE TABLE`

#0  TABLE_SHARE::init_from_binary_frm_image (this=0x7fffe5717f90, thd=0x7fff8c000b00, write=true, frm_image=0x7fff8c167d10 "\376\001\n\f\022", frm_length=432) at /home/midenok/src/mariadb/midenok/src/sql/table.cc:1171
#1  0x0000555555e0d8a6 in ha_create_table (thd=0x7fff8c000b00, path=0x7fffe571a9c0 "./test/u", db=0x7fff8c0116f0 "test", table_name=0x7fff8c011088 "u", create_info=0x7fffe571afd0, frm=0x7fffe571a9b0) at /home/midenok/src/mariadb/midenok/src/sql/handler.cc:4762
#2  0x0000555555c83a39 in rea_create_table (thd=0x7fff8c000b00, frm=0x7fffe571a9b0, path=0x7fffe571a9c0 "./test/u", db=0x7fff8c0116f0 "test", table_name=0x7fff8c011088 "u", create_info=0x7fffe571afd0, file=0x7fff8c011aa0, no_ha_create_table=false) at /home/midenok/src/mariadb/midenok/src/sql/unireg.cc:510
#3  0x0000555555c2d07a in create_table_impl (thd=0x7fff8c000b00, orig_db=0x7fff8c0116f0 "test", orig_table_name=0x7fff8c011088 "u", db=0x7fff8c0116f0 "test", table_name=0x7fff8c011088 "u", path=0x7fffe571a9c0 "./test/u", options=..., create_info=0x7fffe571afd0, alter_info=0x7fffe571af20, create_table_mode=0, is_trans=0x7fffe571ac1e, key_info=0x7fffe571a9a0, key_count=0x7fffe571a994, frm=0x7fffe571a9b0) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:4869
#4  0x0000555555c2d689 in mysql_create_table_no_lock (thd=0x7fff8c000b00, db=0x7fff8c0116f0 "test", table_name=0x7fff8c011088 "u", create_info=0x7fffe571afd0, alter_info=0x7fffe571af20, is_trans=0x7fffe571ac1e, create_table_mode=0, table_list=0x7fff8c0110c0) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:4982
#5  0x0000555555c2da02 in mysql_create_table (thd=0x7fff8c000b00, create_table=0x7fff8c0110c0, create_info=0x7fffe571afd0, alter_info=0x7fffe571af20) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:5055
#6  0x0000555555b61b51 in mysql_execute_command (thd=0x7fff8c000b00) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:4084
#7  0x0000555555b6be16 in mysql_parse (thd=0x7fff8c000b00, rawbuf=0x7fff8c010fe8 "create table u (x int)", length=22, parser_state=0x7fffe571b6b0, is_com_multi=false, is_next_command=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:7902

midenok commented 7 years ago

There should be some init_from_binary_frm_image() variant that will open archive table with column mappings.

midenok commented 7 years ago

When no VTMD table:

alter table t1 change z y int;

Thread 33 "mysqld" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff4058700 (LWP 2125)]
0x0000555555e11d99 in handler::ha_reset (this=0x7fff80338598) at /home/midenok/src/mariadb/midenok/src/sql/handler.cc:5958
5958      DBUG_ASSERT((uchar*) table->def_read_set.bitmap +
(gdb) bt
#0  0x0000555555e11d99 in handler::ha_reset (this=0x7fff80338598) at /home/midenok/src/mariadb/midenok/src/sql/handler.cc:5958
#1  0x0000555555d4a08c in THD::mark_tmp_table_as_free_for_reuse (this=0x7fff80000d50, table=0x7fff80337980) at /home/midenok/src/mariadb/midenok/src/sql/temporary_tables.cc:782
#2  0x0000555555d49f43 in THD::mark_tmp_tables_as_free_for_reuse (this=0x7fff80000d50) at /home/midenok/src/mariadb/midenok/src/sql/temporary_tables.cc:743
#3  0x0000555555ae072b in close_thread_tables (thd=0x7fff80000d50) at /home/midenok/src/mariadb/midenok/src/sql/sql_base.cc:792
#4  0x0000555555ae8e17 in close_tables_for_reopen (thd=0x7fff80000d50, tables=0x7fff80004880, start_of_statement_svp=...) at /home/midenok/src/mariadb/midenok/src/sql/sql_base.cc:5140
#5  0x0000555555ae66b3 in open_tables (thd=0x7fff80000d50, options=..., start=0x7fff80004880, counter=0x7ffff40518b8, flags=0, prelocking_strategy=0x7ffff4051860) at /home/midenok/src/mariadb/midenok/src/sql/sql_base.cc:3955
#6  0x0000555555b2f524 in open_tables (thd=0x7fff80000d50, tables=0x7fff80004880, counter=0x7ffff40518b8, flags=0) at /home/midenok/src/mariadb/midenok/src/sql/sql_base.h:464
#7  0x0000555555c2fa8a in mysql_create_like_table (thd=0x7fff80000d50, table=0x7ffff4053780, src_table=0x7ffff4053150, create_info=0x7ffff4052fb0) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:5451
#8  0x0000555555db925f in VTMD_table::create (this=0x7ffff4054290, thd=0x7fff80000d50) at /home/midenok/src/mariadb/midenok/src/sql/vtmd.cc:48
#9  0x0000555555db978a in VTMD_table::open (this=0x7ffff4054290, thd=0x7fff80000d50, local_da=..., created=0x7ffff4053e51) at /home/midenok/src/mariadb/midenok/src/sql/vtmd.cc:114
#10 0x0000555555db98ed in VTMD_table::update (this=0x7ffff4054290, thd=0x7fff80000d50, archive_name=0x0, col_map=false) at /home/midenok/src/mariadb/midenok/src/sql/vtmd.cc:145
#11 0x0000555555c34bd0 in mysql_inplace_alter_table (thd=0x7fff80000d50, table_list=0x7fff80011360, table=0x7fff800d8290, altered_table=0x7fff80337980, ha_alter_info=0x7ffff4054aa0, inplace_supported=HA_ALTER_INPLACE_NO_LOCK_AFTER_PREPARE, target_mdl_request=0x7ffff4054b10, alter_ctx=0x7ffff40553a0, vers_update_vtmd=true, vers_archive_name=0x0) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:7423
#12 0x0000555555c39e39 in mysql_alter_table (thd=0x7fff80000d50, new_db=0x7fff80011990 "test", new_name=0x0, create_info=0x7ffff40564c0, table_list=0x7fff80011360, alter_info=0x7ffff4056410, order_num=0, order=0x0, ignore=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:9383
#13 0x0000555555cbc574 in Sql_cmd_alter_table::execute (this=0x7fff80011a90, thd=0x7fff80000d50) at /home/midenok/src/mariadb/midenok/src/sql/sql_alter.cc:318
#14 0x0000555555b68108 in mysql_execute_command (thd=0x7fff80000d50) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:6230
#15 0x0000555555b6c500 in mysql_parse (thd=0x7fff80000d50, rawbuf=0x7fff80011278 "alter table t1 change z y int", length=29, parser_state=0x7ffff40576a0, is_com_multi=false, is_next_command=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:7923

(gdb) p table
$1 = (TABLE *) 0x0

frame 1

(gdb) p table->alias
$3 = {
  Ptr = 0x7fff8830a4d0 "#sql-99d_9", 
  str_length = 10, 
  Alloced_length = 16, 
  extra_alloc = 0, 
  alloced = true, 
  thread_specific = false, 
  str_charset = 0x5555570777e0 <my_charset_bin>
}

midenok commented 7 years ago

1. tmp table create started

#0  THD::create_and_open_tmp_table (this=0x7fff88000d50, hton=0x555557b3e7d0, frm=0x7ffff4054a60, path=0x7ffff4055c4c "./test/#sql-f53_8", db=0x7fff88011990 "test", table_name=0x7ffff40553e0 "#sql-f53_8", open_in_engine=false) at /home/midenok/src/mariadb/midenok/src/sql/temporary_tables.cc:68
#1  0x0000555555c39a64 in mysql_alter_table (thd=0x7fff88000d50, new_db=0x7fff88011990 "test", new_name=0x0, create_info=0x7ffff40564c0, table_list=0x7fff88011360, alter_info=0x7ffff4056410, order_num=0, order=0x0, ignore=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:9289
#2  0x0000555555cbc574 in Sql_cmd_alter_table::execute (this=0x7fff88011a90, thd=0x7fff88000d50) at /home/midenok/src/mariadb/midenok/src/sql/sql_alter.cc:318
#3  0x0000555555b68108 in mysql_execute_command (thd=0x7fff88000d50) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:6230
#4  0x0000555555b6c500 in mysql_parse (thd=0x7fff88000d50, rawbuf=0x7fff88011278 "alter table t1 change z y int", length=29, parser_state=0x7ffff40576a0, is_com_multi=false, is_next_command=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:7923

2. `table->file` assigned

#0  get_new_handler (share=0x7fff803196b0, alloc=0x7fff80311d58, db_type=0x555557b3e7d0) at /home/midenok/src/mariadb/midenok/src/sql/handler.cc:252
#1  0x0000555555c6a91d in open_table_from_share (thd=0x7fff80000d50, share=0x7fff803196b0, alias=0x7ffff40553e0 "#sql-22f4_8", db_stat=1, prgflag=8, ha_open_flags=4112, outparam=0x7fff80311290, is_create_table=false) at /home/midenok/src/mariadb/midenok/src/sql/table.cc:3059
#2  0x0000555555d4aba1 in THD::open_temporary_table (this=0x7fff80000d50, share=0x7fff803196b0, alias=0x7ffff40553e0 "#sql-22f4_8", open_in_engine=true) at /home/midenok/src/mariadb/midenok/src/sql/temporary_tables.cc:1120
#3  0x0000555555d4887d in THD::create_and_open_tmp_table (this=0x7fff80000d50, hton=0x555557b3e7d0, frm=0x7ffff4054a60, path=0x7ffff4055c4c "./test/#sql-22f4_8", db=0x7fff80011998 "test", table_name=0x7ffff40553e0 "#sql-22f4_8", open_in_engine=true) at /home/midenok/src/mariadb/midenok/src/sql/temporary_tables.cc:77
#4  0x0000555555c3a2be in mysql_alter_table (thd=0x7fff80000d50, new_db=0x7fff80011998 "test", new_name=0x0, create_info=0x7ffff40564c0, table_list=0x7fff80011368, alter_info=0x7ffff4056410, order_num=0, order=0x0, ignore=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:9487
#5  0x0000555555cbc574 in Sql_cmd_alter_table::execute (this=0x7fff80011a90, thd=0x7fff80000d50) at /home/midenok/src/mariadb/midenok/src/sql/sql_alter.cc:318
#6  0x0000555555b68108 in mysql_execute_command (thd=0x7fff80000d50) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:6230
#7  0x0000555555b6c500 in mysql_parse (thd=0x7fff80000d50, rawbuf=0x7fff80011278 "alter table t0 add column (r int)", length=33, parser_state=0x7ffff40576a0, is_com_multi=false, is_next_command=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:7923

3. `table->file->table` assigned

#0  handler::ha_open (this=0x7fff80311ea8, table_arg=0x7fff80311290, name=0x7fff80319d20 "./test/#sql-22f4_8", mode=2, test_if_locked=4114) at /home/midenok/src/mariadb/midenok/src/sql/handler.cc:2516
#1  0x0000555555c6bb6f in open_table_from_share (thd=0x7fff80000d50, share=0x7fff803196b0, alias=0x7ffff40553e0 "#sql-22f4_8", db_stat=1, prgflag=8, ha_open_flags=4114, outparam=0x7fff80311290, is_create_table=false) at /home/midenok/src/mariadb/midenok/src/sql/table.cc:3394
#2  0x0000555555d4aba1 in THD::open_temporary_table (this=0x7fff80000d50, share=0x7fff803196b0, alias=0x7ffff40553e0 "#sql-22f4_8", open_in_engine=true) at /home/midenok/src/mariadb/midenok/src/sql/temporary_tables.cc:1120
#3  0x0000555555d4887d in THD::create_and_open_tmp_table (this=0x7fff80000d50, hton=0x555557b3e7d0, frm=0x7ffff4054a60, path=0x7ffff4055c4c "./test/#sql-22f4_8", db=0x7fff80011998 "test", table_name=0x7ffff40553e0 "#sql-22f4_8", open_in_engine=true) at /home/midenok/src/mariadb/midenok/src/sql/temporary_tables.cc:77
#4  0x0000555555c3a2be in mysql_alter_table (thd=0x7fff80000d50, new_db=0x7fff80011998 "test", new_name=0x0, create_info=0x7ffff40564c0, table_list=0x7fff80011368, alter_info=0x7ffff4056410, order_num=0, order=0x0, ignore=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:9487
#5  0x0000555555cbc574 in Sql_cmd_alter_table::execute (this=0x7fff80011a90, thd=0x7fff80000d50) at /home/midenok/src/mariadb/midenok/src/sql/sql_alter.cc:318
#6  0x0000555555b68108 in mysql_execute_command (thd=0x7fff80000d50) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:6230
#7  0x0000555555b6c500 in mysql_parse (thd=0x7fff80000d50, rawbuf=0x7fff80011278 "alter table t0 add column (r int)", length=33, parser_state=0x7ffff40576a0, is_com_multi=false, is_next_command=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:7923

midenok commented 7 years ago

    if (!(altered_table=
          thd->create_and_open_tmp_table(new_db_type, &frm,
                                         alter_ctx.get_tmp_path(),
                                         alter_ctx.new_db, alter_ctx.tmp_name,
                                         false)))

Table is open without open_in_engine, so table->file->table is not set.

midenok commented 7 years ago

After this crash fix:

--- a/sql/sql_table.cc
+++ b/sql/sql_table.cc
@@ -7420,7 +7420,11 @@ static bool mysql_inplace_alter_table(THD *thd,
     DBUG_ASSERT(alter_info && table_list);
     DBUG_ASSERT(!(alter_info->flags & Alter_info::ALTER_RENAME));
     VTMD_table vtmd(*table_list);
+    DBUG_ASSERT(!altered_table->open_by_handler);
+    // protect from mark_tmp_table_as_free_for_reuse() caused by mysql_create_like_table()
+    altered_table->open_by_handler= true;
     bool rc= vtmd.update(thd, vers_archive_name);
+    altered_table->open_by_handler= false;

Assertion fails:

#3  0x00007ffff6442fc2 in __GI___assert_fail (assertion=0x5555567e09b8 "ha_thd()->mdl_context.is_lock_owner(MDL_key::TABLE, table->s->db.str, table->s->table_name.str, MDL_EXCLUSIVE) || !commit", file=0x5555567df3d0 "/home/midenok/src/mariadb/midenok/src/sql/handler.cc", line=4270, function=0x5555567e2360 <handler::ha_commit_inplace_alter_table(TABLE*, Alter_inplace_info*, bool)::__PRETTY_FUNCTION__> "bool handler::ha_commit_inplace_alter_table(TABLE*, Alter_inplace_info*, bool)") at assert.c:101
#4  0x0000555555e0dbe5 in handler::ha_commit_inplace_alter_table (this=0x7fff880d4c28, altered_table=0x7fff8830bb70, ha_alter_info=0x7ffff4054aa0, commit=true) at /home/midenok/src/mariadb/midenok/src/sql/handler.cc:4266
#5  0x0000555555c34c7a in mysql_inplace_alter_table (thd=0x7fff88000d50, table_list=0x7fff88011360, table=0x7fff880cfd30, altered_table=0x7fff8830bb70, ha_alter_info=0x7ffff4054aa0, inplace_supported=HA_ALTER_INPLACE_NO_LOCK_AFTER_PREPARE, target_mdl_request=0x7ffff4054b10, alter_ctx=0x7ffff40553a0, vers_update_vtmd=true, vers_archive_name=0x0) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:7432
#6  0x0000555555c39e89 in mysql_alter_table (thd=0x7fff88000d50, new_db=0x7fff88011990 "test", new_name=0x0, create_info=0x7ffff40564c0, table_list=0x7fff88011360, alter_info=0x7ffff4056410, order_num=0, order=0x0, ignore=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:9387
#7  0x0000555555cbc5c4 in Sql_cmd_alter_table::execute (this=0x7fff88011a90, thd=0x7fff88000d50) at /home/midenok/src/mariadb/midenok/src/sql/sql_alter.cc:318
#8  0x0000555555b68108 in mysql_execute_command (thd=0x7fff88000d50) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:6230
#9  0x0000555555b6c500 in mysql_parse (thd=0x7fff88000d50, rawbuf=0x7fff88011278 "alter table t1 change z y int", length=29, parser_state=0x7ffff40576a0, is_com_multi=false, is_next_command=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:7923

(gdb) f 4
#4  0x0000555555e0dbe5 in handler::ha_commit_inplace_alter_table (this=0x7fff880d4c28, altered_table=0x7fff8830bb70, ha_alter_info=0x7ffff4054aa0, commit=true) at /home/midenok/src/mariadb/midenok/src/sql/handler.cc:4266
4266       DBUG_ASSERT(ha_thd()->mdl_context.is_lock_owner(MDL_key::TABLE,
(gdb) p table->alias.Ptr
$4 = 0x7fff880d3fc0 "vtmd_template"
(gdb) p altered_table->alias.Ptr
$5 = 0x7fff88307f30 "#sql-10ac_8"

Expected altered_table and table to be same. mysql_create_like_table() changed context of handler. Need to push/pop handler context.

midenok commented 7 years ago

table before mysql_create_like_table():

(gdb) p table->alias.Ptr
$17 = 0x7fff740da390 "t1"

#0  mysql_inplace_alter_table (thd=0x7fff74000d50, table_list=0x7fff74011360, table=0x7fff740d9790, altered_table=0x7fff74336450, ha_alter_info=0x7ffff4054aa0, inplace_supported=HA_ALTER_INPLACE_NO_LOCK_AFTER_PREPARE, target_mdl_request=0x7ffff4054b10, alter_ctx=0x7ffff40553a0, vers_update_vtmd=true, vers_archive_name=0x0) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:7418
#1  0x0000555555c39e89 in mysql_alter_table (thd=0x7fff74000d50, new_db=0x7fff74011990 "test", new_name=0x0, create_info=0x7ffff40564c0, table_list=0x7fff74011360, alter_info=0x7ffff4056410, order_num=0, order=0x0, ignore=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:9387
#2  0x0000555555cbc5c4 in Sql_cmd_alter_table::execute (this=0x7fff74011a90, thd=0x7fff74000d50) at /home/midenok/src/mariadb/midenok/src/sql/sql_alter.cc:318
#3  0x0000555555b68108 in mysql_execute_command (thd=0x7fff74000d50) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:6230
#4  0x0000555555b6c500 in mysql_parse (thd=0x7fff74000d50, rawbuf=0x7fff74011278 "alter table t1 change z y int", length=29, parser_state=0x7ffff40576a0, is_com_multi=false, is_next_command=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:7923

table is closed after mysql_create_like_table().

midenok commented 7 years ago

mysql_create_like_table() opens src_table (mysql.vtmd_template) and keeps it open. So, it should be closed manually.

midenok commented 7 years ago

After 'Crash fix part 2':

#3  0x00007ffff6442fc2 in __GI___assert_fail (assertion=0x5555568be991 "val > 0", file=0x5555568be890 "/home/midenok/src/mariadb/midenok/src/storage/innobase/row/row0upd.cc", line=511, function=0x5555568bffc0 <row_upd_index_entry_sys_field(dtuple_t*, dict_index_t*, unsigned long, unsigned long)::__PRETTY_FUNCTION__> "void row_upd_index_entry_sys_field(dtuple_t*, dict_index_t*, ulint, ib_uint64_t)") at assert.c:101
#4  0x00005555561d6540 in row_upd_index_entry_sys_field (entry=0x7fff780d7fb8, index=0x5555581ad818, type=1, val=0) at /home/midenok/src/mariadb/midenok/src/storage/innobase/row/row0upd.cc:511
#5  0x000055555615bf95 in vers_row_ins_vtq_low (trx=0x7fffefbffe40, heap=0x7fff780d7e00, tuple=0x7fff780d7e88) at /home/midenok/src/mariadb/midenok/src/storage/innobase/row/row0ins.cc:3941
#6  0x000055555615c318 in vers_notify_vtq (trx=0x7fffefbffe40) at /home/midenok/src/mariadb/midenok/src/storage/innobase/row/row0ins.cc:4003
#7  0x000055555601b42c in innobase_commit (hton=0x555557b3e7d0, thd=0x7fff78000d50, commit_trx=false) at /home/midenok/src/mariadb/midenok/src/storage/innobase/handler/ha_innodb.cc:4846
#8  0x0000555555e06633 in commit_one_phase_2 (thd=0x7fff78000d50, all=false, trans=0x7fff78004040, is_real_trans=true) at /home/midenok/src/mariadb/midenok/src/sql/handler.cc:1558
#9  0x0000555555e06513 in ha_commit_one_phase (thd=0x7fff78000d50, all=false) at /home/midenok/src/mariadb/midenok/src/sql/handler.cc:1539
#10 0x0000555555e05e19 in ha_commit_trans (thd=0x7fff78000d50, all=false) at /home/midenok/src/mariadb/midenok/src/sql/handler.cc:1406
#11 0x0000555555ccc9d0 in trans_commit_stmt (thd=0x7fff78000d50) at /home/midenok/src/mariadb/midenok/src/sql/transaction.cc:513
#12 0x0000555555b685bf in mysql_execute_command (thd=0x7fff78000d50) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:6276
#13 0x0000555555b6c620 in mysql_parse (thd=0x7fff78000d50, rawbuf=0x7fff78011278 "alter table t1 change y z int", length=29, parser_state=0x7ffff40576a0, is_com_multi=false, is_next_command=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:7923

(gdb) f 4
#4  0x00005555561d6540 in row_upd_index_entry_sys_field (entry=0x7fff780d7fb8, index=0x5555581ad818, type=1, val=0) at /home/midenok/src/mariadb/midenok/src/storage/innobase/row/row0upd.cc:511
511                     ut_ad(val > 0);
(gdb) p val
$1 = 0
(gdb) f 5
#5  0x000055555615bf95 in vers_row_ins_vtq_low (trx=0x7fffefbffe40, heap=0x7fff780d7e00, tuple=0x7fff780d7e88) at /home/midenok/src/mariadb/midenok/src/storage/innobase/row/row0ins.cc:3941
3941            row_upd_index_entry_sys_field(entry, index, DATA_TRX_ID, trx->id);
(gdb) p trx->id
$3 = 0

midenok commented 7 years ago

1. `vtq_notify_on_commit` set

#0  row_insert_for_mysql (mysql_rec=0x7fff80348a58 "\377\b\267\005", prebuilt=0x7fff80349878, ins_mode=ROW_INS_VERSIONED) at /home/midenok/src/mariadb/midenok/src/storage/innobase/row/row0mysql.cc:1563
#1  0x0000555556022e60 in ha_innobase::write_row (this=0x7fff80348258, record=0x7fff80348a58 "\377\b\267\005") at /home/midenok/src/mariadb/midenok/src/storage/innobase/handler/ha_innodb.cc:8727
#2  0x0000555555e12437 in handler::ha_write_row (this=0x7fff80348258, buf=0x7fff80348a58 "\377\b\267\005") at /home/midenok/src/mariadb/midenok/src/sql/handler.cc:6016
#3  0x0000555555dba58c in VTMD_table::update (this=0x7ffff4054290, thd=0x7fff80000d50, archive_name=0x0, col_map=false) at /home/midenok/src/mariadb/midenok/src/sql/vtmd.cc:261
#4  0x0000555555c34d32 in mysql_inplace_alter_table (thd=0x7fff80000d50, table_list=0x7fff80011360, table=0x7fff800d9790, altered_table=0x7fff80336450, ha_alter_info=0x7ffff4054aa0, inplace_supported=HA_ALTER_INPLACE_NO_LOCK_AFTER_PREPARE, target_mdl_request=0x7ffff4054b10, alter_ctx=0x7ffff40553a0, vers_update_vtmd=true, vers_archive_name=0x0) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:7426
#5  0x0000555555c39fa9 in mysql_alter_table (thd=0x7fff80000d50, new_db=0x7fff80011990 "test", new_name=0x0, create_info=0x7ffff40564c0, table_list=0x7fff80011360, alter_info=0x7ffff4056410, order_num=0, order=0x0, ignore=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:9387
#6  0x0000555555cbc6e4 in Sql_cmd_alter_table::execute (this=0x7fff80011a90, thd=0x7fff80000d50) at /home/midenok/src/mariadb/midenok/src/sql/sql_alter.cc:318
#7  0x0000555555b68228 in mysql_execute_command (thd=0x7fff80000d50) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:6230
#8  0x0000555555b6c620 in mysql_parse (thd=0x7fff80000d50, rawbuf=0x7fff80011278 "alter table t1 change y z int", length=29, parser_state=0x7ffff40576a0, is_com_multi=false, is_next_command=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:7923

(gdb) p trx->id
$4 = 374536
(gdb) p trx
$5 = (trx_t *) 0x7fffefbffe40

2. `trx->id` cleared

#0  trx_init (trx=0x7fffefbffe40) at /home/midenok/src/mariadb/midenok/src/storage/innobase/trx/trx0trx.cc:134
#1  0x0000555556245a94 in trx_commit_in_memory (trx=0x7fffefbffe40, mtr=0x7ffff4052cc0, serialised=true) at /home/midenok/src/mariadb/midenok/src/storage/innobase/trx/trx0trx.cc:1912
#2  0x0000555556245ecc in trx_commit_low (trx=0x7fffefbffe40, mtr=0x7ffff4052cc0) at /home/midenok/src/mariadb/midenok/src/storage/innobase/trx/trx0trx.cc:2007
#3  0x0000555556245f8e in trx_commit (trx=0x7fffefbffe40) at /home/midenok/src/mariadb/midenok/src/storage/innobase/trx/trx0trx.cc:2031
#4  0x0000555556246781 in trx_commit_for_mysql (trx=0x7fffefbffe40) at /home/midenok/src/mariadb/midenok/src/storage/innobase/trx/trx0trx.cc:2250
#5  0x000055555606edd2 in ha_innobase::commit_inplace_alter_table (this=0x7fff800da438, altered_table=0x7fff80336450, ha_alter_info=0x7ffff4054aa0, commit=true) at /home/midenok/src/mariadb/midenok/src/storage/innobase/handler/handler0alter.cc:8857
#6  0x0000555555e0de3d in handler::ha_commit_inplace_alter_table (this=0x7fff800da438, altered_table=0x7fff80336450, ha_alter_info=0x7ffff4054aa0, commit=true) at /home/midenok/src/mariadb/midenok/src/sql/handler.cc:4272
#7  0x0000555555c34d9a in mysql_inplace_alter_table (thd=0x7fff80000d50, table_list=0x7fff80011360, table=0x7fff800d9790, altered_table=0x7fff80336450, ha_alter_info=0x7ffff4054aa0, inplace_supported=HA_ALTER_INPLACE_NO_LOCK_AFTER_PREPARE, target_mdl_request=0x7ffff4054b10, alter_ctx=0x7ffff40553a0, vers_update_vtmd=true, vers_archive_name=0x0) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:7432
#8  0x0000555555c39fa9 in mysql_alter_table (thd=0x7fff80000d50, new_db=0x7fff80011990 "test", new_name=0x0, create_info=0x7ffff40564c0, table_list=0x7fff80011360, alter_info=0x7ffff4056410, order_num=0, order=0x0, ignore=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_table.cc:9387
#9  0x0000555555cbc6e4 in Sql_cmd_alter_table::execute (this=0x7fff80011a90, thd=0x7fff80000d50) at /home/midenok/src/mariadb/midenok/src/sql/sql_alter.cc:318
#10 0x0000555555b68228 in mysql_execute_command (thd=0x7fff80000d50) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:6230
#11 0x0000555555b6c620 in mysql_parse (thd=0x7fff80000d50, rawbuf=0x7fff80011278 "alter table t1 change z y int", length=29, parser_state=0x7ffff40576a0, is_com_multi=false, is_next_command=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:7923

3. `vers_notify_vtq()` called

#3  0x00007ffff6442fc2 in __GI___assert_fail (assertion=0x5555568be991 "val > 0", file=0x5555568be890 "/home/midenok/src/mariadb/midenok/src/storage/innobase/row/row0upd.cc", line=511, function=0x5555568bffc0 <row_upd_index_entry_sys_field(dtuple_t*, dict_index_t*, unsigned long, unsigned long)::__PRETTY_FUNCTION__> "void row_upd_index_entry_sys_field(dtuple_t*, dict_index_t*, ulint, ib_uint64_t)") at assert.c:101
#4  0x00005555561d6540 in row_upd_index_entry_sys_field (entry=0x7fff800d7fb8, index=0x5555581ac078, type=1, val=0) at /home/midenok/src/mariadb/midenok/src/storage/innobase/row/row0upd.cc:511
#5  0x000055555615bf95 in vers_row_ins_vtq_low (trx=0x7fffefbffe40, heap=0x7fff800d7e00, tuple=0x7fff800d7e88) at /home/midenok/src/mariadb/midenok/src/storage/innobase/row/row0ins.cc:3941
#6  0x000055555615c318 in vers_notify_vtq (trx=0x7fffefbffe40) at /home/midenok/src/mariadb/midenok/src/storage/innobase/row/row0ins.cc:4003
#7  0x000055555601b42c in innobase_commit (hton=0x555557b3e7d0, thd=0x7fff80000d50, commit_trx=false) at /home/midenok/src/mariadb/midenok/src/storage/innobase/handler/ha_innodb.cc:4846
#8  0x0000555555e06633 in commit_one_phase_2 (thd=0x7fff80000d50, all=false, trans=0x7fff80004040, is_real_trans=true) at /home/midenok/src/mariadb/midenok/src/sql/handler.cc:1558
#9  0x0000555555e06513 in ha_commit_one_phase (thd=0x7fff80000d50, all=false) at /home/midenok/src/mariadb/midenok/src/sql/handler.cc:1539
#10 0x0000555555e05e19 in ha_commit_trans (thd=0x7fff80000d50, all=false) at /home/midenok/src/mariadb/midenok/src/sql/handler.cc:1406
#11 0x0000555555ccc9d0 in trans_commit_stmt (thd=0x7fff80000d50) at /home/midenok/src/mariadb/midenok/src/sql/transaction.cc:513
#12 0x0000555555b685bf in mysql_execute_command (thd=0x7fff80000d50) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:6276
#13 0x0000555555b6c620 in mysql_parse (thd=0x7fff80000d50, rawbuf=0x7fff80011278 "alter table t1 change z y int", length=29, parser_state=0x7ffff40576a0, is_com_multi=false, is_next_command=false) at /home/midenok/src/mariadb/midenok/src/sql/sql_parse.cc:7923

midenok commented 7 years ago

Fixed by adding vers_notify_vtq() for commit_inplace_alter_table().

midenok commented 7 years ago

MyISAM uses INPLACE only when column type + count + order is kept intact. I.e. only on column rename it calls mysql_inplace_alter_table(), but in fact it does nothing (MyISAM handler have no inplace iface): field names are not engine-specific.

MyISAM uses INPLACE on:

name change.

InnoDB uses INPLACE on:

order change;
name change.

midenok commented 7 years ago

Handle rename

There is FIELD_IS_RENAMED flag. Detection of field name mappings is done this way (ha_innobase::prepare_inplace_alter_table()):

        for (Field** fp = table->field; *fp; fp++) {
            if (!((*fp)->flags & FIELD_IS_RENAMED)) {
                continue;
            }

            const char* name = 0;

            cf_it.rewind();
            while (Create_field* cf = cf_it++) {
                if (cf->field == *fp) {
                    name = cf->field_name;
                    goto check_if_ok_to_rename;
                }
            }

name will contain new name; fp->field_name: old name. There is also innobase_get_col_names().

Handle order

In mysql_prepare_alter_table() it reorders new_create_list:

    if (!def->after)
      new_create_list.push_back(def, thd->mem_root);
    else
    {
      Create_field *find;
      if (def->change)
      {
        find_it.rewind();
        /*
          For columns being modified with AFTER clause we should first remove
          these columns from the list and then add them back at their correct
          positions.
        */
        while ((find=find_it++))
        {
          /*
            Create_fields representing changed columns are added directly
            from Alter_info::create_list to new_create_list. We can therefore
            safely use pointer equality rather than name matching here.
            This prevents removing the wrong column in case of column rename.
          */
          if (find == def)
          {
            find_it.remove();
            break;
          }
        }
      }
      if (def->after == first_keyword)
        new_create_list.push_front(def, thd->mem_root);
      else
      {
        find_it.rewind();
        while ((find=find_it++))
        {
          if (!my_strcasecmp(system_charset_info, def->after, find->field_name))
            break;
        }
        if (!find)
        {
          my_error(ER_BAD_FIELD_ERROR, MYF(0), def->after, table->s->table_name.str);
          goto err;
        }
        find_it.after(def);         // Put column after this
      }
    }

Then it does:

  alter_info->create_list.swap(new_create_list);

tempesta-tech / mariadb

VTMD column names tracking (CMMD) (4w) #205

TODO

VTMD ctor on rename

How it works now

1. detect if we need archive table (vers_data_mod flag).

2. set backup table name to archive table

3. put info into VTMD

4. skip deleting backup table

vtmd.test fails

Wrong field name on SELECT

Tables opened at connect

Table opened at CREATE TABLE

frame 1

1. tmp table create started

2. table->file assigned

3. table->file->table assigned

1. vtq_notify_on_commit set

2. trx->id cleared

3. vers_notify_vtq() called

Handle rename

Handle order

1. detect if we need archive table (`vers_data_mod` flag).

`vtmd.test` fails

Table opened at `CREATE TABLE`

2. `table->file` assigned

3. `table->file->table` assigned

1. `vtq_notify_on_commit` set

2. `trx->id` cleared

3. `vers_notify_vtq()` called