bug: After upgrading from Stonedb5.7.100 to the Stonedb5.7.101 test version, the old data cannot be read

[adofsauron]

Describe the problem

After upgrading to the stonedb5.7.101 test version, keep the old data and execute the query

DELIMITER $$
CREATE FUNCTION get_value(id INT) RETURNS INT
BEGIN
  DECLARE FINALVAR INT;
  SELECT employee_salary INTO FINALVAR FROM employees WHERE employee_id=id;
RETURN FINALVAR;
END $$
DELIMITER ;

CREATE TABLE `employees` (
  `employee_id` int(11) NOT NULL AUTO_INCREMENT,
  `employee_name` varchar(50) NOT NULL,
  `employee_sex` varchar(10) DEFAULT '男',
  `hire_date` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `employee_mgr` int(11) DEFAULT NULL,
  `employee_salary` float DEFAULT '3000',
  `department_id` int(11) DEFAULT NULL,
  PRIMARY KEY (`employee_id`),
  KEY `idx_department_id` (`department_id`)
) ENGINE=TIANMU DEFAULT CHARSET=utf8mb4;

insert into employees(employee_name) values('David Tian');

mysql>     select
    ->         a.employee_id,
    ->         get_value(b.employee_id)
    ->     from
    ->         employees a
    ->     left join employees b on
    ->         a.employee_id = b.employee_id
    ->         and b.employee_name = 'David Tian';
ERROR 1030 (HY000): Got error 1 from storage engine

An error occurred when viewing the log

[2022-09-18 04:52:27.387429] [30194] [WARN] [exception.cpp:41] MSG: Exception: assert failed on sb.st_size % sizeof(DPN) == 0 at column_share.cpp:83, msg: [].

STACK TRACE BEGIN
        /stonedb57/install//bin/mysqld(Tianmu::common::AssertException::AssertException(char const*, char const*, int, std::string const&)+0x1b7) [0x2c96f53]
        /stonedb57/install//bin/mysqld(Tianmu::core::ColumnShare::map_dpn()+0x276) [0x2cbf5ca]
        /stonedb57/install//bin/mysqld(Tianmu::core::ColumnShare::Init(Tianmu::common::TX_ID)+0x1c) [0x2cbf332]
        /stonedb57/install//bin/mysqld(Tianmu::core::ColumnShare::ColumnShare(Tianmu::core::TableShare*, Tianmu::common::TX_ID, unsigned int, std::experimental::filesystem::v1::path const&, Field const*)+0x14c) [0x2a74582]
        /stonedb57/install//bin/mysqld(std::_MakeUniq<Tianmu::core::ColumnShare>::__single_object std::make_unique<Tianmu::core::ColumnShare, Tianmu::core::TableShare*, Tianmu::common::TX_ID&, unsigned int&, std::experimental::filesystem::v1::path, Field*&>(Tianmu::core::TableShare*&&, Tianmu::common::TX_ID&, unsigned int&, std::experimental::filesystem::v1::path&&, Field*&)+0xa0) [0x2a75d30]
        /stonedb57/install//bin/mysqld(Tianmu::core::TableShare::TableShare(std::experimental::filesystem::v1::path const&, TABLE_SHARE const*)+0x6a0) [0x2a74cd2]
        /stonedb57/install//bin/mysqld(void __gnu_cxx::new_allocator<Tianmu::core::TableShare>::construct<Tianmu::core::TableShare, std::string, TABLE_SHARE const*&>(Tianmu::core::TableShare*, std::string&&, TABLE_SHARE const*&)+0x72) [0x29ca166]
        /stonedb57/install//bin/mysqld(_ZNSt16allocator_traitsISaIN6Tianmu4core10TableShareEEE9constructIS2_ISsRPK11TABLE_SHAREEEEvRS3_PT_DpOT0_+0x4a) [0x29c8adc]
        /stonedb57/install//bin/mysqld(std::_Sp_counted_ptr_inplace<Tianmu::core::TableShare, std::allocator<Tianmu::core::TableShare>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<std::string, TABLE_SHARE const*&>(std::allocator<Tianmu::core::TableShare>, std::string&&, TABLE_SHARE const*&)+0xae) [0x29c654e]
        /stonedb57/install//bin/mysqld(std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<Tianmu::core::TableShare, std::allocator<Tianmu::core::TableShare>, std::string, TABLE_SHARE const*&>(std::_Sp_make_shared_tag, Tianmu::core::TableShare*, std::allocator<Tianmu::core::TableShare> const&, std::string&&, TABLE_SHARE const*&)+0xcb) [0x29c23ed]
        /stonedb57/install//bin/mysqld(std::__shared_ptr<Tianmu::core::TableShare, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<Tianmu::core::TableShare>, std::string, TABLE_SHARE const*&>(std::_Sp_make_shared_tag, std::allocator<Tianmu::core::TableShare> const&, std::string&&, TABLE_SHARE const*&)+0x6e) [0x29bc7c0]
        /stonedb57/install//bin/mysqld(std::shared_ptr<Tianmu::core::TableShare>::shared_ptr<std::allocator<Tianmu::core::TableShare>, std::string, TABLE_SHARE const*&>(std::_Sp_make_shared_tag, std::allocator<Tianmu::core::TableShare> const&, std::string&&, TABLE_SHARE const*&)+0x57) [0x29b4ac5]
        /stonedb57/install//bin/mysqld(std::shared_ptr<Tianmu::core::TableShare> std::allocate_shared<Tianmu::core::TableShare, std::allocator<Tianmu::core::TableShare>, std::string, TABLE_SHARE const*&>(std::allocator<Tianmu::core::TableShare> const&, std::string&&, TABLE_SHARE const*&)+0x52) [0x29ac7e9]
        /stonedb57/install//bin/mysqld(std::shared_ptr<Tianmu::core::TableShare> std::make_shared<Tianmu::core::TableShare, std::string, TABLE_SHARE const*&>(std::string&&, TABLE_SHARE const*&)+0x57) [0x29a387e]
        /stonedb57/install//bin/mysqld(Tianmu::core::Engine::GetTableShare(TABLE_SHARE const*)+0xe7) [0x299a289]
        /stonedb57/install//bin/mysqld(Tianmu::dbhandler::TianmuHandler::open(char const*, int, unsigned int)+0x63) [0x2d77c45]
        /stonedb57/install//bin/mysqld(handler::ha_open(TABLE*, char const*, int, int)+0x47) [0x1cde009]
        /stonedb57/install//bin/mysqld(open_table_from_share(THD*, TABLE_SHARE*, char const*, unsigned int, unsigned int, unsigned int, TABLE*, bool)+0xe50) [0x23c812e]
        /stonedb57/install//bin/mysqld(open_table(THD*, TABLE_LIST*, Open_table_contex
[2022-09-18 04:52:27.387556] [30194] [ERROR] [engine.cpp:1876] MSG: Failed to create table share: assert failed on sb.st_size % sizeof(DPN) == 0 at column_share.cpp:83, msg: []

Expected behavior

No response

How To Reproduce

No response

Environment

No response

Are you interested in submitting a PR to solve the problem?

• [ ] Yes, I will!

[adofsauron]

If the binary data is not compatible after the upgrade, the error message needs to be more explicit than just a prompt (Got error 1 from storage engine).

[konghaiya]

I don't think it is necessary to be compatible with the data of the old version for the following reasons:

Compatibility with the data of the old version requires the addition of many version compatible codes. These codes will not only affect the system performance, but also will be useless in the later versions that run normally, which will cause a lot of performance waste.

I think we can give an important explanation of this problem when releasing the version, and remind users to backup data when upgrading the version.

[RingsC]

we should have tools to support seamless upgrade mannually, if we cannot do these automatically.

[adofsauron]

Yes, we can refer to the Redis tool for reading and verifying binary data files

[root@zsl redis]# redis-check-rdb  ./dump.rdb 
[offset 0] Checking RDB file ./dump.rdb
[offset 27] AUX FIELD redis-ver = '3.2.12'
[offset 41] AUX FIELD redis-bits = '64'
[offset 53] AUX FIELD ctime = '1663566107'
[offset 68] AUX FIELD used-mem = '812464'
[offset 70] Selecting DB ID 0
[offset 89] Checksum OK
[offset 89] \o/ RDB looks OK! \o/
[info] 1 keys read
[info] 0 expires
[info] 0 already expired

[konghaiya]

Upgrade scheme of mainstream database

MySQL

Upgrade method	Applicable scenarios	Time required
Logical backup and restore (mysqldump)	When the data volume is small, this method usually does not failIt is used more when upgrading across major versions, for example, upgrading directly from MySQL 5.6 (or earlier) to version 8.0 Upgrade between different MySQL branches	Depends on the size of the database
mysql_upgrade tool	Minor version upgrade between the same large versions. There is a large amount of data, which can be used when upgrading between two versions after testing in advance.	A few minutes

PostgreSQL

Upgrade method	Applicable scenarios	Time required
Logical backup and restore	Small and medium-sized databases, such as less than 100 GB Support cross platform data migration	Depends on the size of the database
pg_upgrade tool	Medium and large databases, such as less than 100 GB Support cross platform data migration	A few minutes
Logical replication	Medium and large databases, such as less than 100 GB Support cross platform data migration	A few minutes

Mongodb

Upgrade method	Applicable scenarios	Time required
Logical backup and restore (mongodump/mongorestore)	The advantage of this scheme is that it can be migrated and upgraded across multiple versions. The disadvantage is that the data migration speed is slow.	Depends on the size of the database
Physical upgrade	Small version upgrades between the same large versions cannot be performed across multiple large versions. There is a large amount of data, which can be used when upgrading between two versions after testing in advance..	A few minutes

other

The upgrade scheme of SQL Server, Oracle and other data is similar to the above three.

The upgrade scheme of StoneDB

Scheme 1: Data backup and restore (mysqldump)

Advantages: high universality, simple operation, cross system and cross version upgrades Disadvantages: The time required for a large data volume scenario is high. Applicable scenario: It is suitable for upgrading small and medium-sized databases. Currently supported by StoneDB.

Scheme 2: provide tianmu_upgrade tool (upgrade in place)

Use the original mysql_upgrade tool is used to upgrade in place, but the data about the Tianmu engine needs to be provided with the (tianmu_upgrade) tool to upgrade in situ. Not yet supported, to be developed...

Capabilities to be provided by the tool:

1. Scan the version to determine the items to be upgraded.
2. Compatibility check to determine incompatible items.
3. Provide the ability to upgrade metadata files, such as DN (DPN), META, TABLE_ DESC, V, etc. Each file has a corresponding independent upgrade program, which is managed by the (tianmu_upgrade) script.
4. It supports abnormal rollback. When all items have been upgraded and checked before committing, the old data can be deleted/or the old data can be retained.

Scheme 3: Provide physical replication function

When the system is multi node, it can be upgraded through physical replication, which needs to support cross version data replication. Not yet supported, to be developed...

[adofsauron]

Influxdb has faced this issue before,

initially using the RockSDB data format as binary data files, and later writing their own TSM engine to reformat binary data files, they have written work to reformat binary files

https://docs.influxdata.com/influxdb/v1.8/concepts/storage_engine/