[Bug] (good first issue) The in_memory table attribute for table creation failed when upgrading from version 1.x to version 2.x, causing issues with dynamic partitioning. #43995
[X] I had searched in the issues and found no similar issues.
Version
2.1.6
What's Wrong?
Upgrade Process：1.2.4 -> 2.0.15 ->2.1.6
The table was created a long time ago with the in_memory = true parameter. After upgrading to the new version, the creation of new dynamic partitions failed. Upon investigation, it was found that this parameter was causing the issue.
we can force the in_memory parameter to be false. Sometimes, if partitions cannot be created, it leads to data not being able to be ingested, resulting in a series of problems
2024-11-04 12:22:03,459 INFO (DynamicPartitionScheduler|51) [PropertyAnalyzer.analyzeDataProperty():274] Can not assign cool down timestamp to HDD storage medium, ignore user setting.
2024-11-04 12:22:03,460 INFO (DynamicPartitionScheduler|51) [DynamicPartitionScheduler.recordCreatePartitionFailedMsg():627] dynamic add partition failed: errCode = 2, detailMessage = errCode = 2, detailMessage = Not support set 'in_memory'='true' now!, db: dim_blockchain, table: doris_dwm_eth_trans_num_180d
2024-11-04 12:22:03,460 WARN (DynamicPartitionScheduler|51) [DynamicPartitionScheduler.executeDynamicPartition():616] has error
at org.apache.doris.clone.DynamicPartitionScheduler.executeDynamicPartition(DynamicPartitionScheduler.java:612) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.clone.DynamicPartitionScheduler.runAfterCatalogReady(DynamicPartitionScheduler.java:678) ~[doris-fe.jar:1.2-SNAPSHOT]
What You Expected?
When upgrading from the old 1.x version where the table was created with in_memory = true, we ignore the in_memory parameter in the 2.x version so that it does not affect the dynamic partition creation
but #18731 compatibility upgrade issues were not considered
Fix Method：
To resolve this, we can modify the gsonPostProcess functions in both TableProperty.java and PartitionInfo.java to force the in_memory parameter to be false after a restart, and add a fe ut to check it
Search before asking
Version
2.1.6
What's Wrong?
Upgrade Process：1.2.4 -> 2.0.15 ->2.1.6
The table was created a long time ago with the in_memory = true parameter. After upgrading to the new version, the creation of new dynamic partitions failed. Upon investigation, it was found that this parameter was causing the issue. we can force the in_memory parameter to be false. Sometimes, if partitions cannot be created, it leads to data not being able to be ingested, resulting in a series of problems
What You Expected?
When upgrading from the old 1.x version where the table was created with in_memory = true, we ignore the in_memory parameter in the 2.x version so that it does not affect the dynamic partition creation
Anything Else?
https://github.com/apache/doris/pull/18731 This PR deprecates the in_memory parameter, which was removed in version 2.0
https://doris.apache.org/zh-CN/docs/3.0/sql-manual/sql-statements/Data-Definition-Statements/Create/CREATE-TABLE/ Documentation also reflects this deprecation status
but #18731 compatibility upgrade issues were not considered
Fix Method：
To resolve this, we can modify the gsonPostProcess functions in both TableProperty.java and PartitionInfo.java to force the in_memory parameter to be false after a restart, and add a fe ut to check it
How to Reproduce?
No response
Are you willing to submit PR?
Code of Conduct