ClickHouse / ClickHouse

ClickHouse® is a real-time analytics DBMS
https://clickhouse.com
Apache License 2.0
37.77k stars 6.93k forks source link

Load job "system.part_log" failed JSON exception: error: 1 unexpected end of data #65154

Open UnamedRus opened 5 months ago

UnamedRus commented 5 months ago

Describe what's wrong

broken serialization.json can prevent ClickHouse from starting, instead of marking part as broken

https://github.com/ClickHouse/ClickHouse/blob/bd45a366f7d42bd50dd90283c4992a69894ae674/src/Storages/MergeTree/IMergeTreeDataPart.cpp#L1605

CurtizJ commented 5 months ago

Which version do you use? I couldn't reproduce just by corrupting serialization.json file.

UnamedRus commented 5 months ago

24.3

UnamedRus commented 5 months ago

Unfortunately, i have only screenshot.

image

den-crane commented 5 months ago

it could be about ttl.txt

UnamedRus commented 5 months ago

Not sure about that,

As in that case stack trace should include TTL functions https://github.com/ClickHouse/ClickHouse/blob/bd45a366f7d42bd50dd90283c4992a69894ae674/src/Storages/MergeTree/IMergeTreeDataPart.cpp#L1508

den-crane commented 5 months ago

As in that case stack trace should include TTL functions

right,

Anyway, I did 50 different tests and found how to break serialization.json

24.3.2.23

create table x (a Int64, t DateTime) Engine=MergeTree order by a ttl t+interval 1 day  as select *,now() from numbers(1e3);

cat /var/lib/clickhouse/data/default/x/all_1_1_0/serialization.json
{"columns":[{"kind":"Default","name":"a","num_defaults":1,"num_rows":1000},{"kind":"Default","name":"t","num_defaults":0,"num_rows":1000}],"version":0}

-- let's cat >>> "version":0} <<<

echo '{"columns":[{"kind":"Default","name":"a","num_defaults":1,"num_rows":1000},{"kind":"Default","name":"t","num_defaults":0,"num_rows":1000}],' > /var/lib/clickhouse/data/default/x/all_1_1_0/serialization.json

systemctl restart clickhouse-server
Job for clickhouse-server.service failed because the control process exited with error code.
See "systemctl status clickhouse-server.service" and "journalctl -xe" for details.
den-crane commented 5 months ago

though, it does not reproduce with 24.5.3.5

lesandie commented 1 month ago

It seems this is already backported to 24.3

https://github.com/ClickHouse/ClickHouse/pull/69307#issuecomment-2343235909