tavplubix
commented
1 year ago Looks like duplicate of https://github.com/ClickHouse/ClickHouse/issues/41409
Closed acris5 closed 1 year ago
tavplubix
commented
1 year ago Looks like duplicate of https://github.com/ClickHouse/ClickHouse/issues/41409
acris5
commented
1 year ago Is here any quilck fix for this issue? May be manualy droping or creating some files? Or only full database recover from backup can help?
tavplubix
commented
1 year ago Find an empty mutation_X.txt file in /var/lib/clickhouse/store/38d/38d451af-fb28-4bb9-b5e3-b4f83089e8c3 dir and remove the file.
acris5
commented
1 year ago @tavplubix tnx it worked, db started finally after deleting 3 empty mutation files. Question is why clickhouse-server restarted because of large query, I thought there were query memory and time limits to protect from that?
2022.12.14 14:01:48.278001 [ 644537 ] {b4fd321a-be3a-4b91-b9e1-7e996fcd0667} <Error> executeQuery: Code: 24. DB::Exception: Cannot write to ostream at offset 739451980: While executing Native. (CANNOT_WRITE_TO_OSTREAM) (version 22.7.3.5 (official build)) (from 192.168.10.230:2727) (in query: select calculator_name, calculator_uuid, date, datetime, ts_nano, app_id, time_ram, data_time, eq_uuid, asset_uuid, phase, u_hv, qd_u_hv,u1_ccvt, qd_u1_ccvt, i1_ccvt, qd_i1_ccvt, fi_ui1_ccvt, qd_fi_ui1_ccvt, freq_cctv, qd_freq_ccvt,ir_ccvt, qd_ir_ccvt, ic_ccvt, qd_ic_ccvt, c_ccvt, qd_c_ccvt, tg_ccvt, qd_tg_ccvt,ccvt_on_off, qd_ccvt_on_off, time_on_off, t_air, qd_t_air, humidity_air, qd_humidity_air from ccvt_params_0 where ts_nano > 1667088000 order by ts_nano asc), Stack trace (when copying this message, always include the lines below):
2022.12.14 14:01:48.413756 [ 644537 ] {b4fd321a-be3a-4b91-b9e1-7e996fcd0667} <Error> DynamicQueryHandler: Code: 24. DB::Exception: Cannot write to ostream at offset 739451980: While executing Native. (CANNOT_WRITE_TO_OSTREAM), Stack trace (when copying this message, always include the lines below):
2022.12.14 14:01:48.509030 [ 644537 ] {b4fd321a-be3a-4b91-b9e1-7e996fcd0667} <Error> DynamicQueryHandler: Cannot send exception to client: Code: 24. DB::Exception: Cannot write to ostream at offset 738403716. (CANNOT_WRITE_TO_OSTREAM), Stack trace (when copying this message, always include the lines below):
2022.12.14 14:01:48.879227 [ 644537 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 32, I/O error: Broken pipe, Stack trace (when copying this message, always include the lines below):
2022.12.14 14:09:27.864531 [ 646470 ] {5d1b6de8-8a52-42e6-81f8-84fe256799a7} <Error> executeQuery: Code: 24. DB::Exception: Cannot write to ostream at offset 1152578610: While executing Native. (CANNOT_WRITE_TO_OSTREAM) (version 22.7.3.5 (official build)) (from 192.168.10.230:2924) (in query: select calculator_name, calculator_uuid, date, datetime, ts_nano, app_id, time_ram, data_time, eq_uuid, asset_uuid, phase, u_hv, qd_u_hv,u1_ccvt, qd_u1_ccvt, i1_ccvt, qd_i1_ccvt, fi_ui1_ccvt, qd_fi_ui1_ccvt, freq_cctv, qd_freq_ccvt,ir_ccvt, qd_ir_ccvt, ic_ccvt, qd_ic_ccvt, c_ccvt, qd_c_ccvt, tg_ccvt, qd_tg_ccvt,ccvt_on_off, qd_ccvt_on_off, time_on_off, t_air, qd_t_air, humidity_air, qd_humidity_air from ccvt_params_0 where ts_nano > 1667088000 order by ts_nano asc), Stack trace (when copying this message, always include the lines below):
2022.12.14 14:09:28.001010 [ 646470 ] {5d1b6de8-8a52-42e6-81f8-84fe256799a7} <Error> DynamicQueryHandler: Code: 24. DB::Exception: Cannot write to ostream at offset 1152578610: While executing Native. (CANNOT_WRITE_TO_OSTREAM), Stack trace (when copying this message, always include the lines below):
2022.12.14 14:09:28.001137 [ 646470 ] {5d1b6de8-8a52-42e6-81f8-84fe256799a7} <Error> DynamicQueryHandler: Cannot send exception to client: Code: 24. DB::Exception: Cannot write to ostream at offset 1151530348. (CANNOT_WRITE_TO_OSTREAM), Stack trace (when copying this message, always include the lines below):
2022.12.14 14:09:28.001655 [ 646470 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 32, I/O error: Broken pipe, Stack trace (when copying this message, always include the lines below):
2022.12.14 14:18:17.633916 [ 646470 ] {4afb7352-ae44-425c-a0b4-fbdcb60575a3} <Error> executeQuery: Code: 24. DB::Exception: Cannot write to ostream at offset 677870882: While executing Native. (CANNOT_WRITE_TO_OSTREAM) (version 22.7.3.5 (official build)) (from 192.168.10.230:3269) (in query: select calculator_name, calculator_uuid, date, datetime, ts_nano, app_id, time_ram, data_time, eq_uuid, asset_uuid, phase, u_hv, qd_u_hv,u1_ccvt, qd_u1_ccvt, i1_ccvt, qd_i1_ccvt, fi_ui1_ccvt, qd_fi_ui1_ccvt, freq_cctv, qd_freq_ccvt,ir_ccvt, qd_ir_ccvt, ic_ccvt, qd_ic_ccvt, c_ccvt, qd_c_ccvt, tg_ccvt, qd_tg_ccvt,ccvt_on_off, qd_ccvt_on_off, time_on_off, t_air, qd_t_air, humidity_air, qd_humidity_air from ccvt_params_0 where ts_nano > 1667088000 order by ts_nano asc), Stack trace (when copying this message, always include the lines below):
2022.12.14 14:18:17.726725 [ 646470 ] {4afb7352-ae44-425c-a0b4-fbdcb60575a3} <Error> DynamicQueryHandler: Code: 24. DB::Exception: Cannot write to ostream at offset 677870882: While executing Native. (CANNOT_WRITE_TO_OSTREAM), Stack trace (when copying this message, always include the lines below):
2022.12.14 14:18:17.726840 [ 646470 ] {4afb7352-ae44-425c-a0b4-fbdcb60575a3} <Error> DynamicQueryHandler: Cannot send exception to client: Code: 24. DB::Exception: Cannot write to ostream at offset 676822618. (CANNOT_WRITE_TO_OSTREAM), Stack trace (when copying this message, always include the lines below):
2022.12.14 14:18:17.742934 [ 646470 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 32, I/O error: Broken pipe, Stack trace (when copying this message, always include the lines below):
2022.12.14 14:20:16.168114 [ 646470 ] {6635423b-295e-4d6c-824c-135354ce3eb6} <Error> executeQuery: Code: 24. DB::Exception: Cannot write to ostream at offset 100760255: While executing Native. (CANNOT_WRITE_TO_OSTREAM) (version 22.7.3.5 (official build)) (from 192.168.10.230:3365) (in query: select calculator_name, calculator_uuid, date, datetime, ts_nano, app_id, time_ram, data_time, eq_uuid, asset_uuid, phase, u_hv, qd_u_hv,u1_ccvt, qd_u1_ccvt, i1_ccvt, qd_i1_ccvt, fi_ui1_ccvt, qd_fi_ui1_ccvt, freq_cctv, qd_freq_ccvt,ir_ccvt, qd_ir_ccvt, ic_ccvt, qd_ic_ccvt, c_ccvt, qd_c_ccvt, tg_ccvt, qd_tg_ccvt,ccvt_on_off, qd_ccvt_on_off, time_on_off, t_air, qd_t_air, humidity_air, qd_humidity_air from ccvt_params_0 where ts_nano > 1667088000 order by ts_nano asc), Stack trace (when copying this message, always include the lines below):
2022.12.14 14:20:16.259929 [ 646470 ] {6635423b-295e-4d6c-824c-135354ce3eb6} <Error> DynamicQueryHandler: Code: 24. DB::Exception: Cannot write to ostream at offset 100760255: While executing Native. (CANNOT_WRITE_TO_OSTREAM), Stack trace (when copying this message, always include the lines below):
2022.12.14 14:20:16.260056 [ 646470 ] {6635423b-295e-4d6c-824c-135354ce3eb6} <Error> DynamicQueryHandler: Cannot send exception to client: Code: 24. DB::Exception: Cannot write to ostream at offset 99711991. (CANNOT_WRITE_TO_OSTREAM), Stack trace (when copying this message, always include the lines below):
2022.12.14 14:20:16.260256 [ 646470 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 32, I/O error: Broken pipe, Stack trace (when copying this message, always include the lines below):
2022.12.14 14:20:58.228586 [ 646470 ] {2585c448-c8bc-4318-925c-68e5b662e1f8} <Error> executeQuery: Code: 24. DB::Exception: Cannot write to ostream at offset 44422037: While executing Native. (CANNOT_WRITE_TO_OSTREAM) (version 22.7.3.5 (official build)) (from 192.168.10.230:3394) (in query: select calculator_name, calculator_uuid, date, datetime, ts_nano, app_id, time_ram, data_time, eq_uuid, asset_uuid, phase, u_hv, qd_u_hv,u1_ccvt, qd_u1_ccvt, i1_ccvt, qd_i1_ccvt, fi_ui1_ccvt, qd_fi_ui1_ccvt, freq_cctv, qd_freq_ccvt,ir_ccvt, qd_ir_ccvt, ic_ccvt, qd_ic_ccvt, c_ccvt, qd_c_ccvt, tg_ccvt, qd_tg_ccvt,ccvt_on_off, qd_ccvt_on_off, time_on_off, t_air, qd_t_air, humidity_air, qd_humidity_air from ccvt_params_0 where ts_nano > 1667088000 order by ts_nano asc), Stack trace (when copying this message, always include the lines below):
2022.12.14 14:20:58.318107 [ 646470 ] {2585c448-c8bc-4318-925c-68e5b662e1f8} <Error> DynamicQueryHandler: Code: 24. DB::Exception: Cannot write to ostream at offset 44422037: While executing Native. (CANNOT_WRITE_TO_OSTREAM), Stack trace (when copying this message, always include the lines below):
2022.12.14 14:20:58.318223 [ 646470 ] {2585c448-c8bc-4318-925c-68e5b662e1f8} <Error> DynamicQueryHandler: Cannot send exception to client: Code: 24. DB::Exception: Cannot write to ostream at offset 43373771. (CANNOT_WRITE_TO_OSTREAM), Stack trace (when copying this message, always include the lines below):
2022.12.14 14:20:58.318434 [ 646470 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 32, I/O error: Broken pipe, Stack trace (when copying this message, always include the lines below):
2022.12.14 14:29:23.196905 [ 646510 ] {} <Error> DynamicQueryHandler: Code: 373. DB::Exception: Session is locked by a concurrent client. (SESSION_IS_LOCKED), Stack trace (when copying this message, always include the lines below):
2022.12.14 14:30:05.955452 [ 646510 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 32, I/O error: Broken pipe, Stack trace (when copying this message, always include the lines below):
2022.12.14 14:31:29.095059 [ 646510 ] {2aa5e119-7031-467e-9f5a-64a3b8f189e0} <Error> DynamicQueryHandler: Cannot flush data to client: Code: 24. DB::Exception: Cannot write to ostream at offset 170. (CANNOT_WRITE_TO_OSTREAM), Stack trace (when copying this message, always include the lines below):
2022.12.14 14:32:05.472556 [ 646511 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 32, I/O error: Broken pipe, Stack trace (when copying this message, always include the lines below):
2022.12.14 14:32:35.144246 [ 646510 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 32, I/O error: Broken pipe, Stack trace (when copying this message, always include the lines below):
2022.12.14 14:33:01.385107 [ 646499 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 32, I/O error: Broken pipe, Stack trace (when copying this message, always include the lines below):
alexey-milovidov
commented
1 year ago You can check the logs to see why the server was restarted. An empty file on the local filesystem usually indicates an unclean restart of the machine with clickhouse-server, so the filesystem changes were not committed.
We have production server with 2Tb of data, once someone tried to Select large table without limits in query, and server restarted unexpectedly and can't start now with following error: