search

oxidecomputer / omicron

Omicron: Oxide control plane
Mozilla Public License 2.0
252 stars 40 forks source link

ClickHouse schema updates still use the HTTP client #7015

Closed bnaecker closed 3 weeks ago

bnaecker commented 3 weeks ago

On dogfood, folks noted the ClickHouse schema updater not completing:

root@oxz_oximeter_da510a57:~# /opt/oxide/oximeter-collector/bin/clickhouse-schema-updater --native-host [fd00:1122:3344:107::4]:9000 up
Nov 07 22:37:10.932 WARN skipping non-SQL schema dir entry, path: /opt/oxide/oximeter/schema/single-node/13/timeseries-to-delete.txt, dir: /opt/oxide/oximeter/schema/single-node/13, id: 3d48cb5b-abb6-4396-ae3c-1dc521704a9a, component: clickhouse-client, unit: clickhouse_schema_updater
Nov 07 22:37:10.942 WARN failed to delete some timeseries, retry_after: 281.062µs, call_count: 0, error: DatabaseUnavailable("error sending request for url (http://[::1]:8123/?output_format_json_quote_64bit_integers=0&wait_end_of_query=1)"), id: 3d48cb5b-abb6-4396-ae3c-1dc521704a9a, component: clickhouse-client, unit: clickhouse_schema_updater
Nov 07 22:37:11.263 WARN failed to delete some timeseries, retry_after: 321.483829ms, call_count: 1, error: DatabaseUnavailable("error sending request for url (http://[::1]:8123/?output_format_json_quote_64bit_integers=0&wait_end_of_query=1)"), id: 3d48cb5b-abb6-4396-ae3c-1dc521704a9a, component: clickhouse-client, unit: clickhouse_schema_updater
Nov 07 22:37:11.628 WARN failed to delete some timeseries, retry_after: 686.67286ms, call_count: 2, error: DatabaseUnavailable("error sending request for url (http://[::1]:8123/?output_format_json_quote_64bit_integers=0&wait_end_of_query=1)"), id: 3d48cb5b-abb6-4396-ae3c-1dc521704a9a, component: clickhouse-client, unit: clickhouse_schema_updater
Nov 07 22:37:12.231 WARN failed to delete some timeseries, retry_after: 1.28890752s, call_count: 3, error: DatabaseUnavailable("error sending request for url (http://[::1]:8123/?output_format_json_quote_64bit_integers=0&wait_end_of_query=1)"), id: 3d48cb5b-abb6-4396-ae3c-1dc521704a9a, component: clickhouse-client, unit: clickhouse_schema_updater
Nov 07 22:37:13.306 WARN failed to delete some timeseries, retry_after: 2.36422718s, call_count: 4, error: DatabaseUnavailable("error sending request for url (http://[::1]:8123/?output_format_json_quote_64bit_integers=0&wait_end_of_query=1)"), id: 3d48cb5b-abb6-4396-ae3c-1dc521704a9a, component: clickhouse-client, unit: clickhouse_schema_updater
Nov 07 22:37:16.298 WARN failed to delete some timeseries, retry_after: 5.355674443s, call_count: 5, error: DatabaseUnavailable("error sending request for url (http://[::1]:8123/?output_format_json_quote_64bit_integers=0&wait_end_of_query=1)"), id: 3d48cb5b-abb6-4396-ae3c-1dc521704a9a, component: clickhouse-client, unit: clickhouse_schema_updater
Nov 07 22:37:26.470 WARN failed to delete some timeseries, retry_after: 15.528033963s, call_count: 6, error: DatabaseUnavailable("error sending request for url (http://[::1]:8123/?output_format_json_quote_64bit_integers=0&wait_end_of_query=1)"), id: 3d48cb5b-abb6-4396-ae3c-1dc521704a9a, component: clickhouse-client, unit: clickhouse_schema_updater
Nov 07 22:37:37.768 WARN failed to delete some timeseries, retry_after: 26.825412105s, call_count: 7, error: DatabaseUnavailable("error sending request for url (http://[::1]:8123/?output_format_json_quote_64bit_integers=0&wait_end_of_query=1)"), id: 3d48cb5b-abb6-4396-ae3c-1dc521704a9a, component: clickhouse-client, unit: clickhouse_schema_updater
Nov 07 22:37:56.224 WARN failed to delete some timeseries, retry_after: 45.28001898s, call_count: 8, error: DatabaseUnavailable("error sending request for url (http://[::1]:8123/?output_format_json_quote_64bit_integers=0&wait_end_of_query=1)"), id: 3d48cb5b-abb6-4396-ae3c-1dc521704a9a, component: clickhouse-client, unit: clickhouse_schema_updater

That's using the instructions I recently updated in the meta repo, where I left off the --host argument. I conflated the timeseries schema handling I added in #6943 with the database schema. The latter still uses the HTTP client. This tracks changing the database schema modifications to use the native client as well.

bnaecker commented 3 weeks ago

Ugh, this was even smaller than that. Most of the schema upgrade path uses the native client. The only exception is the small part of expunging old timeseries schema that reads the existing database tables to figure out what to scrub.