search

GreptimeTeam / greptimedb

An open-source, cloud-native, unified time series database for metrics, logs and events with SQL/PromQL supported. Available on GreptimeCloud.
https://greptime.com/
Apache License 2.0
4.26k stars 307 forks source link

`COPY FROM` cannot coerce bigint to timestamp #2199

Closed v0y4g3r closed 1 year ago

v0y4g3r commented 1 year ago

What type of bug is this?

Unexpected error

What subsystems are affected?

Datanode

What happened?

When importing a parquet file to existing table, if the parquet file's timestamp type is bigint while the timestamp column in table is timestamp(whatever time unit), the COPY FROM statement will fail.

What operating system did you use?

NA

Relevant log output and stack trace

mysql> copy cpu from '/home/lei/part-0.parquet' with (format='parquet');
ERROR 1815 (HY000): Failed to execute query: copy cpu from '/home/lei/part-0.parquet' with (format='parquet'), source: Failed to insert value into table: cpu, source: Failed to operate table, source: Type of column ts does not match type in schema, expect Timestamp(Microsecond(TimestampMicrosecondType)), given Int64(Int64Type)

How can we reproduce the bug?

  1. create a table with timestamps

    create table cpu(
hostname varchar,
region varchar,
datacenter varchar,
rack varchar,
os varchar,
arch varchar,
team varchar,
service varchar,
service_version varchar,
service_environment varchar,
usage_user double,
usage_system double,
usage_idle double,
usage_nice double,
usage_iowait double,
usage_irq double,
usage_softirq double,
usage_steal double,
usage_guest double,
usage_guest_nice double,
ts timestamp(6),
primary key (hostname,region,datacenter,rack,os,arch,team,service,service_version,service_environment),
time index (ts)
);

  2. Create a sample file with bigint timestamp:

echo UEFSMRUEFRQVFEwVAhUAEgAABgAAAGhvc3RfMBUAFRAVECwVAhUQFQYVBhxYBmhvc3RfMBgGaG9zdF8wAAAAAgAAAAIBAAIVDBk1AAYQGRgIaG9zdG5hbWUVABYCFoYBFoYBJjgmCBxYBmhvc3RfMBgGaG9zdF8wAAAVBBUgFSBMFQIVABIAAAwAAABldS1jZW50cmFsLTEVABUQFRAsFQIVEBUGFQYcWAxldS1jZW50cmFsLTEYDGV1LWNlbnRyYWwtMQAAAAIAAAACAQACFQwZNQAGEBkYBnJlZ2lvbhUAFgIWqgEWqgEmsAIm9AEcWAxldS1jZW50cmFsLTEYDGV1LWNlbnRyYWwtMQAAFQQVIhUiTBUCFQASAAANAAAAZXUtY2VudHJhbC0xYRUAFRAVECwVAhUQFQYVBhxYDWV1LWNlbnRyYWwtMWEYDWV1LWNlbnRyYWwtMWEAAAACAAAAAgEAAhUMGTUABhAZGApkYXRhY2VudGVyFQAWAhawARawASbaBCacBBxYDWV1LWNlbnRyYWwtMWEYDWV1LWNlbnRyYWwtMWEAABUEFQoVCkwVAhUAEgAAAQAAADYVABUQFRAsFQIVEBUGFQYcWAE2GAE2AAAAAgAAAAIBAAIVDBk1AAYQGRgEcmFjaxUAFgIWaBZoJvwGJtYGHFgBNhgBNgAAFQQVHhUeTBUCFQASAAALAAAAVWJ1bnR1MTUuMTAVABUQFRAsFQIVEBUGFQYcWAtVYnVudHUxNS4xMBgLVWJ1bnR1MTUuMTAAAAACAAAAAgEAAhUMGTUABhAZGAJvcxUAFgIWpAEWpAEmwggmiAgcWAtVYnVudHUxNS4xMBgLVWJ1bnR1MTUuMTAAABUEFQ4VDkwVAhUAEgAAAwAAAHg4NhUAFRAVECwVAhUQFQYVBhxYA3g4NhgDeDg2AAAAAgAAAAIBAAIVDBk1AAYQGRgEYXJjaBUAFgIWdBZ0JsgKJp4KHFgDeDg2GAN4ODYAABUEFQwVDEwVAhUAEgAAAgAAAFNGFQAVEBUQLBUCFRAVBhUGHFgCU0YYAlNGAAAAAgAAAAIBAAIVDBk1AAYQGRgEdGVhbRUAFgIWbhZuJowMJuQLHFgCU0YYAlNGAAAVBBUMFQxMFQIVABIAAAIAAAAxORUAFRAVECwVAhUQFQYVBhxYAjE5GAIxOQAAAAIAAAACAQACFQwZNQAGEBkYB3NlcnZpY2UVABYCFm4WbibIDSagDRxYAjE5GAIxOQAAFQQVChUKTBUCFQASAAABAAAAMRUAFRAVECwVAhUQFQYVBhxYATEYATEAAAACAAAAAgEAAhUMGTUABhAZGA9zZXJ2aWNlX3ZlcnNpb24VABYCFmgWaCaIDybiDhxYATEYATEAABUEFRAVEEwVAhUAEgAABAAAAHRlc3QVABUQFRAsFQIVEBUGFQYcWAR0ZXN0GAR0ZXN0AAAAAgAAAAIBAAIVDBk1AAYQGRgTc2VydmljZV9lbnZpcm9ubWVudBUAFgIWehZ6JtYQJqoQHFgEdGVzdBgEdGVzdAAAFQQVEBUQTBUCFQASAAAAAAAAAABNQBUAFRAVECwVAhUQFQYVBhxYCAAAAAAAAE1AGAgAAAAAAABNQAAAAAIAAAACAQACFQoZNQAGEBkYCnVzYWdlX3VzZXIVABYCFooBFooBJsQSJpgSHFgIAAAAAAAATUAYCAAAAAAAAE1AAAAVBBUQFRBMFQIVABIAAAAAAAAAAABAFQAVEBUQLBUCFRAVBhUGHFgIAAAAAAAAAEAYCAAAAAAAAABAAAAAAgAAAAIBAAIVChk1AAYQGRgMdXNhZ2Vfc3lzdGVtFQAWAhaKARaKASbEFCaYFBxYCAAAAAAAAABAGAgAAAAAAAAAQAAAFQQVEBUQTBUCFQASAAAAAAAAAAA4QBUAFRAVECwVAhUQFQYVBhxYCAAAAAAAADhAGAgAAAAAAAA4QAAAAAIAAAACAQACFQoZNQAGEBkYCnVzYWdlX2lkbGUVABYCFooBFooBJsgWJpwWHFgIAAAAAAAAOEAYCAAAAAAAADhAAAAVBBUQFRBMFQIVABIAAAAAAAAAgE5AFQAVEBUQLBUCFRAVBhUGHFgIAAAAAACATkAYCAAAAAAAgE5AAAAAAgAAAAIBAAIVChk1AAYQGRgKdXNhZ2VfbmljZRUAFgIWigEWigEmyBgmnBgcWAgAAAAAAIBOQBgIAAAAAACATkAAABUEFRAVEEwVAhUAEgAAAAAAAAAANkAVABUQFRAsFQIVEBUGFQYcWAgAAAAAAAA2QBgIAAAAAAAANkAAAAACAAAAAgEAAhUKGTUABhAZGAx1c2FnZV9pb3dhaXQVABYCFooBFooBJsgaJpwaHFgIAAAAAAAANkAYCAAAAAAAADZAAAAVBBUQFRBMFQIVABIAAAAAAAAAgE9AFQAVEBUQLBUCFRAVBhUGHFgIAAAAAACAT0AYCAAAAAAAgE9AAAAAAgAAAAIBAAIVChk1AAYQGRgJdXNhZ2VfaXJxFQAWAhaKARaKASbMHCagHBxYCAAAAAAAgE9AGAgAAAAAAIBPQAAAFQQVEBUQTBUCFQASAAAAAAAAAAAYQBUAFRAVECwVAhUQFQYVBhxYCAAAAAAAABhAGAgAAAAAAAAYQAAAAAIAAAACAQACFQoZNQAGEBkYDXVzYWdlX3NvZnRpcnEVABYCFooBFooBJsoeJp4eHFgIAAAAAAAAGEAYCAAAAAAAABhAAAAVBBUQFRBMFQIVABIAAAAAAAAAAEZAFQAVEBUQLBUCFRAVBhUGHFgIAAAAAAAARkAYCAAAAAAAAEZAAAAAAgAAAAIBAAIVChk1AAYQGRgLdXNhZ2Vfc3RlYWwVABYCFooBFooBJtAgJqQgHFgIAAAAAAAARkAYCAAAAAAAAEZAAAAVBBUQFRBMFQIVABIAAAAAAAAAAFRAFQAVEBUQLBUCFRAVBhUGHFgIAAAAAAAAVEAYCAAAAAAAAFRAAAAAAgAAAAIBAAIVChk1AAYQGRgLdXNhZ2VfZ3Vlc3QVABYCFooBFooBJtIiJqYiHFgIAAAAAAAAVEAYCAAAAAAAAFRAAAAVBBUQFRBMFQIVABIAAAAAAAAAAENAFQAVEBUQLBUCFRAVBhUGHFgIAAAAAAAAQ0AYCAAAAAAAAENAAAAAAgAAAAIBAAIVChk1AAYQGRgQdXNhZ2VfZ3Vlc3RfbmljZRUAFgIWigEWigEm1CQmqCQcWAgAAAAAAABDQBgIAAAAAAAAQ0AAABUEFRAVEEwVAhUAEgAAAGDTczooBQAVABUQFRAsFQIVEBUGFQYcWAgAYNNzOigFABgIAGDTczooBQAAAAACAAAAAgEAAhUEGTUABhAZGAJ0cxUAFgIWigEWigEm4CYmtCYcWAgAYNNzOigFABgIAGDTczooBQAAABkRAhkYBmhvc3RfMBkYBmhvc3RfMBUAGRYAABkRAhkYDGV1LWNlbnRyYWwtMRkYDGV1LWNlbnRyYWwtMRUAGRYAABkRAhkYDWV1LWNlbnRyYWwtMWEZGA1ldS1jZW50cmFsLTFhFQAZFgAAGRECGRgBNhkYATYVABkWAAAZEQIZGAtVYnVudHUxNS4xMBkYC1VidW50dTE1LjEwFQAZFgAAGRECGRgDeDg2GRgDeDg2FQAZFgAAGRECGRgCU0YZGAJTRhUAGRYAABkRAhkYAjE5GRgCMTkVABkWAAAZEQIZGAExGRgBMRUAGRYAABkRAhkYBHRlc3QZGAR0ZXN0FQAZFgAAGRECGRgIAAAAAAAATUAZGAgAAAAAAABNQBUAGRYAABkRAhkYCAAAAAAAAABAGRgIAAAAAAAAAEAVABkWAAAZEQIZGAgAAAAAAAA4QBkYCAAAAAAAADhAFQAZFgAAGRECGRgIAAAAAACATkAZGAgAAAAAAIBOQBUAGRYAABkRAhkYCAAAAAAAADZAGRgIAAAAAAAANkAVABkWAAAZEQIZGAgAAAAAAIBPQBkYCAAAAAAAgE9AFQAZFgAAGRECGRgIAAAAAAAAGEAZGAgAAAAAAAAYQBUAGRYAABkRAhkYCAAAAAAAAEZAGRgIAAAAAAAARkAVABkWAAAZEQIZGAgAAAAAAABUQBkYCAAAAAAAAFRAFQAZFgAAGRECGRgIAAAAAAAAQ0AZGAgAAAAAAABDQBUAGRYAABkRAhkYCABg03M6KAUAGRgIAGDTczooBQAVABkWAAAZHBY4FVYWAAAAGRwWsAIVbhYAAAAZHBbaBBVyFgAAABkcFvwGFUIWAAAAGRwWwggVahYAAAAZHBbIChVKFgAAABkcFowMFUYWAAAAGRwWyA0VRhYAAAAZHBaIDxVCFgAAABkcFtYQFU4WAAAAGRwWxBIVXhYAAAAZHBbEFBVeFgAAABkcFsgWFV4WAAAAGRwWyBgVXhYAAAAZHBbIGhVeFgAAABkcFswcFV4WAAAAGRwWyh4VXhYAAAAZHBbQIBVeFgAAABkcFtIiFV4WAAAAGRwW1CQVXhYAAAAZHBbgJhVeFgAAABUCGfwWSAxhcnJvd19zY2hlbWEVKgAVDCUCGAhob3N0bmFtZSUATBwAAAAVDCUCGAZyZWdpb24lAEwcAAAAFQwlAhgKZGF0YWNlbnRlciUATBwAAAAVDCUCGARyYWNrJQBMHAAAABUMJQIYAm9zJQBMHAAAABUMJQIYBGFyY2glAEwcAAAAFQwlAhgEdGVhbSUATBwAAAAVDCUCGAdzZXJ2aWNlJQBMHAAAABUMJQIYD3NlcnZpY2VfdmVyc2lvbiUATBwAAAAVDCUCGBNzZXJ2aWNlX2Vudmlyb25tZW50JQBMHAAAABUKJQIYCnVzYWdlX3VzZXIAFQolAhgMdXNhZ2Vfc3lzdGVtABUKJQIYCnVzYWdlX2lkbGUAFQolAhgKdXNhZ2VfbmljZQAVCiUCGAx1c2FnZV9pb3dhaXQAFQolAhgJdXNhZ2VfaXJxABUKJQIYDXVzYWdlX3NvZnRpcnEAFQolAhgLdXNhZ2Vfc3RlYWwAFQolAhgLdXNhZ2VfZ3Vlc3QAFQolAhgQdXNhZ2VfZ3Vlc3RfbmljZQAVBCUCGAJ0cwAWAhkcGfwVJo4BHBUMGTUABhAZGAhob3N0bmFtZRUAFgIWhgEWhgEmOCYIHFgGaG9zdF8wGAZob3N0XzAAABbWMRUUFqQoFTYAJp4DHBUMGTUABhAZGAZyZWdpb24VABYCFqoBFqoBJrACJvQBHFgMZXUtY2VudHJhbC0xGAxldS1jZW50cmFsLTEAABbqMRUWFtooFU4AJswFHBUMGTUABhAZGApkYXRhY2VudGVyFQAWAhawARawASbaBCacBBxYDWV1LWNlbnRyYWwtMWEYDWV1LWNlbnRyYWwtMWEAABaAMhUWFqgpFVIAJr4HHBUMGTUABhAZGARyYWNrFQAWAhZoFmgm/AYm1gYcWAE2GAE2AAAWljIVFhb6KRUiACasCRwVDBk1AAYQGRgCb3MVABYCFqQBFqQBJsIIJogIHFgLVWJ1bnR1MTUuMTAYC1VidW50dTE1LjEwAAAWrDIVFhacKhVKACaSCxwVDBk1AAYQGRgEYXJjaBUAFgIWdBZ0JsgKJp4KHFgDeDg2GAN4ODYAABbCMhUWFuYqFSoAJtIMHBUMGTUABhAZGAR0ZWFtFQAWAhZuFm4mjAwm5AscWAJTRhgCU0YAABbYMhUWFpArFSYAJo4OHBUMGTUABhAZGAdzZXJ2aWNlFQAWAhZuFm4myA0moA0cWAIxORgCMTkAABbuMhUWFrYrFSYAJsoPHBUMGTUABhAZGA9zZXJ2aWNlX3ZlcnNpb24VABYCFmgWaCaIDybiDhxYATEYATEAABaEMxUWFtwrFSIAJqQRHBUMGTUABhAZGBNzZXJ2aWNlX2Vudmlyb25tZW50FQAWAhZ6Fnom1hAmqhAcWAR0ZXN0GAR0ZXN0AAAWmjMVFhb+KxUuACaiExwVChk1AAYQGRgKdXNhZ2VfdXNlchUAFgIWigEWigEmxBImmBIcWAgAAAAAAABNQBgIAAAAAAAATUAAABawMxUWFqwsFT4AJqIVHBUKGTUABhAZGAx1c2FnZV9zeXN0ZW0VABYCFooBFooBJsQUJpgUHFgIAAAAAAAAAEAYCAAAAAAAAABAAAAWxjMVFhbqLBU+ACamFxwVChk1AAYQGRgKdXNhZ2VfaWRsZRUAFgIWigEWigEmyBYmnBYcWAgAAAAAAAA4QBgIAAAAAAAAOEAAABbcMxUWFqgtFT4AJqYZHBUKGTUABhAZGAp1c2FnZV9uaWNlFQAWAhaKARaKASbIGCacGBxYCAAAAAAAgE5AGAgAAAAAAIBOQAAAFvIzFRYW5i0VPgAmphscFQoZNQAGEBkYDHVzYWdlX2lvd2FpdBUAFgIWigEWigEmyBomnBocWAgAAAAAAAA2QBgIAAAAAAAANkAAABaINBUWFqQuFT4AJqodHBUKGTUABhAZGAl1c2FnZV9pcnEVABYCFooBFooBJswcJqAcHFgIAAAAAACAT0AYCAAAAAAAgE9AAAAWnjQVFhbiLhU+ACaoHxwVChk1AAYQGRgNdXNhZ2Vfc29mdGlycRUAFgIWigEWigEmyh4mnh4cWAgAAAAAAAAYQBgIAAAAAAAAGEAAABa0NBUWFqAvFT4AJq4hHBUKGTUABhAZGAt1c2FnZV9zdGVhbBUAFgIWigEWigEm0CAmpCAcWAgAAAAAAABGQBgIAAAAAAAARkAAABbKNBUWFt4vFT4AJrAjHBUKGTUABhAZGAt1c2FnZV9ndWVzdBUAFgIWigEWigEm0iImpiIcWAgAAAAAAABUQBgIAAAAAAAAVEAAABbgNBUWFpwwFT4AJrIlHBUKGTUABhAZGBB1c2FnZV9ndWVzdF9uaWNlFQAWAhaKARaKASbUJCaoJBxYCAAAAAAAAENAGAgAAAAAAABDQAAAFvY0FRYW2jAVPgAmviccFQQZNQAGEBkYAnRzFQAWAhaKARaKASbgJia0JhxYCABg03M6KAUAGAgAYNNzOigFAAAAFow1FRYWmDEVPgAWjBYWAgAZHBgMQVJST1c6c2NoZW1hGIwMLy8vLy80QUVBQUFRQUFBQUFBQUtBQXdBQ2dBSkFBUUFDZ0FBQUJBQUFBQUFBUVFBQ0FBSUFBQUFCQUFJQUFBQUJBQUFBQlVBQUFBZ0JBQUE1QU1BQUxRREFBQ0lBd0FBWUFNQUFEUURBQUFJQXdBQTNBSUFBS2dDQUFCd0FnQUFOQUlBQU93QkFBQzRBUUFBaEFFQUFFd0JBQUFZQVFBQTRBQUFBS3dBQUFCNEFBQUFQQUFBQUFRQUFBQkUvUC8vR0FBQUFDQUFBQUFBQUFFQ0hBQUFBQWdBREFBRUFBc0FDQUFBQUVBQUFBQUFBQUFCQUFBQUFBSUFBQUIwY3dBQWVQei8veEFBQUFBVUFBQUFBQUFCQXhBQUFBQXEvdi8vQUFBQ0FBQUFBQUFRQUFBQWRYTmhaMlZmWjNWbGMzUmZibWxqWlFBQUFBQ3cvUC8vRUFBQUFCUUFBQUFBQUFFREVBQUFBR0wrLy84QUFBSUFBQUFBQUFzQUFBQjFjMkZuWlY5bmRXVnpkQURnL1AvL0VBQUFBQlFBQUFBQUFBRURFQUFBQUpMKy8vOEFBQUlBQUFBQUFBc0FBQUIxYzJGblpWOXpkR1ZoYkFBUS9mLy9FQUFBQUJRQUFBQUFBQUVERUFBQUFNTCsvLzhBQUFJQUFBQUFBQTBBQUFCMWMyRm5aVjl6YjJaMGFYSnhBQUFBUlAzLy94QUFBQUFVQUFBQUFBQUJBeEFBQUFEMi92Ly9BQUFDQUFBQUFBQUpBQUFBZFhOaFoyVmZhWEp4QUFBQWRQMy8veEFBQUFBVUFBQUFBQUFCQXhBQUFBQW0vLy8vQUFBQ0FBQUFBQUFNQUFBQWRYTmhaMlZmYVc5M1lXbDBBQUFBQUtqOS8vOFFBQUFBRkFBQUFBQUFBUU1RQUFBQVd2Ly8vd0FBQWdBQUFBQUFDZ0FBQUhWellXZGxYMjVwWTJVQUFOajkvLzhRQUFBQUZBQUFBQUFBQVFNUUFBQUFpdi8vL3dBQUFnQUFBQUFBQ2dBQUFIVnpZV2RsWDJsa2JHVUFBQWorLy84UUFBQUFGQUFBQUFBQUFRTVFBQUFBdXYvLy93QUFBZ0FBQUFBQURBQUFBSFZ6WVdkbFgzTjVjM1JsYlFBQUFBQVFBQllBRUFBT0FBOEFCQUFBQUFnQUVBQUFBQmdBQUFBY0FBQUFBQUFCQXhnQUFBQUFBQVlBQ0FBR0FBWUFBQUFBQUFJQUFBQUFBQW9BQUFCMWMyRm5aVjkxYzJWeUFBQ0Uvdi8vRkFBQUFBd0FBQUFBQUFFRkRBQUFBQUFBQUFCMC92Ly9Fd0FBQUhObGNuWnBZMlZmWlc1MmFYSnZibTFsYm5RQXVQNy8veFFBQUFBTUFBQUFBQUFCQlF3QUFBQUFBQUFBcVA3Ly93OEFBQUJ6WlhKMmFXTmxYM1psY25OcGIyNEE2UDcvL3hRQUFBQU1BQUFBQUFBQkJRd0FBQUFBQUFBQTJQNy8vd2NBQUFCelpYSjJhV05sQUJELy8vOFVBQUFBREFBQUFBQUFBUVVNQUFBQUFBQUFBQUQvLy84RUFBQUFkR1ZoYlFBQUFBQTQvLy8vRkFBQUFBd0FBQUFBQUFFRkRBQUFBQUFBQUFBby8vLy9CQUFBQUdGeVkyZ0FBQUFBWVAvLy94UUFBQUFNQUFBQUFBQUJCUXdBQUFBQUFBQUFVUC8vL3dJQUFBQnZjd0FBaFAvLy94UUFBQUFNQUFBQUFBQUJCUXdBQUFBQUFBQUFkUC8vL3dRQUFBQnlZV05yQUFBQUFLei8vLzhVQUFBQURBQUFBQUFBQVFVTUFBQUFBQUFBQUp6Ly8vOEtBQUFBWkdGMFlXTmxiblJsY2dBQTJQLy8veFFBQUFBTUFBQUFBQUFCQlF3QUFBQUFBQUFBeVAvLy93WUFBQUJ5WldkcGIyNEFBQkFBRkFBUUFBNEFEd0FFQUFBQUNBQVFBQUFBR0FBQUFBd0FBQUFBQUFFRkVBQUFBQUFBQUFBRUFBUUFCQUFBQUFnQUFBQm9iM04wYm1GdFpRQUFBQUE9ABgZcGFycXVldC1ycyB2ZXJzaW9uIDI5LjAuMAC0DQAAUEFSMQ== | base64 --decode > /tmp/sample.parquet
  1. Import sample file to cpu table 
    copy cpu from '/tmp/sample.parquet' with (format='parquet');
waynexia commented 1 year ago

CSV has the similar issue:

table struct:

MySQL [(none)]> desc table host_memory_used_bytes;
+-----------+----------------------+------+---------+---------------+
| Field     | Type                 | Null | Default | Semantic Type |
+-----------+----------------------+------+---------+---------------+
| ts        | TimestampMillisecond | NO   |         | TIME INDEX    |
| collector | String               | YES  |         | PRIMARY KEY   |
| host      | String               | YES  |         | PRIMARY KEY   |
| val       | Float64              | YES  |         | FIELD         |
+-----------+----------------------+------+---------+---------------+
4 rows in set (0.001 sec)

CSV head:

ts,collector, host, val
1692270325901,"memory","f52aa51213c3",2015292000.0
1692270329712,"memory","25a3ccdaf3c6",4079696000.0
1692270332056,"memory","4147c672f2d4",767628000.0
1692270340901,"memory","f52aa51213c3",2018216000.0
1692270344712,"memory","25a3ccdaf3c6",4076592000.0

Error:

2023-08-17T12:49:09.487401Z ERROR frontend::instance: Failed to execute query: copy host_memory_used_bytes from 'data.csv' with (format = 'csv') trace_id=7097919323603341306 err.msg=Failed to read record batch, source: Arrow error: Parser error: Error parsing column 0 at line 2: Parser error: Error parsing timestamp from '1692270329712': error parsing date err.code=Unexpected err.source=Arrow error: Parser error: Error parsing column 0 at line 2: Parser error: Error parsing timestamp from '1692270329712': error parsing date err.source.sources=[Parser error: Error parsing column 0 at line 2: Parser error: Error parsing timestamp from '1692270329712': error parsing date]
WenyXu commented 1 year ago

CSV has the similar issue:

table struct:

MySQL [(none)]> desc table host_memory_used_bytes;
+-----------+----------------------+------+---------+---------------+
| Field     | Type                 | Null | Default | Semantic Type |
+-----------+----------------------+------+---------+---------------+
| ts        | TimestampMillisecond | NO   |         | TIME INDEX    |
| collector | String               | YES  |         | PRIMARY KEY   |
| host      | String               | YES  |         | PRIMARY KEY   |
| val       | Float64              | YES  |         | FIELD         |
+-----------+----------------------+------+---------+---------------+
4 rows in set (0.001 sec)

CSV head:

ts,collector, host, val
1692270325901,"memory","f52aa51213c3",2015292000.0
1692270329712,"memory","25a3ccdaf3c6",4079696000.0
1692270332056,"memory","4147c672f2d4",767628000.0
1692270340901,"memory","f52aa51213c3",2018216000.0
1692270344712,"memory","25a3ccdaf3c6",4076592000.0

Error:

2023-08-17T12:49:09.487401Z ERROR frontend::instance: Failed to execute query: copy host_memory_used_bytes from 'data.csv' with (format = 'csv') trace_id=7097919323603341306 err.msg=Failed to read record batch, source: Arrow error: Parser error: Error parsing column 0 at line 2: Parser error: Error parsing timestamp from '1692270329712': error parsing date err.code=Unexpected err.source=Arrow error: Parser error: Error parsing column 0 at line 2: Parser error: Error parsing timestamp from '1692270329712': error parsing date err.source.sources=[Parser error: Error parsing column 0 at line 2: Parser error: Error parsing timestamp from '1692270329712': error parsing date]

The arrow-csv doesn't support to casting Unix timestamp to timestamp.