search

housepower / clickhouse_sinker

Easily load data from kafka to ClickHouse
https://housepower.github.io/clickhouse_sinker
Apache License 2.0
515 stars 118 forks source link

Timestamp format rfc3339_1 (i.e. 2021-07-14T21:45:09Z) not parsing #126

Closed wallflower closed 3 years ago

wallflower commented 3 years ago

I have a json structure with the following structure:

{
  "timestamp": "2021-07-14T21:45:09Z",
  "value": 9331.516,
  "mqtt_topic": "SOME/MQTT/TOPIC"
}

Looking at these two documents: there seems to some ambiguity to whether we should be able to parse that timestamp format?

Seems to be missing that format in the list: https://github.com/housepower/clickhouse_sinker/blob/4196a35f8f509a8a9fd1e483c3b8533fb26621c1/parser/parser.go#L29

Has that format as a test: https://github.com/housepower/clickhouse_sinker/blob/8417dcd51d73b1aafedce641fe58ae333a23c431/parser/parser_test.go#L47

Here is the error message I am receiving for the payload above:

{"level":"error","ts":"2021-07-15T19:42:11.105Z","msg":"flush batch failed","task":"opc_permian","try":5,"error":"stmt.Exec: parsing time \"2021-07-14T21:45:09Z\" as \"2006-01-02 15:04:05\": cannot parse \"T21:45:09Z\" as \" \"","errorVerbose":"parsing time \"2021-07-14T21:45:09Z\" as \"2006-01-02 15:04:05\": cannot parse \"T21:45:09Z\" as \" \"\nstmt.Exec\ngithub.com/housepower/clickhouse_sinker/output.(*ClickHouse).write\n\t/home/zhichyu/go/src/github.com/housepower/clickhouse_sinker/output/clickhouse.go:107\ngithub.com/housepower/clickhouse_sinker/output.(*ClickHouse).loopWrite\n\t/home/zhichyu/go/src/github.com/housepower/clickhouse_sinker/output/clickhouse.go:147\ngithub.com/housepower/clickhouse_sinker/output.(*ClickHouse).Send.func1\n\t/home/zhichyu/go/src/github.com/housepower/clickhouse_sinker/output/clickhouse.go:78\ngithub.com/housepower/clickhouse_sinker/util.(*WorkerPool).wokerFunc\n\t/home/zhichyu/go/src/github.com/housepower/clickhouse_sinker/util/workerpool.go:59\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/housepower/clickhouse_sinker/output.(*ClickHouse).loopWrite\n\t/home/zhichyu/go/src/github.com/housepower/clickhouse_sinker/output/clickhouse.go:162\ngithub.com/housepower/clickhouse_sinker/output.(*ClickHouse).Send.func1\n\t/home/zhichyu/go/src/github.com/housepower/clickhouse_sinker/output/clickhouse.go:78\ngithub.com/housepower/clickhouse_sinker/util.(*WorkerPool).wokerFunc\n\t/home/zhichyu/go/src/github.com/housepower/clickhouse_sinker/util/workerpool.go:59"}
wallflower commented 3 years ago

Sorry, I have my dim set to String not Datetime. It parsed as expected after I fixed my code.