"Attempt to read after end of file" upon insert

[keliss]

Hi. I'm not able to feed any data into CH because it keeps throwing errors like this when carbon-clickhouse inserts data:

DB::Exception: Attempt to read after eof (version 21.8.4.51 (official build)) (from [::ffff:ip]:47634) (in query: INSERT INTO graphite.graphite_data (Path, Value, Time, Date, Timestamp) FORMAT RowBinary )

What am I doing wrong? There's nothing in the logs, I can't even see what data is actually being fed into CH. I tried capturing requests to CH, and looks like there's even no data being sent:

POST /?query=INSERT+INTO+graphite.graphite_data+%28Path%2C+Value%2C+Time%2C+Date%2C+Timestamp%29+FORMAT+RowBinary HTTP/1.1.
Host: my-storage:8123.
User-Agent: Go-http-client/1.1.
Transfer-Encoding: chunked.
Content-Encoding: gzip.
Accept-Encoding: gzip.
Connection: close.
.

Here's my config (by the way, it would be great to actualize all the config params, table SQL queries and provide a more user-friendly description for each param for both carbon-clickhouse and graphite-clickhouse because they are really hard to configure optimally as of now, you have to understand the internals very well):

[common]
metric-prefix = "carbon.agents.{host}"
metric-endpoint = "local"
max-cpu = 1
metric-interval = "1m0s"

[data]
path = "/opt/carbon-clickhouse/"
compression-level = 0
chunk-interval = "30s"
chunk-auto-interval = ""
compression = "none"

[upload.graphite]
type = "points"
table = "graphite.graphite_data"
#date = ""
zero-timestamp = false
threads = 1
url = "http://my-storage:8123/"
compress-data = true
timeout = "1m0s"

[upload.graphite_index]
type = "index"
table = "graphite.graphite_index"
threads = 1
url = "http://my-storage:8123/"
timeout = "1m0s"
cache-ttl = "12h0m0s"
hash = ""

[upload.graphite_tagged]
 type = "tagged"
 table = "graphite.graphite_tagged"
 threads = 1
 url = "http://my-storage:8123/"
 timeout = "1m0s"
 cache-ttl = "12h0m0s"

[udp]
listen = ":2004"
enabled = true
log-incomplete = false
drop-future = 0
drop-past = 0
drop-longer-than = 0

[tcp]
listen = ":2004"
enabled = true
drop-future = 0
drop-past = 0
drop-longer-than = 0
read-timeout = "2m0s"

[pickle]
listen = ":2005"
enabled = true
drop-future = 0
drop-past = 0
drop-longer-than = 0

[grpc]
listen = ":2006"
enabled = false
drop-future = 0
drop-past = 0
drop-longer-than = 0

[prometheus]
listen = ":2007"
enabled = false
drop-future = 0
drop-past = 0
drop-longer-than = 0

[telegraf_http_json]
listen = ":2008"
enabled = false
drop-future = 0
drop-past = 0
drop-longer-than = 0

[pprof]
listen = "localhost:7007"
enabled = false

[[logging]]
logger = ""
file = "/var/log/carbon-clickhouse/carbon-clickhouse.log"
level = "warn"
encoding = "mixed"
encoding-time = "iso8601"
encoding-duration = "seconds"

#[convert_to_tagged]
#enabled = false
#separator = ""

Thanks.

[keliss]

Also, enabling debug level for logs doesn't change anything.

[keliss]

Damn, compression doesn't work. Setting compress-data = false solved the issue. I bet this is a bug.

[Felixoid]

Maybe it just was solved in https://github.com/ClickHouse/ClickHouse/pull/28150

Nevertheless, regarding the point about documentation. Do you find this, for example, difficult to follow? https://github.com/lomik/graphite-clickhouse/blob/master/doc/config.md

It is the same thing I'd like to implement in cch. Unfortunately, I don't have anymore so much time for it since in my current position I don't use the go graphite stack.

As well, worth mentioning, here you can quickly get support https://t.me/ru_go_graphite

[keliss]

Thanks for the links, this is useful.

Regarding the documentation:

1. The SQL queries listed here are pretty much outdated because they don't suggest using any codecs which should be the default: I'm not aware of any major downsides of using codecs. For example, why not make the schema mentioned here the default?
2. There are no recommendations anywhere on how different table parameters like index_granularity or partitioning interval or primary key selection would affect the time series use-case performance and graphite/carbon-clickhouse specifically.
3. There are no recommendations on when a user doesn't need the daily index and in what cases it can be beneficial.
4. There is no hint as to what the cache (cache-ttl = "12h0m0s") is used for and what problems it can create like getting stale data or not getting fresh data. Right now I can only hope that everything will work fine with that cache.
5. There's no clear description of why a user might need the hash parameter (hash = "").
6. There are no recommendations on when a user should use an inverted index and when not.
7. The resources are outdated and scattered across the repos, difficult to find what you need and you can't be sure it's up to date. Right now the documentation is in the following places: https://github.com/lomik/graphite-clickhouse/wiki/, https://github.com/lomik/graphite-clickhouse-tldr, https://github.com/lomik/graphite-clickhouse/tree/master/doc, and https://github.com/lomik/carbon-clickhouse/tree/master. And some of them suggest other storage schemas and configuration parameters which is very confusing because you don't know where the correct configuration as of today is.
8. Not clear how the min-interval/age, max-interval/age parameters affect queries when there are other tables defined (I assume that other data tables can be defined because otherwise, I don't understand why these parameters even exist).
9. Not clear how graphite-clickhouse decides which data tables to query when there are multiple matching ones.
10. What is the context = [] parameter?
11. Some outdated config params are mentioned in the graphite-clickhouse example which creates an additional mental load.
12. There's no clear guideline in the documentation on when zero-timestamp=true is a bad thing, you have to search through PRs to find the answer.
13. Having some use-case examples with fine-tuned configurations would be great.

[Felixoid]

huh, well.. ok =) Most of your questions I added to #91

1. They are still valid, although can be improved for sure. PR is very welcome, the scheme defined in https://github.com/lomik/carbon-clickhouse/issues/64#issuecomment-711680712 works in production for a few years quite well
2. These parameters are load-specific unfortunately. And they are more related to fine tunings. One should refer to ClickHouse documentation to get how to set them up.
3. Almost never it brings benefits when one requests data with more than one-day interval from my experience.
4. It's the internal thing. Never tried to change and didn't read the code, so the default is reasonable. Although, related to PR
5. Added
6. User never uses it. It's selected automatically. Or do you mean reversed tree? Tree tables are deprecated, see https://github.com/lomik/graphite-clickhouse/blob/master/doc/index-table.md
7. Right now the most recent documentation is placed in https://github.com/lomik/graphite-clickhouse/tree/master/doc, every wiki page is moved there except Russians. But I just decided to skip RU pages. As I mentioned, refactoring of cch is stuck, and https://github.com/lomik/graphite-clickhouse-tldr is quite outdated.
8. It's the former implementation of the whisper retention scheme on top of ReplacingMergeTree tables. I would suggest to use GraphiteMergeTree with a proper retention and do not bother yourself by it.
9. In order defined in the config.
10. If only graphite or prometheus is defined there, then the table is used only for according requests.
11. You mean the distributed one? Previously they were even not commented, btw :-j Don't really see what may be fixed there.
12. Never used, don't have opinion here.
13. Sure, would be. The meaning of effort. I don't have time in the closest future for it.

[keliss]

Cool, thanks. Sad to hear the project slowly becomes abandoned.