TCeason commented 4 years ago

Add IoT case for ClickHouse

CLAassistant commented 4 years ago

All committers have signed the CLA.

codecov-io commented 4 years ago

Codecov Report

Merging #120 into master will increase coverage by 1.93%. The diff coverage is 94.57%.

@@            Coverage Diff             @@
##           master     #120      +/-   ##
==========================================
+ Coverage   69.59%   71.52%   +1.93%     
==========================================
  Files         104      105       +1     
  Lines        5812     6245     +433     
==========================================
+ Hits         4045     4467     +422     
- Misses       1689     1698       +9     
- Partials       78       80       +2

Impacted Files	Coverage Δ
cmd/tsbs_load_clickhouse/process.go	`2.01% <0.00%> (-0.03%)`	:arrow_down:
internal/inputs/generator_queries.go	`73.17% <ø> (ø)`
internal/inputs/utils.go	`78.26% <ø> (ø)`
query/stat_processor.go	`16.12% <0.00%> (ø)`
cmd/tsbs_load_clickhouse/creator.go	`36.47% <38.88%> (-0.01%)`	:arrow_down:
...bs_generate_queries/databases/clickhouse/common.go	`83.33% <75.00%> (-4.17%)`	:arrow_down:
.../tsbs_generate_queries/databases/clickhouse/iot.go	`100.00% <100.00%> (ø)`
cmd/tsbs_load_clickhouse/main.go	`47.72% <100.00%> (+2.48%)`	:arrow_up:
query/benchmarker.go	`75.78% <100.00%> (ø)`
query/stats.go	`100.00% <100.00%> (ø)`
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 9f1523d...b05b72b. Read the comment docs.

den-crane commented 4 years ago

ENGINE = MergeTree(created_date, (tags_id, created_at), 8192)

This is outdated syntax and this partitioning does not use partition pruning which makes part of queries much slower.

I recommend to replace it with

ENGINE = MergeTree 
Partition by toYYYYMMDD(created_at) 
Order by (tags_id,created_at) 
SETTINGS index_granularity=1024;

TCeason commented 4 years ago

ENGINE = MergeTree(created_date, (tags_id, created_at), 8192)

This is outdated syntax and this partitioning does not use partition pruning which makes part of queries much slower.

I recommend to replace it with
ENGINE = MergeTree 
Partition by toYYYYMMDD(created_at) 
Order by (tags_id,created_at) 
SETTINGS index_granularity=1024;

Ok

RobAtticus commented 4 years ago

@TCeason let us know when you'd like this looked at it. It appears to have conflicts at the moment.

TCeason commented 4 years ago

@TCeason let us know when you'd like this looked at it. It appears to have conflicts at the moment.

I see it and will fix it today.

TCeason commented 4 years ago

@RobAtticus waiting check.

https://github.com/timescale/tsbs/commit/270a6196abd87e5a19b5d4fb196fb05b55164f7f#diff-08d222aeabbb444fcb19d4df104ae679R21

codecov-commenter commented 4 years ago

Codecov Report

Merging #120 (1b8512c) into master (45b6321) will increase coverage by 2.13%. The diff coverage is 97.77%.

@@            Coverage Diff             @@
##           master     #120      +/-   ##
==========================================
+ Coverage   65.25%   67.38%   +2.13%     
==========================================
  Files         145      146       +1     
  Lines        5966     6363     +397     
==========================================
+ Hits         3893     4288     +395     
- Misses       1960     1961       +1     
- Partials      113      114       +1

Impacted Files	Coverage Δ
pkg/query/stats.go	`100.00% <ø> (ø)`
pkg/targets/clickhouse/processor.go	`1.66% <0.00%> (ø)`
...bs_generate_queries/databases/clickhouse/common.go	`80.00% <71.42%> (-4.62%)`	:arrow_down:
.../tsbs_generate_queries/databases/clickhouse/iot.go	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 45b6321...1b8512c. Read the comment docs.