Continuous downsampling

[snimavat]

Search before asking

• [X] I searched in the issues and found nothing similar.

Motivation

Is there any support for continuous downsampling, like it can be done with influx or timescale db ? If no, it would be really nice to have feature. If yes, thn I dont see any documentation, please update.

Solution

No response

Alternatives

Influx, timescaledb

Are you willing to submit a PR?

• [ ] I'm willing to submit a PR!

[github-actions[bot]]

Hi, this is your first issue in IoTDB project. Thanks for your report. Welcome to join the community!

[SeanGaluzzi]

I'm looking for the same, i think it depends on the IoTDB version that you have

V0.13.x: https://iotdb.apache.org/UserGuide/V0.13.x/Process-Data/Continuous-Query.html

V13.0.x: nothing found

My main concern is this:

Note that the current distributed version of IoTDB does not support continuous queries. Please stay tuned.

I have IoTDB 1.13.x and trying to create a CQ it fails every time with the provided example:

Error occurred while parsing SQL to physical plan: line 5:2 no viable alternative at input 'SELECT max_value(temperature) \n  INTO temperature_max \n  FROM'

Even with a real path:

IoTDB> CREATE CONTINUOUS QUERY cq1
> BEGIN
>   SELECT max_value(uptime)
>   INTO test
>   FROM root.Switch.**
>   GROUP BY time(10s)
> END
Msg: 700: Error occurred while parsing SQL to physical plan: line 5:2 no viable alternative at input 'SELECT max_value(uptime) \n  INTO test \n  FROM'

@snimavat Did you find anything? @HTHou @CritasWang Can someone help us with this question? i mean, does CQ are at least supported in version 1.13.x? ( or any other way to create a downsample similtar to InfluxDB: Reference: http://wiki.webperfect.ch/index.php?title=InfluxDB_2.x:_Task:_Downsampling )

[CritasWang]

In https://iotdb.apache.org/UserGuide/latest/User-Manual/Database-Programming.html#continuous-query-cq

发件人: SeanGaluzzi @.> 日期: 星期三, 2024年5月15日 20:50 收件人: apache/iotdb @.> 抄送: CritasWang @.>, Mention @.> 主题: Re: [apache/iotdb] Continuous downsampling (Issue #12226)

I'm looking for the same, i think it depends on the IoTDB version that you have

V0.13.x: https://iotdb.apache.org/UserGuide/V0.13.x/Process-Data/Continuous-Query.html V13.0.x: nothing found My main concern is this:

Note that the current distributed version of IoTDB does not support continuous queries. Please stay tuned.

I have IoTDB 1.13.x and trying to create a CQ it fails every time with the provided example:

Error occurred while parsing SQL to physical plan: line 5:2 no viable alternative at input 'SELECT max_value(temperature) \n INTO temperature_max \n FROM'

Even with a real path:

IoTDB> CREATE CONTINUOUS QUERY cq1

BEGIN

SELECT max_value(uptime)

INTO test

FROM root.Switch.**

GROUP BY time(10s)

END

Msg: 700: Error occurred while parsing SQL to physical plan: line 5:2 no viable alternative at input 'SELECT max_value(uptime) \n INTO test \n FROM'

@snimavathttps://github.com/snimavat Did you find anything? @HTHouhttps://github.com/HTHou @CritasWanghttps://github.com/CritasWang Can someone help us with this question? i mean, does CQ are at least supported in version 1.13.x? ( or any other way to create a downsample similtar to InfluxDB: Reference: http://wiki.webperfect.ch/index.php?title=InfluxDB_2.x:_Task:_Downsampling )

― Reply to this email directly, view it on GitHubhttps://github.com/apache/iotdb/issues/12226#issuecomment-2112435154, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEWO4EFO7W7UEXMSXIYUXBDZCNKZJAVCNFSM6AAAAABFEOSO3WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJSGQZTKMJVGQ. You are receiving this because you were mentioned.Message ID: @.***>

[SeanGaluzzi]

Thanks!!!

i'm working on a python script to create every CQS that are needed to downsample an entire db and with only two database i got a lot of CQS (it's pretty messy and hard to debug in case of any kind of issue)

as i pointed out in the previus message with influxDB we can downsample an entire db with a pretty easy query:

option task = {name: "_30d", every: 5m}

data =
    from(bucket: "foo")
        |> range(start: -task.every)
        |> filter(fn: (r) => r._measurement =~ /.*/)

data
    |> aggregateWindow(fn: mean, every: 5m)
    |> filter(fn: (r) => exists r._value)
    |> to(bucket: "foo_30d", org: "bar")

@CritasWang there are any ways to create a full db downsample without the need to create a cqs for every metrics in the db? Thanks in advance

[CritasWang]

If there are many such CQ, then I feel like you can use pipe to handle them https://github.com/apache/iotdb/blob/master/example/pipe-count-point-processor/src/main/java/org/apache/iotdb/CountPointProcessor.java

[SeanGaluzzi]

@CritasWang i found this in the documentation: https://iotdb.apache.org/UserGuide/V1.2.x/User-Manual/Data-Sync.html but i'm note sure how to create the pipe that downsample the entire DB it seems that the pipe is use to transfer data between two IoTDB Instances.

my schema is something like this:

foo_db <- 7 day retention policy

I want to create 2 new db with the pipe to get

• foo30d_db <- 30 day retention policy (20m aggregation data)
• foohystory_db <- no retention policy (1h aggregation data)

Can you please give me an example pipe that downsample the foo_db (and send the data to the same IoTDB instance)?

thanks in advice

[CritasWang]

Sink allows data to be written back to its own IOTDB instance