Open juggledad opened 9 months ago
Let's say I want to collect temperature, humidity and pressure readings from more than one location (say 'location1' and 'location2').
You need to design your time series first. In Opentsdb format, a time series should have a metric name, one or more tag-value pairs. There are a series of data points in a time series. Each data point defines what value the time series is at a specific timestamp.
In you case, you should have 3 metrics (temperature, humidity, and pressure) and all of them have one tag, location.
1)How would I code a put (or multiple put's) for the three values and for the two locations.
Assume there are two locations, 'garden' and 'living_room'. At 11am 9/26/2023 PDT (i.e., Unix epoch time 1695751200), the temperatures are 80 and 70 degree in the garden and living_room respectively.
If you use curl, run these two commands: (Note you need to use your TT host and port in the following, e.g., 192.168.0.2:6182)
curl -v -XPOST 'http://<TT host>:<TT port>/api/put' -d 'put temperature 1695751200 70 location=living_room'
curl -v -XPOST 'http://<TT host>:<TT port>/api/put' -d 'put temperature 1695751200 80 location=garden'
Humidity and pressure are similar.
If you want to send writes in other languages, e.g., python, you can look at a wiki here
2) how would I code a GET to get the data from one of the locations?
Let's say you want to query the temperature in the garden 11am 9/26/2023 PDT, do this:
curl -v 'http://localhost:6182/api/query?start=1695751200&m=avg:temperature\{location=garden\}'
There are lots of examples in this wiki. Hope they will help.
According to teh documentation a put ie in this form:
put /usr/bin/curl -s -XPOST 'http://192.168.1.197:6182/api/put' -d 'put temperature 1696695267 78.62 location=office'
and
/usr/bin/curl -s -XPOST 'http://192.168.1.197:6182/api/put' -d 'put temperature 1696695375 68.9 location=foxboro'
but when I retrieve the data like this:
curl -s 'http://192.168.1.197:6182/api/query?start=1600000000&m=avg:10m-avg:temperature&\{location=office\}'
The results are tee same.
results: pmw@ticktockpi:~/ticktock $ curl -s 'http://192.168.1.197:6182/api/query?start=1600000000&m=avg:10m-avg:temperature&\{location=foxboro}' [{"metric":"temperature","tags":{},"aggregateTags":["location"],"dps":{"1696690200":78.0440000000000111,"1696690800":78.1226315789473915,"1696691400":72.9816216216216276,"1696692000":78.089473684210617,"1696692600":77.8999999999999631,"1696693200":73.1986842105263236,"1696693800":78.4643243243243376,"1696694400":78.5205263157895246,"1696695000":78.4399999999999977}}]pmw@ticktockpi:~/ticktock $ pmw@ticktockpi:~/ticktock $ pmw@ticktockpi:~/ticktock $ curl -s 'http://192.168.1.197:6182/api/query?start=1600000000&m=avg:10m-avg:temperature&\{location=office}' [{"metric":"temperature","tags":{},"aggregateTags":["location"],"dps":{"1696690200":78.0440000000000111,"1696690800":78.1226315789473915,"1696691400":72.9816216216216276,"1696692000":78.089473684210617,"1696692600":77.8999999999999631,"1696693200":73.1986842105263236,"1696693800":78.4643243243243376,"1696694400":78.5205263157895246,"1696695000":78.4399999999999977}}]pmw@ticktockpi:~/ticktock $
Why am I doing wrong?
Looks there are some other data points than the two you just added in the time range [1696695000, 1696695000+10minutes). Can you run the query without 10m-avg (i.e., m=avg:temperature instead of m=avg:10m-avg:temperature) to see what you get?
Just to let you know I have a sensor send in temperature and pressure every 30 seconds and I grab weather data from openweatermap every 30 minutes (temperature, Humidity and pressure)
Here it is: pmw@ticktockpi:~ $ curl -v 'http://192.168.1.197:6182/api/query?start=1600000000&m=avg:temperature&\{location=office}'
GET /api/query?start=1600000000&m=avg:temperature&{location=office} HTTP/1.1 Host: 192.168.1.197:6182 User-Agent: curl/7.74.0 Accept: /
I missed your question since the format was messed up. I believe you meant to ask why two queries with different locations returns the same result. What I would do is to make sure the specific data points are actually inserted in TT first.
According to teh documentation a put ie in this form: put = so I use
/usr/bin/curl -s -XPOST 'http://192.168.1.197:6182/api/put' -d 'put temperature 1696695267 78.62 location=office'
This data point was actually successfully inserted in TT, as you can see it in the query result without downsampling (i.e., m=avg:temperature{location=office}). Just search 1696695267 in your query result, you can see the value 78.6200000000000045 (the last two digit 45 is due to precision lost in data compression).
and
/usr/bin/curl -s -XPOST 'http://192.168.1.197:6182/api/put' -d 'put temperature 1696695375 68.9 location=foxboro'
This data point is not supposed to be in your query result since its location is not 'office'. I don't know what's wrong with your first query. Maybe the two time series happen to have same values at the time range [1696695000,+10min). You can do the same query without downsampling but with location='foxboro' (i.e, m=avg:temperature{location=foxboro}) to see if the data point is correctly inserted or not.
BTW, to make query result concise, you can adjust start and end timestamps to limit the time range, e.g., start=1696695000 &end=1696696000
I ran a test to show the problem so there are just two data points, note: I change the resolution to milliseconds:
; tsdb.timestamp.resolution = millisecond
here is what I did:
pmw@ticktockpi:~/ticktock $ ls
admin bin docker include Makefile Makefile.docker objs scripts test
api-examples conf docs LICENSE Makefile.centos Makefile.ubuntu README.md src tools
pmw@ticktockpi:~/ticktock $
pmw@ticktockpi:~/ticktock $ ./admin/ping.sh
pong
pmw@ticktockpi:~/ticktock $ /usr/bin/curl -s -XPOST 'http://192.168.1.197:6182/api/put' -d 'put temperature 1696847275 77.54 location=office'
pmw@ticktockpi:~/ticktock $ /usr/bin/curl -s 'http://192.168.1.197:6182/api/query?start=1600000000&m=avg:temperature&\{location=office\}'
[{"metric":"temperature","tags":{"location":"office"},"aggregateTags":[],"dps":{"1696847275":77.5400000000000063}}]pmw@ticktockpi:~/ticktock $
pmw@ticktockpi:~/ticktock $ /usr/bin/curl -s -XPOST 'http://192.168.1.197:6182/api/put' -d 'put temperature 1696847401 64.22 location=foxboro'
pmw@ticktockpi:~/ticktock $ /usr/bin/curl -s 'http://192.168.1.197:6182/api/query?start=1600000000&m=avg:temperature&\{location=office\}'
[{"metric":"temperature","tags":{},"aggregateTags":["location"],"dps":{"1696847275":77.5400000000000063,"1696847401":64.2199999999999989}}]pmw@ticktockpi:~/ticktock $
pmw@ticktockpi:~/ticktock $ /usr/bin/curl -s 'http://192.168.1.197:6182/api/query?start=1600000000&m=avg:temperature&\{location=foxboro\}'
[{"metric":"temperature","tags":{},"aggregateTags":["location"],"dps":{"1696847275":77.5400000000000063,"1696847401":64.2199999999999989}}]pmw@ticktockpi:~/ticktock $
The same two entries are returned
pmw@ticktockpi:~/ticktock $ cat data/ticktock.meta
temperature location=office 0
temperature location=foxboro 1
How can I retrieve the data from just one location?
p.s. just to see what would happen, I changed things in the put and crashed ticktock
pmw@ticktockpi:~/ticktock $ /usr/bin/curl -s -XPOST 'http://192.168.1.197:6182/api/put' -d 'put office 1696847275 77.54 temperature=location'
./bin/tt[0x17898]
/lib/arm-linux-gnueabihf/libc.so.6(__default_sa_restorer+0x0)[0x76be2900]
./bin/tt[0x249c4]
./bin/tt[0x5b330]
./bin/tt[0x4041c]
./bin/tt[0x2d828]
./bin/tt[0x41278]
./bin/tt[0x41428]
./bin/tt[0x2adec]
./bin/tt[0x2b7e8]
./bin/tt[0x30438]
/lib/arm-linux-gnueabihf/libstdc++.so.6(+0xa9150)[0x76e77150]
Interrupted (11), shutting down...
Start shutdown process...
in addition, the data files and mete were wiped out
pmw@ticktockpi:~/ticktock/data $ ls -al
total 8
drwxr--r-- 2 pmw pmw 4096 Oct 9 16:25 .
drwxr-xr-x 16 pmw pmw 4096 Oct 9 16:25 ..
-rw-r--r-- 1 pmw pmw 0 Oct 9 16:25 ticktock.meta
(I'm good at breaking things (grin)
- then queried it location=office...TWO reading returned:
pmw@ticktockpi:~/ticktock $ /usr/bin/curl -s 'http://192.168.1.197:6182/api/query?start=1600000000&m=avg:temperature&\{location=office\}' [{"metric":"temperature","tags":{},"aggregateTags":["location"],"dps":{"1696847275":77.5400000000000063,"1696847401":64.2199999999999989}}]pmw@ticktockpi:~/ticktock $
- then queried it location=foxboro...TWO reading returned:
pmw@ticktockpi:~/ticktock $ /usr/bin/curl -s 'http://192.168.1.197:6182/api/query?start=1600000000&m=avg:temperature&\{location=foxboro\}' [{"metric":"temperature","tags":{},"aggregateTags":["location"],"dps":{"1696847275":77.5400000000000063,"1696847401":64.2199999999999989}}]pmw@ticktockpi:~/ticktock $
The same two entries are returned
hm, this is a serious bug. Let me repro it in my side. I assume you use the latest version 0.12.1, don't you?
p.s. just to see what would happen, I changed things in the put and crashed ticktock
pmw@ticktockpi:~/ticktock $ /usr/bin/curl -s -XPOST 'http://192.168.1.197:6182/api/put' -d 'put office 1696847275 77.54 temperature=location'
The put statement looks fine to me. TT shouldn't crash anyway. Are you running it in what hardware (RPI on bullseye 32bit) and what version of TT (v0.12.1)?
Running on a PI 2 Model B running Bullseye 32 bit pmw@ticktockpi:~/ticktock/data $ cat /etc/os-release PRETTY_NAME="Raspbian GNU/Linux 11 (bullseye)" NAME="Raspbian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=raspbian ID_LIKE=debian HOME_URL="http://www.raspbian.org/" SUPPORT_URL="http://www.raspbian.org/RaspbianForums" BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
Oh, and it is ticktock.0.12.1
oh, I know why. Your query format is wrong. You have an extra '&' between temperature and {location=office}
ylin30@raspberrypi:~/ticktock $ /usr/bin/curl -s 'http:/127.0.0.1:6182/api/query?start=1600000000&m=avg:temperature&{location=office}'
[{"metric":"temperature","tags":{},"aggregateTags":["location"],"dps":{"1696847275":77.5400000000000063,"1696847401":64.2199999999999989}}]
ylin30@raspberrypi:~/ticktock $
I can repro it.
ylin30@raspberrypi:~/ticktock $
ylin30@raspberrypi:~/ticktock $ /usr/bin/curl -s 'http:/127.0.0.1:6182/api/query?start=1600000000&m=avg:temperature\{location=office\}'
[{"metric":"temperature","tags":{"location":"office"},"aggregateTags":[],"dps":{"1696847275":77.5400000000000063}}]
ylin30@raspberrypi:~/ticktock $
ylin30@raspberrypi:~/ticktock $ /usr/bin/curl -s 'http:/127.0.0.1:6182/api/query?start=1600000000&m=avg:temperature\{location=foxboro\}'
[{"metric":"temperature","tags":{"location":"foxboro"},"aggregateTags":[],"dps":{"1696847401":64.2199999999999989}}]ylin30@raspberrypi:~/ticktock $
ylin30@raspberrypi:~/ticktock $
Note there is no '&' after temperature. Another hint is "tags":{"location":"office"}
which means the result is really limited to 'location=office'. While in the wrong result, there is no tag but only "aggregateTags":["location"]
, which means that the location is completed ignored and all data are aggregated.
We will think about how to add validation message to protect users from mis-typos.
I am still trying to repro your crash case.
I was basing the query off this the Usage Examples:
[2.5 READ a metric testM1 since 1600000000 epoch time (second) to 1633412176.](https://github.com/ytyou/ticktock/issues/58#25-read-a-metric-testm1-since-1600000000-epoch-time-second-to-1633412176)
curl -v 'http://localhost:6182/api/query?start=1600000000&end=1633412176&m=avg:testM1\{host=foo\}'
Unfortunately I am not able to repro the crash. Please refer to my execution below.
ylin30@raspberrypi:~/ticktock $ /usr/bin/curl -s -v -XPOST 'http://127.0.0.1:6182/api/put' -d 'put office 1696847275 77.54 temperature=locatiion'
* Trying 127.0.0.1:6182...
* Connected to 127.0.0.1 (127.0.0.1) port 6182 (#0)
> POST /api/put HTTP/1.1
> Host: 127.0.0.1:6182
> User-Agent: curl/7.74.0
> Accept: */*
> Content-Length: 49
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 49 out of 49 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Length: 0
< Content-Type: text/plain
<
* Connection #0 to host 127.0.0.1 left intact
ylin30@raspberrypi:~/ticktock $ /usr/bin/curl -s 'http:/127.0.0.1:6182/api/query?start=1600000000&m=avg:office\{temperature=locatiion\}'
[{"metric":"office","tags":{"temperature":"locatiion"},"aggregateTags":[],"dps":{"1696847275":77.5400000000000063}}]ylin30@raspberrypi:~/ticktock $
ylin30@raspberrypi:~/ticktock $
ylin30@raspberrypi:~/ticktock $
ylin30@raspberrypi:~/ticktock $ cat ./data/ticktock.meta
temperature location=office 0
temperature location=foxboro 1
office temperature=locatiion 2
ylin30@raspberrypi:~/ticktock $
I was basing the query off this the Usage Examples:
[2.5 READ a metric testM1 since 1600000000 epoch time (second) to 1633412176.](https://github.com/ytyou/ticktock/issues/58#25-read-a-metric-testm1-since-1600000000-epoch-time-second-to-1633412176) curl -v 'http://localhost:6182/api/query?start=1600000000&end=1633412176&m=avg:testM1\{host=foo\}'
I add another note after this example in the wiki.
Also note that there is no & in between m=avg:testM1 and {host=foo}. If you add a & in between, then anything after & won't be considered as part of query parameters m=aggregator:downsampler:metric{tag1=val1,tag2=val2...}. The results would be like no tags is specified.
Yea!! Thank you so much. That works!!
And I was wrong about where I got that from. At this point I've tried so many iterations I probably added it in at some point.
Do you want me to try and reproduce the crash in another thread and if so is there any information I can gather to help debug it?
Do you want me to try and reproduce the crash in another thread and if so is there any information I can gather to help debug it?
Crash is hard to debug unless with core dump and in debug mode. It would be great if you can repro reliably. Just collect the steps and maybe logs.
So I have TickTock running and I the ping/pong works and the example curl statements to put and get readings work fine but I can seem to get others to work.
Let's say I want to collect temperature, humidity and pressure readings from more than one location (say 'location1' and 'location2').
1)How would I code a put (or multiple put's) for the three values and for the two locations. 2) how would I code a GET to get the data from one of the locations?
Once I can do that I should be able to work more things out myself. p.s. I have been looking at the documentation at opentsdb.net/docs but so far it hasn't helped.
Thanks!