ytyou / ticktock

TickTockDB is an OpenTSDB-like time series database, with much better performance.
GNU General Public License v3.0
72 stars 8 forks source link

Trying to under stand the format of PUT's and GET's #58

Open juggledad opened 9 months ago

juggledad commented 9 months ago

So I have TickTock running and I the ping/pong works and the example curl statements to put and get readings work fine but I can seem to get others to work.

Let's say I want to collect temperature, humidity and pressure readings from more than one location (say 'location1' and 'location2').

1)How would I code a put (or multiple put's) for the three values and for the two locations. 2) how would I code a GET to get the data from one of the locations?

Once I can do that I should be able to work more things out myself. p.s. I have been looking at the documentation at opentsdb.net/docs but so far it hasn't helped.

Thanks!

ylin30 commented 9 months ago

Let's say I want to collect temperature, humidity and pressure readings from more than one location (say 'location1' and 'location2').

You need to design your time series first. In Opentsdb format, a time series should have a metric name, one or more tag-value pairs. There are a series of data points in a time series. Each data point defines what value the time series is at a specific timestamp.

In you case, you should have 3 metrics (temperature, humidity, and pressure) and all of them have one tag, location.

1)How would I code a put (or multiple put's) for the three values and for the two locations.

Assume there are two locations, 'garden' and 'living_room'. At 11am 9/26/2023 PDT (i.e., Unix epoch time 1695751200), the temperatures are 80 and 70 degree in the garden and living_room respectively.

If you use curl, run these two commands: (Note you need to use your TT host and port in the following, e.g., 192.168.0.2:6182)

curl -v -XPOST 'http://<TT host>:<TT port>/api/put' -d 'put temperature 1695751200 70 location=living_room'
curl -v -XPOST 'http://<TT host>:<TT port>/api/put' -d 'put temperature 1695751200 80 location=garden'

Humidity and pressure are similar.

If you want to send writes in other languages, e.g., python, you can look at a wiki here

2) how would I code a GET to get the data from one of the locations?

Let's say you want to query the temperature in the garden 11am 9/26/2023 PDT, do this:

curl -v 'http://localhost:6182/api/query?start=1695751200&m=avg:temperature\{location=garden\}'

There are lots of examples in this wiki. Hope they will help.

juggledad commented 9 months ago

According to teh documentation a put ie in this form: put = so I use /usr/bin/curl -s -XPOST 'http://192.168.1.197:6182/api/put' -d 'put temperature 1696695267 78.62 location=office' and /usr/bin/curl -s -XPOST 'http://192.168.1.197:6182/api/put' -d 'put temperature 1696695375 68.9 location=foxboro'

but when I retrieve the data like this: curl -s 'http://192.168.1.197:6182/api/query?start=1600000000&m=avg:10m-avg:temperature&\{location=office\}' The results are tee same.

results: pmw@ticktockpi:~/ticktock $ curl -s 'http://192.168.1.197:6182/api/query?start=1600000000&m=avg:10m-avg:temperature&\{location=foxboro}' [{"metric":"temperature","tags":{},"aggregateTags":["location"],"dps":{"1696690200":78.0440000000000111,"1696690800":78.1226315789473915,"1696691400":72.9816216216216276,"1696692000":78.089473684210617,"1696692600":77.8999999999999631,"1696693200":73.1986842105263236,"1696693800":78.4643243243243376,"1696694400":78.5205263157895246,"1696695000":78.4399999999999977}}]pmw@ticktockpi:~/ticktock $ pmw@ticktockpi:~/ticktock $ pmw@ticktockpi:~/ticktock $ curl -s 'http://192.168.1.197:6182/api/query?start=1600000000&m=avg:10m-avg:temperature&\{location=office}' [{"metric":"temperature","tags":{},"aggregateTags":["location"],"dps":{"1696690200":78.0440000000000111,"1696690800":78.1226315789473915,"1696691400":72.9816216216216276,"1696692000":78.089473684210617,"1696692600":77.8999999999999631,"1696693200":73.1986842105263236,"1696693800":78.4643243243243376,"1696694400":78.5205263157895246,"1696695000":78.4399999999999977}}]pmw@ticktockpi:~/ticktock $

Why am I doing wrong?

ylin30 commented 9 months ago

Looks there are some other data points than the two you just added in the time range [1696695000, 1696695000+10minutes). Can you run the query without 10m-avg (i.e., m=avg:temperature instead of m=avg:10m-avg:temperature) to see what you get?

juggledad commented 9 months ago

Just to let you know I have a sensor send in temperature and pressure every 30 seconds and I grab weather data from openweatermap every 30 minutes (temperature, Humidity and pressure)

Here it is: pmw@ticktockpi:~ $ curl -v 'http://192.168.1.197:6182/api/query?start=1600000000&m=avg:temperature&\{location=office}'

ylin30 commented 9 months ago

I missed your question since the format was messed up. I believe you meant to ask why two queries with different locations returns the same result. What I would do is to make sure the specific data points are actually inserted in TT first.

According to teh documentation a put ie in this form: put = so I use /usr/bin/curl -s -XPOST 'http://192.168.1.197:6182/api/put' -d 'put temperature 1696695267 78.62 location=office'

This data point was actually successfully inserted in TT, as you can see it in the query result without downsampling (i.e., m=avg:temperature{location=office}). Just search 1696695267 in your query result, you can see the value 78.6200000000000045 (the last two digit 45 is due to precision lost in data compression).

and /usr/bin/curl -s -XPOST 'http://192.168.1.197:6182/api/put' -d 'put temperature 1696695375 68.9 location=foxboro'

This data point is not supposed to be in your query result since its location is not 'office'. I don't know what's wrong with your first query. Maybe the two time series happen to have same values at the time range [1696695000,+10min). You can do the same query without downsampling but with location='foxboro' (i.e, m=avg:temperature{location=foxboro}) to see if the data point is correctly inserted or not.

BTW, to make query result concise, you can adjust start and end timestamps to limit the time range, e.g., start=1696695000 &end=1696696000

juggledad commented 9 months ago

I ran a test to show the problem so there are just two data points, note: I change the resolution to milliseconds:

; tsdb.timestamp.resolution = millisecond

here is what I did:


1) With ticktock down, I deleted the data and log folders

pmw@ticktockpi:~/ticktock $ ls
admin         bin   docker  include  Makefile         Makefile.docker  objs       scripts  test
api-examples  conf  docs    LICENSE  Makefile.centos  Makefile.ubuntu  README.md  src      tools
pmw@ticktockpi:~/ticktock $ 

2) started ticktock and ran ping

pmw@ticktockpi:~/ticktock $ ./admin/ping.sh
pong

3) posted a temperature reading (77.54) with location=office

pmw@ticktockpi:~/ticktock $ /usr/bin/curl -s  -XPOST 'http://192.168.1.197:6182/api/put'  -d 'put temperature 1696847275 77.54 location=office'

4) then queried it {location=office}...one reading returned:

pmw@ticktockpi:~/ticktock $ /usr/bin/curl -s 'http://192.168.1.197:6182/api/query?start=1600000000&m=avg:temperature&\{location=office\}'
[{"metric":"temperature","tags":{"location":"office"},"aggregateTags":[],"dps":{"1696847275":77.5400000000000063}}]pmw@ticktockpi:~/ticktock $ 

5) posted a temperature reading (64.22) with location=foxboro

pmw@ticktockpi:~/ticktock $ /usr/bin/curl -s  -XPOST 'http://192.168.1.197:6182/api/put'  -d 'put temperature 1696847401 64.22 location=foxboro'

6) then queried it location=office...TWO reading returned:

pmw@ticktockpi:~/ticktock $ /usr/bin/curl -s 'http://192.168.1.197:6182/api/query?start=1600000000&m=avg:temperature&\{location=office\}'
[{"metric":"temperature","tags":{},"aggregateTags":["location"],"dps":{"1696847275":77.5400000000000063,"1696847401":64.2199999999999989}}]pmw@ticktockpi:~/ticktock $ 

7) then queried it location=foxboro...TWO reading returned:

pmw@ticktockpi:~/ticktock $ /usr/bin/curl -s 'http://192.168.1.197:6182/api/query?start=1600000000&m=avg:temperature&\{location=foxboro\}'
[{"metric":"temperature","tags":{},"aggregateTags":["location"],"dps":{"1696847275":77.5400000000000063,"1696847401":64.2199999999999989}}]pmw@ticktockpi:~/ticktock $

The same two entries are returned


8) listed the ticktock.meta file:

pmw@ticktockpi:~/ticktock $ cat data/ticktock.meta
temperature location=office 0
temperature location=foxboro 1

How can I retrieve the data from just one location?

juggledad commented 9 months ago

p.s. just to see what would happen, I changed things in the put and crashed ticktock

pmw@ticktockpi:~/ticktock $ /usr/bin/curl -s  -XPOST 'http://192.168.1.197:6182/api/put'  -d 'put office 1696847275 77.54 temperature=location'

./bin/tt[0x17898]
/lib/arm-linux-gnueabihf/libc.so.6(__default_sa_restorer+0x0)[0x76be2900]
./bin/tt[0x249c4]
./bin/tt[0x5b330]
./bin/tt[0x4041c]
./bin/tt[0x2d828]
./bin/tt[0x41278]
./bin/tt[0x41428]
./bin/tt[0x2adec]
./bin/tt[0x2b7e8]
./bin/tt[0x30438]
/lib/arm-linux-gnueabihf/libstdc++.so.6(+0xa9150)[0x76e77150]
Interrupted (11), shutting down...
Start shutdown process...

in addition, the data files and mete were wiped out

pmw@ticktockpi:~/ticktock/data $ ls -al
total 8
drwxr--r--  2 pmw pmw 4096 Oct  9 16:25 .
drwxr-xr-x 16 pmw pmw 4096 Oct  9 16:25 ..
-rw-r--r--  1 pmw pmw    0 Oct  9 16:25 ticktock.meta

(I'm good at breaking things (grin)

ylin30 commented 9 months ago
  1. then queried it location=office...TWO reading returned:
pmw@ticktockpi:~/ticktock $ /usr/bin/curl -s 'http://192.168.1.197:6182/api/query?start=1600000000&m=avg:temperature&\{location=office\}'
[{"metric":"temperature","tags":{},"aggregateTags":["location"],"dps":{"1696847275":77.5400000000000063,"1696847401":64.2199999999999989}}]pmw@ticktockpi:~/ticktock $ 
  1. then queried it location=foxboro...TWO reading returned:
pmw@ticktockpi:~/ticktock $ /usr/bin/curl -s 'http://192.168.1.197:6182/api/query?start=1600000000&m=avg:temperature&\{location=foxboro\}'
[{"metric":"temperature","tags":{},"aggregateTags":["location"],"dps":{"1696847275":77.5400000000000063,"1696847401":64.2199999999999989}}]pmw@ticktockpi:~/ticktock $

The same two entries are returned

hm, this is a serious bug. Let me repro it in my side. I assume you use the latest version 0.12.1, don't you?

ylin30 commented 9 months ago

p.s. just to see what would happen, I changed things in the put and crashed ticktock


pmw@ticktockpi:~/ticktock $ /usr/bin/curl -s  -XPOST 'http://192.168.1.197:6182/api/put'  -d 'put office 1696847275 77.54 temperature=location'

The put statement looks fine to me. TT shouldn't crash anyway. Are you running it in what hardware (RPI on bullseye 32bit) and what version of TT (v0.12.1)?

juggledad commented 9 months ago

Running on a PI 2 Model B running Bullseye 32 bit pmw@ticktockpi:~/ticktock/data $ cat /etc/os-release PRETTY_NAME="Raspbian GNU/Linux 11 (bullseye)" NAME="Raspbian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=raspbian ID_LIKE=debian HOME_URL="http://www.raspbian.org/" SUPPORT_URL="http://www.raspbian.org/RaspbianForums" BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

juggledad commented 9 months ago

Oh, and it is ticktock.0.12.1

ylin30 commented 9 months ago

oh, I know why. Your query format is wrong. You have an extra '&' between temperature and {location=office}

ylin30@raspberrypi:~/ticktock $ /usr/bin/curl -s 'http:/127.0.0.1:6182/api/query?start=1600000000&m=avg:temperature&{location=office}'
[{"metric":"temperature","tags":{},"aggregateTags":["location"],"dps":{"1696847275":77.5400000000000063,"1696847401":64.2199999999999989}}]
ylin30@raspberrypi:~/ticktock $

I can repro it.

ylin30@raspberrypi:~/ticktock $
ylin30@raspberrypi:~/ticktock $ /usr/bin/curl -s 'http:/127.0.0.1:6182/api/query?start=1600000000&m=avg:temperature\{location=office\}'
[{"metric":"temperature","tags":{"location":"office"},"aggregateTags":[],"dps":{"1696847275":77.5400000000000063}}]
ylin30@raspberrypi:~/ticktock $
ylin30@raspberrypi:~/ticktock $ /usr/bin/curl -s 'http:/127.0.0.1:6182/api/query?start=1600000000&m=avg:temperature\{location=foxboro\}'
[{"metric":"temperature","tags":{"location":"foxboro"},"aggregateTags":[],"dps":{"1696847401":64.2199999999999989}}]ylin30@raspberrypi:~/ticktock $
ylin30@raspberrypi:~/ticktock $

Note there is no '&' after temperature. Another hint is "tags":{"location":"office"} which means the result is really limited to 'location=office'. While in the wrong result, there is no tag but only "aggregateTags":["location"], which means that the location is completed ignored and all data are aggregated.

We will think about how to add validation message to protect users from mis-typos.

I am still trying to repro your crash case.

juggledad commented 9 months ago

I was basing the query off this the Usage Examples:

[2.5 READ a metric testM1 since 1600000000 epoch time (second) to 1633412176.](https://github.com/ytyou/ticktock/issues/58#25-read-a-metric-testm1-since-1600000000-epoch-time-second-to-1633412176)

curl -v 'http://localhost:6182/api/query?start=1600000000&end=1633412176&m=avg:testM1\{host=foo\}'
ylin30 commented 9 months ago

Unfortunately I am not able to repro the crash. Please refer to my execution below.

ylin30@raspberrypi:~/ticktock $ /usr/bin/curl -s -v  -XPOST 'http://127.0.0.1:6182/api/put'  -d 'put office 1696847275 77.54 temperature=locatiion'
*   Trying 127.0.0.1:6182...
* Connected to 127.0.0.1 (127.0.0.1) port 6182 (#0)
> POST /api/put HTTP/1.1
> Host: 127.0.0.1:6182
> User-Agent: curl/7.74.0
> Accept: */*
> Content-Length: 49
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 49 out of 49 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Length: 0
< Content-Type: text/plain
<
* Connection #0 to host 127.0.0.1 left intact
ylin30@raspberrypi:~/ticktock $ /usr/bin/curl -s 'http:/127.0.0.1:6182/api/query?start=1600000000&m=avg:office\{temperature=locatiion\}'
[{"metric":"office","tags":{"temperature":"locatiion"},"aggregateTags":[],"dps":{"1696847275":77.5400000000000063}}]ylin30@raspberrypi:~/ticktock $
ylin30@raspberrypi:~/ticktock $
ylin30@raspberrypi:~/ticktock $
ylin30@raspberrypi:~/ticktock $ cat ./data/ticktock.meta
temperature location=office 0
temperature location=foxboro 1
office temperature=locatiion 2
ylin30@raspberrypi:~/ticktock $
ylin30 commented 9 months ago

I was basing the query off this the Usage Examples:

[2.5 READ a metric testM1 since 1600000000 epoch time (second) to 1633412176.](https://github.com/ytyou/ticktock/issues/58#25-read-a-metric-testm1-since-1600000000-epoch-time-second-to-1633412176)

curl -v 'http://localhost:6182/api/query?start=1600000000&end=1633412176&m=avg:testM1\{host=foo\}'

I add another note after this example in the wiki.

Also note that there is no & in between m=avg:testM1 and {host=foo}. If you add a & in between, then anything after & won't be considered as part of query parameters m=aggregator:downsampler:metric{tag1=val1,tag2=val2...}. The results would be like no tags is specified.

juggledad commented 9 months ago

Yea!! Thank you so much. That works!!

And I was wrong about where I got that from. At this point I've tried so many iterations I probably added it in at some point.

Do you want me to try and reproduce the crash in another thread and if so is there any information I can gather to help debug it?

ylin30 commented 9 months ago

Do you want me to try and reproduce the crash in another thread and if so is there any information I can gather to help debug it?

Crash is hard to debug unless with core dump and in debug mode. It would be great if you can repro reliably. Just collect the steps and maybe logs.