OpenTSDB against bigtable (Google)

[Ali-Mousa-CK]

I'm trying to use OpenTSDB against big table, but it it takes time show the last emitted point.

Here is my config

google.bigtable.project.id = #####
google.bigtable.instance.id = ####
google.bigtable.zone.id = us-central1-a
hbase.client.connection.impl = com.google.cloud.bigtable.hbase1_2.BigtableConnection
google.bigtable.auth.service.account.enable = true

tsd.http.request.cors_domains = *
tsd.network.port = 4242
tsd.core.auto_create_metrics = true
tsd.core.meta.enable_realtime_ts = true
tsd.core.meta.enable_realtime_uid = true
tsd.core.meta.enable_tsuid_tracking = true
tsd.http.request.enable_chunked = true
tsd.http.request.max_chunk = 131072
tsd.storage.fix_duplicates = true
tsd.storage.enable_compaction = false
tsd.storage.max_tags = 12
tsd.http.staticroot = /opentsdb/build/staticroot
tsd.http.cachedir = /tmp/opentsdb 

[Ali-Mousa-CK]

Any idea how to make it more real time ?

[manolama]

How much time does it take between writing and reading for your data?

[Ali-Mousa-CK]

will seconds ?

[manolama]

I'm not sure what the write to read latency in Bigtable is but by default, the TSD will batch data for up to 1 second before flushing it to storage. Is that what you're seeing?

[mjb512]

I've just experienced (on an otherwise unused test instance) OpenTSDB buffering a handful of writes over the weekend, only pushing to Bigtable once I restarted OpenTSDB two days later. This is a minimal configuration, and starting from truncated tables, which were still empty prior to the restart.

I see there's a config setting tsd.storage.flush_interval with a default of 1000, but AFAICT, it's not actually used?

[manolama]

@mjb512 You're right, that setting doesn't seem to be used in the AsyncBigtable client. It looks like there have been a few good patches to the client lately so I'll try to get a patch in to use the latest client.