Error sending samples to remote storage

[gangsta]

Hi ,

Getting

Jan 24 15:15:51 prometheus prometheus: level=warn ts=2018-01-24T14:15:51.973934966Z caller=queue_manager.go:485 component=remote msg="Error sending samples to remote storage" count=100 err="server returned HTTP status 400 Bad Request: proto: wrong wireType = 2 for field StartTimestampMs"

• prometheus.yaml

  
  remote_read:

• url: http://localhost:9201/write remote_write:

• url: http://localhost:9201/read

• remote_stroage_adapter

/usr/sbin/remote_storage_adapter \
  -influxdb-url=http://domain.com:8086/ \
  -influxdb.database=db \
  -influxdb.retention-policy=autogen \
  -influxdb.username=user \
  -send-timeout=30s

• influxdb

version: influxdb-1.4.2

# Influxdb 
influx 
auth  
CREATE DATABASE db; 
CREATE USER "user" with password 'psswd'; 
GRANT ALL ON db TO user; 
ALTER RETENTION POLICY "autogen" ON "db" DURATION 1d REPLICATION 1 SHARD DURATION 1d DEFAULT;
SHOW RETENTION POLICIES ON db;
#shown above retention

SHOW MEASUREMENTS;
# nothings is shown

• Prometheus

version: 
[root@prometheus ~]# prometheus --version
prometheus, version 2.1.0 (branch: HEAD, revision: 85f23d82a045d103ea7f3c89a91fba4a93e6367a)
  build user:       root@6e784304d3ff
  build date:       20180119-12:01:23
  go version:       go1.9.2

[tomwilkie]

Thanks for the report @kalinux. I think the remote storage adapters have rotted a little. As of influx v1.4.0 you don't need the remote storage adapter IIRC, influx supports our remote read/write APIs natively.

[gangsta]

Hi @tomwilkie Can you give me prometheus.yaml example (or point to some documentation) how to configure Prometheus with Influxdb ? as far as I can find documentation it touches remote_storage_adapter

Regards Karen

[gangsta]

Or you meant ?

remote_read:
- url: http://domain.com:8086/
remote_write:
- url: http://domain.com:8086/

[tomwilkie]

I've not used it before, but this blog post has some details: https://www.influxdata.com/blog/influxdb-now-supports-prometheus-remote-read-write-natively/

[fmoessbauer]

By now there is also official documentation on how to use the native API: https://docs.influxdata.com/influxdb/v1.4/supported_protocols/prometheus#prometheus-remote-read-and-write-api-support

However until https://github.com/influxdata/influxdb/issues/9244 is not resolved, the native backend is not backwards compatible with the remote_storage_adapter.

[gangsta]

Hi @tomwilkie @fmoessbauer , so that means we can close this issue until InfluxDB fixes issue/adds feature ?

[fmoessbauer]

@kalinux this depends on how the prometheus team is going to proceed with the remote_storage_adapter. If support is going to be dropped in favor of the influxdb native one, this issue should be at least documented as a known error.

[tomwilkie]

I chatted to @brian-brazil this weekend at FOSDEM and remote_storage_adapter is going to hang around a little longer, as it serves as a good example. I'm going to update it, add it to our CI etc. I'll chat to influx and see if we should remove the influx path from the adapter or not.

[gangsta]

@tomwilkie the same from me , chatted with @brian-brazil at FOSDEM about how to solve this Influxdb and remote_storage_adapter story , seems only way so far is just to use url: influxdburl+credentials cause Influxdb is not providing basic http authentication . so to me its more in Influxdb side ?

[brian-brazil]

@kalinux That's a different problem you're thinking of.

I note that our influx adpater is the only example of a read adapter we have, though others (such as the CrateDB adapter) also exist out there.

[gangsta]

so what is solution ? use remote_storage_adapter or not ? use url: influxdb+credentials or not ?

[simonszu]

When referring to the original issue in the first post, i now have the same problem as well. I won't drop remote_storage_adapter from my setup until InfluxDB fixes the bug @fmoessbauer mentioned - although it seems that for now i have to live without influxdb integration at all.

[gangsta]

@simonszu you can use as temporary solution putting all in URL, that's works for as well, but again - as temporary solution.

[simonszu]

What do you mean @kalinux ? Adding username and password as GET parameters to the remote storage adapter url?

[gangsta]

copied from Docs.

# Remote write configuration (for Graphite, OpenTSDB, or InfluxDB).
remote_write:
* url: "http://localhost:8086/api/v1/prom/write?u=paul&p=foo&db=prometheus"
# Remote read configuration (for InfluxDB only at the moment).
remote_read:
* url: "http://localhost:8086/api/v1/prom/read?u=paul&p=foo&db=prometheus"

Insecure I know , but there is no other solution so far ,

[simonszu]

Yeah, ok, this is for InfluxDBs new native Prometheus API. Unfortunately there's https://github.com/influxdata/influxdb/issues/9244 which causes the data sent by Prometheus not properly written into InfluxDBs storage.

[weih1214]

seems only way so far is just to use url: influxdburl+credentials cause Influxdb is not providing basic http authentication . so to me its more in Influxdb side

@gangsta Does this mean that basic_auth config doesn't work at all?

[weih1214]

Hi all, I am trying to config adapter for prometheus. I want to use basic_auth for the authentication, but I got error from adapter "err="{\"error\":\"unable to parse authentication credentials\"}", which I guess is something wrong in the authentication. How can I use adapter to communicate with influxdb using certain credentials?

[gangsta]

@weih1214 I`m using influxdburl+credentials as remote storage url, as far as I get, basic_auth works from Prometheus side, but Influxdb is not supporting it , probably thats why you get this error.

[gouthamve]

We went through this issue in our bugscrub and have decided to close this as the upstream issue from Influx is fixed. Please feel free to reopen this if you're still seeing problems!