connorourke
commented
1 year ago If i expose and pass through a differernt port with query-exporter -p <port_no_here> config.yaml I get the same thing.
Closed connorourke closed 1 year ago
connorourke
commented
1 year ago If i expose and pass through a differernt port with query-exporter -p <port_no_here> config.yaml I get the same thing.
connorourke
commented
1 year ago Though the port is exposed. Running the following in the container:
from aiohttp import web
async def handle(request):
name = request.match_info.get('name', "World!")
text = "Hello, " + name
print('received request, replying with "{}".'.format(text))
return web.Response(text=text)
app = web.Application()
app.router.add_get('/', handle)
app.router.add_get('/{name}', handle)
web.run_app(app, port=8082)Works fine and serves up the hello world page.
albertodonato
commented
1 year ago What's the exact command line that's being called for query-exporter? it seems it's trying to bind IPv6, which is possibly not enabled in docker
connorourke
commented
1 year ago query-exporter config.yaml -p 8082
(when i am using port 8082 rather than 9056 - but the error is the same either way)
albertodonato
commented
1 year ago by default, that will try to bind ipv6 as well, but that's likely not enabled in docker.
For this reason the docker image only binds ipv4:
connorourke
commented
1 year ago OK - thanks. I'm not using the query-exporter image - I'm installing query-exporter on the container with pip. Is there a way to enable it to bind IPv6 then?
albertodonato
commented
1 year ago connorourke
commented
1 year ago I have enabled IPv6, following the instructions by putting:
{
"experimental": true,
"ip6tables": true
}into my docker/daemon.json and restarting. But the exporter still doesn't work. My compose file with the IPv6 network looks like:
version: "3.3"
services:
prometheus:
container_name: prometheus
image: prom/prometheus
restart: always
volumes:
- ./prometheus:/etc/prometheus/
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/usr/share/prometheus/console_libraries'
- '--web.console.templates=/usr/share/prometheus/consoles'
ports:
- 9090:9090
networks:
- prom_app_net
grafana:
container_name: grafana
image: grafana/grafana
user: '472'
restart: always
environment:
GF_INSTALL_PLUGINS: 'grafana-clock-panel,grafana-simple-json-datasource'
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/provisioning/:/etc/grafana/provisioning/
- './grafana/grafana.ini:/etc/grafana/grafana.ini'
env_file:
- ./grafana/.env_grafana
ports:
- 3000:3000
depends_on:
- prometheus
networks:
- prom_app_net
mysql:
image: mariadb:10.10
hostname: mysql
container_name: mysql
environment:
MYSQL_RANDOM_ROOT_PASSWORD: "yes"
MYSQL_DATABASE: slurm_acct_db
MYSQL_USER: slurm
MYSQL_PASSWORD: password
volumes:
- var_lib_mysql:/var/lib/mysql
networks:
- slurm
# network_mode: host
slurmdbd:
image: prom-slurm-cluster:${IMAGE_TAG:-21.08.6}
build:
context: .
args:
SLURM_TAG: ${SLURM_TAG:-slurm-21-08-6-1}
command: ["slurmdbd"]
container_name: slurmdbd
hostname: slurmdbd
volumes:
- etc_munge:/etc/munge
- etc_slurm:/etc/slurm
- var_log_slurm:/var/log/slurm
- cgroups:/sys/fs/cgroup:ro
expose:
- "6819"
ports:
- "6819:6819"
depends_on:
- mysql
privileged: true
cgroup: host
networks:
- slurm
#network_mode: host
slurmctld:
image: prom-slurm-cluster:${IMAGE_TAG:-21.08.6}
command: ["slurmctld"]
container_name: slurmctld
hostname: slurmctld
volumes:
- etc_munge:/etc/munge
- etc_slurm:/etc/slurm
- slurm_jobdir:/data
- var_log_slurm:/var/log/slurm
- etc_prometheus:/etc/prometheus
- /sys/fs/cgroup:/sys/fs/cgroup:rw
expose:
- "6817"
- "8080"
- "8081"
- "8082/tcp"
ports:
- 8080:8080
- 8081:8081
- 8082:8082/tcp
depends_on:
- "slurmdbd"
privileged: true
cgroup: host
#network_mode: host
networks:
- slurm
c1:
image: prom-slurm-cluster:${IMAGE_TAG:-21.08.6}
command: ["slurmd"]
hostname: c1
container_name: c1
volumes:
- etc_munge:/etc/munge
- etc_slurm:/etc/slurm
- slurm_jobdir:/data
- var_log_slurm:/var/log/slurm
- cgroups:/sys/fs/cgroup:ro
expose:
- "6818"
depends_on:
- "slurmctld"
privileged: true
cgroup: host
#network_mode: host
networks:
- slurm
c2:
image: prom-slurm-cluster:${IMAGE_TAG:-21.08.6}
command: ["slurmd"]
hostname: c2
container_name: c2
volumes:
- etc_munge:/etc/munge
- etc_slurm:/etc/slurm
- slurm_jobdir:/data
- var_log_slurm:/var/log/slurm
- cgroups:/sys/fs/cgroup:ro
expose:
- "6818"
- "22"
depends_on:
- "slurmctld"
privileged: true
cgroup: host
networks:
- slurm
#network_mode: host
volumes:
etc_munge:
etc_slurm:
slurm_jobdir:
var_lib_mysql:
var_log_slurm:
grafana_data:
prometheus_data:
cgroups:
etc_prometheus:
networks:
prom_app_net:
slurm:
enable_ipv6: true
ipam:
config:
- subnet: 2001:0DB8::/112When I run query-exporter like query-exporter config.yaml -p 8082 with the following config file:
databases:
db1:
dsn: sqlite:////test.db
connect-sql:
- PRAGMA application_id = 123
- PRAGMA auto_vacuum = 1
labels:
region: us1
app: app1
metrics:
metric1:
type: gauge
description: A sample gauge
queries:
query1:
interval: 5
databases: [db1]
metrics: [metric1]
sql: SELECT random() / 1000000000000000 AS metric1It doesnt work:
But if i run the following simple exporter on the same port:
from prometheus_client import start_http_server, Summary
import random
import time
# Create a metric to track time spent and requests made.
REQUEST_TIME = Summary('request_processing_seconds', 'Time spent processing request')
# Decorate function with metric.
@REQUEST_TIME.time()
def process_request(t):
"""A dummy function that takes some time."""
time.sleep(t)
if __name__ == '__main__':
# Start up the server to expose the metrics.
start_http_server(8082)
# Generate some requests.
while True:
process_request(random.random())It works fine:
Is this a bug, or is there some other step that I am missing?
Thanks!
connorourke
commented
1 year ago Ah - ok. If I start it with query-exporter config.yaml -p 8082 -H 0.0.0.0 it works.
Describe the bug
I'm running query-exporter within a Docker container. WHen I try to start it with the example config.yaml I get the following error:
The port
9560is exposed (i am running in theslurmctldcontainer:Installation details
RUN wget https://www.python.org/ftp/python/3.10.12/Python-3.10.12.tgz \ && tar -xzvf Python-3.10.12.tgz\ && pushd Python-3.10.12 \ && ./configure --enable-optimizations --prefix=/usr/local\ && make \ && make altinstall
databases: db1: dsn: sqlite:// connect-sql:
metrics: metric1: type: gauge description: A sample gauge metric2: type: summary description: A sample summary labels: [l1, l2] expiration: 24h metric3: type: histogram description: A sample histogram buckets: [10, 20, 50, 100, 1000] metric4: type: enum description: A sample enum states: [foo, bar, baz]
queries: query1: interval: 5 databases: [db1] metrics: [metric1] sql: SELECT random() / 1000000000000000 AS metric1 query2: interval: 20 timeout: 0.5 databases: [db1, db2] metrics: [metric2, metric3] sql: | SELECT abs(random() / 1000000000000000) AS metric2, abs(random() / 10000000000000000) AS metric3, "value1" AS l1, "value2" AS l2 query3: schedule: "/5 *" databases: [db2] metrics: [metric4] sql: | SELECT value FROM ( SELECT "foo" AS metric4 UNION SELECT "bar" AS metric3 UNION SELECT "baz" AS metric4 ) ORDER BY random() LIMIT 1