autometrics-dev / am

Autometrics Companion CLI app
Apache License 2.0
16 stars 2 forks source link

[bug] too many open files #149

Open yukkit opened 1 year ago

yukkit commented 1 year ago

When I start Autometrics CLI I get an error

cmd

# am start http://localhost:8902/prom_metrics

error log

Checking if provided metrics endpoints work...
Failed to make request to http://localhost:8902/prom_metrics (job am_0)
Now sampling the following endpoints for metrics: http://localhost:8902/prom_metrics
Explorer endpoint: http://127.0.0.1:6789
Prometheus endpoint: http://127.0.0.1:9090/prometheus
Using Prometheus version: 2.45.0
Starting prometheus
Error proxying request: reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(9090), path: "/prometheus/api/v1/query_range", query: Some("query=sum+by+%28alertname%2C+objective_name%2C+severity%2C+category%2C+objective_percentile%2C+sloth_id%29+%28ALERTS%29&start=2023-10-18T02%3A53%3A47.240Z&end=2023-10-18T02%3A54%3A18.242Z&step=1s"), fragment: None }, source: hyper::Error(Connect, ConnectError("tcp open error", Os { code: 24, kind: Uncategorized, message: "Too many open files" })) }
Error proxying request: reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(9090), path: "/prometheus/api/v1/query_range", query: Some("query=sum+by+%28alertname%2C+objective_name%2C+severity%2C+category%2C+objective_percentile%2C+sloth_id%29+%28ALERTS%29&start=2023-10-18T02%3A54%3A18.242Z&end=2023-10-18T02%3A54%3A49.238Z&step=1s"), fragment: None }, source: hyper::Error(Connect, ConnectError("tcp open error", Os { code: 24, kind: Uncategorized, message: "Too many open files" })) }

os env

# sysctl kern.maxfiles
kern.maxfiles: 184320
# sysctl kern.maxfilesperproc
kern.maxfilesperproc: 92160
# launchctl limit maxfiles
    maxfiles    256            unlimited
gagbo commented 1 year ago

Hello,

It is an issue we've been unable to reproduce locally, whatever the size of the project we want to deal with (the biggest one we tried being lemmy), so we'll have to try to debug the issue with creating a socket from your machine.

Can you try to run lsof -p `pgrep am` | wc -l to see how many files are handled by the am process? Also, when you just run lsof -p `pgrep am` is there something that looks off in the output?