apptality / aws-ecs-cloudmap-prometheus-sd

A comprehensive solution for dynamic Prometheus service discovery in AWS ECS using OpenTelemetry. This tool enables HTTP-based discovery of ECS tasks and CloudMap services, allowing for flexible filtering, label management, and scalable monitoring. Ideal for teams leveraging AWS ECS.
https://medium.com/@alxtsbk/scraping-prometheus-metrics-from-aws-ecs-9c8d9a1ca1bd
MIT License
5 stars 1 forks source link

How to enable debug #2

Open jseiser opened 3 days ago

jseiser commented 3 days ago

We are wanting to use this software to extend our monitoring to our ECS services, but havent been able to get it to work. We would love a way to enable the debug logging, so we can try to see whats going on.

A few notes, Im running it locally, so it does not have any network access to these actual services, I assume thats fine. We do use govcloud, which maybe is an issue?

services:
  ecs:
    image: apptality/aws-ecs-cloudmap-prometheus-discovery:latest 
    ports:
      - "9001:9001"
    environment:
      DiscoveryOptions__EcsClusters: "ecs-dev;"
      AWS_REGION: "us-gov-west-1"
      AWS_ACCESS_KEY_ID: "xxxxxx"
      AWS_SECRET_ACCESS_KEY: "xxxxxx"
      DiscoveryOptions__EcsTaskTags: "*"
      DiscoveryOptions__EcsServiceTags: "*"
      ASPNETCORE_URLS: "http://*:9001"
    healthcheck:
      test: curl --fail http://localhost:9001/health || exit 1
      interval: 40s
      timeout: 30s
      retries: 3
      start_period: 5s
    networks:
      - ecs
networks:
  ecs:
    name: ecs
āÆ docker compose up                                                                                                    ā”‚
[+] Running 1/1                                                                                                        ā”‚~ [šŸ³ desktop-linux]took 3s 
 āœ” Container justin-ecs-1  Recreated                                                                              0.2s ā”‚āÆ curl http://localhost:9001/prometheus-targets
Attaching to ecs-1                                                                                                     ā”‚[]%                                                                                                                   
ecs-1  | 2024-10-17 19:06:35.400 +00:00 [INF] Started 'ECS Prometheus Discovery' application                           ā”‚
ecs-1  | 2024-10-17 19:06:41.375 +00:00 [INF] Found credentials using the AWS SDK's default credential search          ā”‚~ [šŸ³ desktop-linux]took 2s 
ecs-1  | 2024-10-17 19:06:41.386 +00:00 [INF] Found credentials using the AWS SDK's default credential search          ā”‚āÆ curl http://localhost:9001/prometheus-targets
ecs-1  | 2024-10-17 19:06:44.604 +00:00 [INF] HTTP GET /prometheus-targets responded 200 in 3238.5394 ms    
curl http://localhost:9001/prometheus-targets
[]%  

So its def. doing something, but the cluster listed has like 20+ services in it, so I would hope to see something.

Thanks.

d141 commented 3 days ago

Following up on the above.

I was able to get debug logs but in order to do so I had to get to shell of the container as root so that I could edit appsettings.json. The environment variables are not making into that file.

I changed "EcsClusters": "", to "EcsClusters": "ecs-dev", and changed the log levels to Debug. Then I restarted ./CloudMapEcsPrometheusDiscovery.

When I curl /prometheus-targets, I can see in the debug logs that it is discovering the ECS tasks as expected.

But the response is empty.

Is there additional configuration to be done?

In the logs it will list the full descriptions of everything it finds and then say:

2024-10-17 20:36:31.050 +00:00 [DBG] CloudMap namespaces fetched and cached
2024-10-17 20:36:31.054 +00:00 [INF] Setting HTTP status code 200.
2024-10-17 20:36:31.057 +00:00 [INF] Writing value of type 'PrometheusResponse' as Json.
2024-10-17 20:36:31.058 +00:00 [INF] Executed endpoint 'HTTP: GET /prometheus-targets'

The response from curl is [].