vagrant demo prometheus file is missing.

[thatsk]

https://github.com/kumahq/kuma-demo/blob/master/vagrant/metrics/app/prometheus/prometheus.yaml#L6

this file is missing and that's why Prometheus is not able to scrape anything because the rule file is missing.

[thatsk]

@jpeach any idea why this is not working?

[thatsk]

vagrant@metrics:~$ sudo journalctl -u kuma-prometheus-sd -f
-- Logs begin at Tue 2021-09-07 04:49:01 UTC. --
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.772Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        creating a gRPC client for Monitoring Assignment Discovery Service (MADS) server ...        {"streamID": 22674}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.773Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        starting an xDS stream ...        {"streamID": 22674}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.773Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        closing a connection ...        {"streamID": 22674}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.773Z        ERROR        kuma-prometheus-sd.run.xds_sd.discoverer        xDS stream terminated with an error        {"streamID": 22674, "error": "failed to start an xDS stream: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:5676: connect: connection refused\"", "errorVerbose": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:5676: connect: connection refused\"\nfailed to start an xDS stream\ngithub.com/kumahq/kuma/app/kuma-prometheus-sd/pkg/discovery/xds/v1.(*streamDiscoverer).Run\n\t/home/circleci/project/app/kuma-prometheus-sd/pkg/discovery/xds/v1/stream.go:55\ngithub.com/kumahq/kuma/app/kuma-prometheus-sd/pkg/discovery/xds.(*discoverer).Run.func1\n\t/home/circleci/project/app/kuma-prometheus-sd/pkg/discovery/xds/xds.go:57\nruntime.goexit\n\t/home/circleci/go/src/runtime/asm_amd64.s:1371"}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.774Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        creating a gRPC client for Monitoring Assignment Discovery Service (MADS) server ...        {"streamID": 22675}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.774Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        starting an xDS stream ...        {"streamID": 22675}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.775Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        closing a connection ...        {"streamID": 22675}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.775Z        ERROR        kuma-prometheus-sd.run.xds_sd.discoverer        xDS stream terminated with an error        {"streamID": 22675, "error": "failed to start an xDS stream: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:5676: connect: connection refused\"", "errorVerbose": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:5676: connect: connection refused\"\nfailed to start an xDS stream\ngithub.com/kumahq/kuma/app/kuma-prometheus-sd/pkg/discovery/xds/v1.(*streamDiscoverer).Run\n\t/home/circleci/project/app/kuma-prometheus-sd/pkg/discovery/xds/v1/stream.go:55\ngithub.com/kumahq/kuma/app/kuma-prometheus-sd/pkg/discovery/xds.(*discoverer).Run.func1\n\t/home/circleci/project/app/kuma-prometheus-sd/pkg/discovery/xds/xds.go:57\nruntime.goexit\n\t/home/circleci/go/src/runtime/asm_amd64.s:1371"}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.775Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        creating a gRPC client for Monitoring Assignment Discovery Service (MADS) server ...        {"streamID": 22676}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.775Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        starting an xDS stream ...        {"streamID": 22676}

[thatsk]

looks like we are not able to generate /var/run/kuma-prometheus-sd/kuma.file_sd.json this file in vagrant provisioning script

[kunalsingthakur]

@jpeach can you take look at this

[jpeach]

IIUC this service should produce the kuma.file_sd.json

[kunalsingthakur]

@jpeach its showing this logs

-- Logs begin at Tue 2021-09-07 04:49:01 UTC. --
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.772Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        creating a gRPC client for Monitoring Assignment Discovery Service (MADS) server ...        {"streamID": 22674}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.773Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        starting an xDS stream ...        {"streamID": 22674}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.773Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        closing a connection ...        {"streamID": 22674}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.773Z        ERROR        kuma-prometheus-sd.run.xds_sd.discoverer        xDS stream terminated with an error        {"streamID": 22674, "error": "failed to start an xDS stream: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:5676: connect: connection refused\"", "errorVerbose": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:5676: connect: connection refused\"\nfailed to start an xDS stream\ngithub.com/kumahq/kuma/app/kuma-prometheus-sd/pkg/discovery/xds/v1.(*streamDiscoverer).Run\n\t/home/circleci/project/app/kuma-prometheus-sd/pkg/discovery/xds/v1/stream.go:55\ngithub.com/kumahq/kuma/app/kuma-prometheus-sd/pkg/discovery/xds.(*discoverer).Run.func1\n\t/home/circleci/project/app/kuma-prometheus-sd/pkg/discovery/xds/xds.go:57\nruntime.goexit\n\t/home/circleci/go/src/runtime/asm_amd64.s:1371"}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.774Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        creating a gRPC client for Monitoring Assignment Discovery Service (MADS) server ...        {"streamID": 22675}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.774Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        starting an xDS stream ...        {"streamID": 22675}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.775Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        closing a connection ...        {"streamID": 22675}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.775Z        ERROR        kuma-prometheus-sd.run.xds_sd.discoverer        xDS stream terminated with an error        {"streamID": 22675, "error": "failed to start an xDS stream: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:5676: connect: connection refused\"", "errorVerbose": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:5676: connect: connection refused\"\nfailed to start an xDS stream\ngithub.com/kumahq/kuma/app/kuma-prometheus-sd/pkg/discovery/xds/v1.(*streamDiscoverer).Run\n\t/home/circleci/project/app/kuma-prometheus-sd/pkg/discovery/xds/v1/stream.go:55\ngithub.com/kumahq/kuma/app/kuma-prometheus-sd/pkg/discovery/xds.(*discoverer).Run.func1\n\t/home/circleci/project/app/kuma-prometheus-sd/pkg/discovery/xds/xds.go:57\nruntime.goexit\n\t/home/circleci/go/src/runtime/asm_amd64.s:1371"}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.775Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        creating a gRPC client for Monitoring Assignment Discovery Service (MADS) server ...        {"streamID": 22676}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.775Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        starting an xDS stream ...        {"streamID": 22676}```

[kunalsingthakur]

somehow this is not discovering https://github.com/kumahq/kuma-demo/blob/master/vagrant/metrics/kuma/kuma-prometheus-sd.service#L8 thats why it's not producing a file

[jpeach]

Oh that makes sense. I guess it's configures for the wrong host/port :-(

[kunalsingthakur]

Can you fix that?

On Tue, 21 Sep 2021, 05:20 James Peach, @.***> wrote:

Oh that makes sense. I guess it's configures for the wrong host/port :-(

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kumahq/kuma-demo/issues/122#issuecomment-923441738, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUPZJGS7XW3DQZX3CVFS3RTUC7CEZANCNFSM5DNKMMKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[github-actions[bot]]

This issue was inactive for 30 days it will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant please comment on it promptly or attend the next triage meeting.

[lahabana]

Closing in favour of the more generic #145