search

maestre3d / alexandria

The Alexandria Project is an open-source platform where people can share their knowledge through books, podcasts, docs and videos.
https://alexandria.damascus-engineering.com
GNU Affero General Public License v3.0
9 stars 1 forks source link

Connection refused by zipkins service and Kafka client has run out of available brokers #42

Open templatedop opened 3 years ago

templatedop commented 3 years ago

Hi! Let me congratulate you for the work done.

In order to understand all the technologies I just ran docker compose after git cloning the repo.

Found that most of the services are not running due to zipkins. Can you please help me on this.

Zipkins logs say it is running out of memory.

OS: macOS Big Sur RAM:16GB

Logs of author service:

2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:02 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused

Log of blob service:

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/blob-service/cmd/alexandria-server/main.go:24 +0x737
2021/02/18 18:22:45 starting event service
2021/02/18 18:22:45 starting http service
2021/02/18 18:23:30 stopping services
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/blob-service/cmd/alexandria-server/main.go:24 +0x737
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/blob-service/cmd/alexandria-server/main.go:24 +0x737
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/blob-service/cmd/alexandria-server/main.go:24 +0x737
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/blob-service/cmd/alexandria-server/main.go:24 +0x737
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/blob-service/cmd/alexandria-server/main.go:24 +0x737
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/blob-service/cmd/alexandria-server/main.go:24 +0x737
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/blob-service/cmd/alexandria-server/main.go:24 +0x737
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/blob-service/cmd/alexandria-server/main.go:24 +0x737
2021/02/19 11:26:17 starting http service
2021/02/19 11:26:17 starting event service
2021/02/19 11:38:56 failed to send the request: Post "http://zipkin:9411/api/v2/spans": EOF
2021/02/19 11:38:56 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp: lookup zipkin on 127.0.0.11:53: no such host
2021/02/19 11:38:57 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:38:58 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:38:59 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:00 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:39:01 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:51:36 failed to send the request: Post "http://zipkin:9411/api/v2/spans": EOF
2021/02/19 11:51:37 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:51:37 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:51:38 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:51:40 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:51:41 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:51:42 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:51:43 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused

Log of Identity service:

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/identity-service/cmd/alexandria-server/main.go:22 +0x4cd
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/identity-service/cmd/alexandria-server/main.go:22 +0x4cd
2021/02/19 11:26:17 starting event service
2021/02/19 11:38:56 failed to send the request: Post "http://zipkin:9411/api/v2/spans": EOF
2021/02/19 11:38:57 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused
2021/02/19 11:38:57 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused

log of media service:

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/media-service/cmd/alexandria-server/main.go:24 +0xa67
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
main.main()
        /go/src/github.com/maestre3d/alexandria/media-service/cmd/alexandria-server/main.go:24 +0xa67
2021/02/19 11:26:10 starting http service
2021/02/19 11:26:10 starting event service
2021/02/19 11:26:10 starting grpc service
2021/02/19 11:38:56 failed to send the request: Post "http://zipkin:9411/api/v2/spans": read tcp 172.18.0.3:48376->172.18.0.5:9411: read: connection reset by peer
2021/02/19 11:38:56 failed to send the request: Post "http://zipkin:9411/api/v2/spans": dial tcp 172.18.0.5:9411: connect: connection refused

log of Kafka:

[2021-02-19 11:26:17,403] INFO [GroupCoordinator 1001]: Dynamic Member with unknown member id joins group BLOB in Stable state. Created a new member id sarama-0393cc0d-256f-4ffd-b223-f6a8f5c2e2c4 for this member and add to the group. (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:17,404] INFO [GroupCoordinator 1001]: Preparing to rebalance group BLOB in state PreparingRebalance with old generation 4 (__consumer_offsets-7) (reason: Adding new member sarama-0393cc0d-256f-4ffd-b223-f6a8f5c2e2c4 with group instance id None) (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,219] INFO [GroupCoordinator 1001]: Member sarama-822e38f0-a93f-4b75-8c26-691e7b6f36eb in group BLOB has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,228] INFO [GroupCoordinator 1001]: Stabilized group BLOB generation 5 (__consumer_offsets-7) (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,239] INFO [GroupCoordinator 1001]: Member sarama-843fec96-1f1c-4cfa-82b4-c1bb3741cf5b in group MEDIA has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,240] INFO [GroupCoordinator 1001]: Assignment received from leader for group BLOB for generation 5 (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,240] INFO [GroupCoordinator 1001]: Member sarama-be35b581-6c0c-4035-b966-9722e590d926 in group MEDIA has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,241] INFO [GroupCoordinator 1001]: Member sarama-86bfb6b4-30b7-4ee9-8700-03cd6e8e761d in group MEDIA has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,242] INFO [GroupCoordinator 1001]: Member sarama-69889ea2-f1e6-4df6-a4c0-7223b4aaae28 in group MEDIA has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,243] INFO [GroupCoordinator 1001]: Member sarama-ec49ff0b-f652-4bd2-88c9-ef04fd2a7142 in group MEDIA has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,245] INFO [GroupCoordinator 1001]: Member sarama-7ec8ad41-81c2-488a-908b-4d1960e4461a in group MEDIA has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,247] INFO [GroupCoordinator 1001]: Stabilized group MEDIA generation 4 (__consumer_offsets-22) (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,256] INFO [GroupCoordinator 1001]: Assignment received from leader for group MEDIA for generation 4 (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,257] INFO [GroupCoordinator 1001]: Member sarama-e7612267-93fd-45f3-a6d6-61566fca8d9b in group AUTHOR has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,259] INFO [GroupCoordinator 1001]: Member sarama-a2e0a140-5409-433b-8d20-5c23af0ac6e9 in group AUTHOR has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,261] INFO [GroupCoordinator 1001]: Member sarama-d6a7d600-5dd1-483e-b97b-ca20b7b8521b in group AUTHOR has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,265] INFO [GroupCoordinator 1001]: Member sarama-0fa018de-82b3-4662-9f9c-97f0f09aa8fa in group AUTHOR has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,271] INFO [GroupCoordinator 1001]: Member sarama-f6735991-37a1-411e-82ea-f34cf67cc129 in group AUTHOR has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,275] INFO [GroupCoordinator 1001]: Stabilized group AUTHOR generation 3 (__consumer_offsets-17) (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,284] INFO [GroupCoordinator 1001]: Assignment received from leader for group AUTHOR for generation 3 (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,286] WARN [GroupCoordinator 1001]: Setting empty assignments for members Set(sarama-c656e792-e201-4965-8382-e38e8bcd9668) of AUTHOR for generation 3 (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,287] INFO [GroupCoordinator 1001]: Member sarama-4fc614db-133a-4f46-8e76-305cb8f443b6 in group IDENTITY has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,288] INFO [GroupCoordinator 1001]: Member sarama-bb1f58a2-9cc5-400c-a1e9-877a801c2fdc in group IDENTITY has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,288] INFO [GroupCoordinator 1001]: Member sarama-aa250f0d-0b6c-49eb-b9b8-567b55f0494b in group IDENTITY has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,288] INFO [GroupCoordinator 1001]: Stabilized group IDENTITY generation 4 (__consumer_offsets-36) (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,331] WARN [GroupCoordinator 1001]: Sending empty assignment to member sarama-c656e792-e201-4965-8382-e38e8bcd9668 of AUTHOR for generation 3 with no errors (kafka.coordinator.group.GroupCoordinator)
[2021-02-19 11:26:18,333] INFO [GroupCoordinator 1001]: Assignment received from leader for group IDENTITY for generation 4 (kafka.coordinator.group.GroupCoordinator)

log of zipkin:

:: version 2.23.2 :: commit 7bf3aab ::

2021-02-19 11:39:02.035  INFO [/] 1 --- [oss-http-*:9411] c.l.a.s.Server                           : Serving HTTP at /0.0.0.0:9411 - http://127.0.0.1:9411/
Terminating due to java.lang.OutOfMemoryError: Java heap space