Closed sermars closed 3 weeks ago
Hi @jason-fox
I have also tried to deploy the connector in a local Kind cluster and apisix is the only service that is not deployed.
Checking the Apisix logs, in all cases the error is this:
Defaulted container "apisix" out of: apisix, wait-for-etcd (init), prepare-apisix (init)
2024/09/30 10:11:04 [emerg] 1#1: bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
2024/09/30 10:11:04 [emerg] 1#1: bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
2024/09/30 10:11:04 [emerg] 1#1: bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
2024/09/30 10:11:04 [emerg] 1#1: bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
2024/09/30 10:11:04 [emerg] 1#1: bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
2024/09/30 10:11:04 [emerg] 1#1: still could not bind()
nginx: [emerg] still could not bind()
When deploying the local version of the DSC, the provider-apisix-control-plane
log shows the same error:
joan@joan-HP-250-G8-Notebook-PC:~/data-space-connector$ kubectl --kubeconfig=target/k3s.yaml logs provider-apisix-control-plane-6cd6bdbb5-gdcp9 -n provider
Defaulted container "apisix" out of: apisix, wait-for-etcd (init), prepare-apisix (init)
2024/09/30 11:18:24 [emerg] 1#1: bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
2024/09/30 11:18:24 [emerg] 1#1: bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
2024/09/30 11:18:24 [emerg] 1#1: bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
2024/09/30 11:18:24 [emerg] 1#1: bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
2024/09/30 11:18:24 [emerg] 1#1: bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/usr/local/apisix/logs/worker_events.sock failed (98: Address already in use)
2024/09/30 11:18:24 [emerg] 1#1: still could not bind()
nginx: [emerg] still could not bind()
joan@joan-HP-250-G8-Notebook-PC:~/data-space-connector$ kubectl --kubeconfig=target/k3s.yaml get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
consumer consumer-keycloak-0 1/1 Running 11 (8m22s ago) 25d
consumer did-helper-7bf56b686f-gmp92 1/1 Running 11 (6m35s ago) 25d
consumer postgresql-0 1/1 Running 11 (6m35s ago) 25d
infra traefik-699bb489cc-jz5tm 1/1 Running 11 (6m35s ago) 25d
kube-system coredns-b5657867f-mz7sq 1/1 Running 0 4m15s
kube-system svclb-provider-apisix-data-plane-c2402009-mj42j 2/2 Running 22 (6m35s ago) 25d
kube-system svclb-traefik-loadbalancer-10b2a178-dtx6j 1/1 Running 11 (6m35s ago) 25d
provider authentication-mysql-0 1/1 Running 11 (6m35s ago) 25d
provider credentials-config-service-75ff6d8c98-99gxf 1/1 Running 26 (5m51s ago) 25d
provider data-service-postgis-0 1/1 Running 11 (6m35s ago) 25d
provider data-service-scorpio-68c89fb7f7-tl2gr 1/1 Running 11 (6m35s ago) 25d
provider did-helper-588dbf95bf-wlhnz 1/1 Running 11 (6m35s ago) 25d
provider dsconfig-644f7fdd-szc4h 1/1 Running 11 (6m35s ago) 25d
provider odrl-pap-c4ff4cdc5-ltdnw 1/1 Running 11 (6m35s ago) 25d
provider postgresql-0 1/1 Running 11 (6m35s ago) 25d
provider provider-apisix-control-plane-6cd6bdbb5-gdcp9 0/1 CrashLoopBackOff 536 (2m1s ago) 25d
provider provider-apisix-data-plane-66686b7455-hrwwn 0/2 Error 275 25d
provider provider-contract-management-5c5c76c997-s6hfm 1/1 Running 11 (6m35s ago) 25d
provider provider-etcd-0 1/1 Running 11 (6m35s ago) 25d
provider provider-etcd-1 1/1 Running 11 (6m35s ago) 25d
provider provider-etcd-2 1/1 Running 11 (6m35s ago) 25d
provider provider-tm-forum-api-customer-bill-management-998d97ddd-mt2bw 1/1 Running 11 (6m35s ago) 25d
provider provider-tm-forum-api-customer-management-586698485f-fg4lp 1/1 Running 11 (6m35s ago) 25d
provider provider-tm-forum-api-envoy-847f4d6cc8-6sx77 1/1 Running 11 (6m35s ago) 25d
provider provider-tm-forum-api-party-catalog-797b89d98b-rn56r 1/1 Running 11 (6m35s ago) 25d
provider provider-tm-forum-api-product-catalog-8658d8c59d-64x5l 1/1 Running 11 (6m35s ago) 25d
provider provider-tm-forum-api-product-inventory-bb5d45c87-754hs 1/1 Running 11 (6m35s ago) 25d
provider provider-tm-forum-api-product-ordering-management-6dc78777xfhr7 1/1 Running 11 (6m35s ago) 25d
provider provider-tm-forum-api-resource-catalog-5fd9bcc997-6wk76 1/1 Running 11 (6m35s ago) 25d
provider provider-tm-forum-api-resource-function-activation-59f9f44vr7fx 1/1 Running 11 (6m35s ago) 25d
provider provider-tm-forum-api-resource-inventory-548fdbf5b4-rgz9k 1/1 Running 11 (6m35s ago) 25d
provider provider-tm-forum-api-service-catalog-66c5c89dfd-smg27 1/1 Running 11 (6m35s ago) 25d
provider tmf-api-registration-265r6 0/1 Error 0 25d
provider tmf-api-registration-5lljx 0/1 Completed 0 25d
provider tmf-api-registration-88n97 0/1 Error 0 25d
provider tmf-api-registration-nwhlz 0/1 Error 0 25d
provider tmf-api-registration-ttsl9 0/1 Error 0 25d
provider trusted-issuers-list-5d8fb688b7-r9k85 1/1 Running 26 (6m11s ago) 25d
provider verifier-95694dc4c-ltn7s 1/1 Running 11 (6m35s ago) 25d
trust-anchor tir-5fb8c4b6d-fjp7j 1/1 Running 27 (6m11s ago) 25d
trust-anchor trust-anchor-mysql-0 1/1 Running 11 (6m35s ago) 25d
@sermars @joancipria Hi, just found the issue. This seems to be an OOM error within apisix. The chart sets resource limits by default(see doc https://github.com/bitnami/charts/tree/main/bitnami/apisix) to "nano", which leads to the container getting killed inside the pod and then entering the crashloop. See the available presets here: https://github.com/bitnami/charts/blob/main/bitnami/common/templates/_resources.tpl#L15 As of now, I would recommend to use small in the local environment. For real deployements, I would highly recommend to set specific resource limits, depending on your needs and cluster and not use one of the presets.(its also what bitnami recommends)
Hello,
We are experiencing a timeout error when deploying the local dataspace. We have tried on different computers and virtual machines, but the error persists. In two computers we have managed to deploy it, although it is not consistent. Most of the time you can't shut down and restart the data space, because this error appears.
We have tried it on a couple of linux operating systems (Ubuntu and Fedora). With docker 27.2.0 and java (jdk) in versions 11.0.24 and 21.0.2. The computers we have tested all have intel i5 or i7 processors and between 16GB and 64GB of RAM.