dapr run fails to start

[lazyshawn]

Ask your question here

I'm using dapr with self-host environments. I'm pretty sure that dapr is installed correctly, since I did run a demo last week. But it seems something bad has happened, and I cannot bring up any dapr application.

For example, I run dapr run --app-id myapp --dapr-http-port 3500 will get an error lookup localhost on [::1]:53: read udp [::1]:49104->[::1]:53: read: connection refused'. I don't find the same issue and cannot figure it out.

The containers are running.

> docker ps
CONTAINER ID   IMAGE                COMMAND                  CREATED             STATUS                    PORTS                                                 NAMES
bb6e4a76d499   openzipkin/zipkin    "start-zipkin"           About an hour ago   Up 25 minutes (healthy)   9410/tcp, 0.0.0.0:9411->9411/tcp, :::9411->9411/tcp   dapr_zipkin
05773c890d5a   redis:6              "docker-entrypoint.s…"   About an hour ago   Up 25 minutes             0.0.0.0:6379->6379/tcp, :::6379->6379/tcp             dapr_redis
03d923a0c55a   daprio/dapr:1.11.1   "./placement"            About an hour ago   Up 25 minutes             0.0.0.0:50005->50005/tcp, :::50005->50005/tcp         dapr_placement

Here's the version and platform information.

CLI version: 1.11.0
Runtime version: 1.11.1
OS: archlinux

[pravinpushkar]

@lazyshawn Would you be able to share the complete logs here. Also, please check port 3500 free and connection is not getting blocked by any firewall

[lazyshawn]

@pravinpushkar Thank you for your suggestion. Actually, the complete logs is exactly the connection refused info, and here it is.

❯ dapr run --app-id myapp --dapr-http-port 3500
WARNING: no application command found.
❌  lookup localhost on [::1]:53: read udp [::1]:55453->[::1]:53: read: connection refused

I checked that firewall is turned off.

❯ sudo systemctl status firewalld
○ firewalld.service - firewalld - dynamic firewall daemon
     Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; preset: disabled)
     Active: inactive (dead)
       Docs: man:firewalld(1)

Then I checked the port 3500 is not occupied using ss,

❯ ss -tuln
Netid            State             Recv-Q            Send-Q                       Local Address:Port                          Peer Address:Port            Process
udp              UNCONN            0                 0                                  0.0.0.0:5353                               0.0.0.0:*
udp              UNCONN            0                 0                                  0.0.0.0:5355                               0.0.0.0:*
udp              UNCONN            0                 0                               127.0.0.54:53                                 0.0.0.0:*
udp              UNCONN            0                 0                            127.0.0.53%lo:53                                 0.0.0.0:*
udp              UNCONN            0                 0                                     [::]:5353                                  [::]:*
udp              UNCONN            0                 0                                     [::]:5355                                  [::]:*
tcp              LISTEN            0                 4096                         127.0.0.53%lo:53                                 0.0.0.0:*
tcp              LISTEN            0                 4096                               0.0.0.0:9411                               0.0.0.0:*
tcp              LISTEN            0                 4096                               0.0.0.0:50005                              0.0.0.0:*
tcp              LISTEN            0                 4096                            127.0.0.54:53                                 0.0.0.0:*
tcp              LISTEN            0                 4096                               0.0.0.0:5355                               0.0.0.0:*
tcp              LISTEN            0                 4096                             127.0.0.1:41823                              0.0.0.0:*
tcp              LISTEN            0                 4096                               0.0.0.0:6379                               0.0.0.0:*
tcp              LISTEN            0                 4096                                  [::]:9411                                  [::]:*
tcp              LISTEN            0                 4096                                  [::]:50005                                 [::]:*
tcp              LISTEN            0                 4096                                  [::]:5355                                  [::]:*
tcp              LISTEN            0                 4096                                  [::]:6379                                  [::]:*

That's all I got. Are these info helpful?

[pravinpushkar]

@lazyshawn This is weird. have not faced this earlier. Could you please locate your daprd binary and see if this working for you - <path of your daprd bindary> --app-id test --dapr-http-port 3500 in my case daprd is present in my $HOME/.dapr/bin. For example, I would run this - ~/.dapr/bin/daprd --app-id test --dapr-http-port 3500

[lazyshawn]

@pravinpushkar Running daprd directly does not work either, here are the logs,

❯ ~/.dapr/bin/daprd --app-id test --dapr-http-port 3500
INFO[0000] starting Dapr Runtime -- version 1.11.1 -- commit 204163e75fcfd4d56b6cdbb849265b223372aab0  app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] log level set to: info                        app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] metrics server started on :9090/              app_id=test instance=shawn scope=dapr.metrics type=log ver=1.11.1
FATA[0000] failed to get free port for internal grpc server: lookup localhost on [::1]:53: read udp [::1]:50185->[::1]:53: read: connection refused  app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1

So far, I have found something which may help. First, dapr works well when using wifi at home, using hostpot, while does not work when using wifi in my company. Therefore, I think it may be related to the network. But dapr works on my win machine all the time, and my colleagues haven't encountered the problem, on win and ubuntu, which is weird.

Moreover, dapr should work without network, right? So I tried to turn off wifi and ran the dapr, however nothing changed. Is there any settings for network should I check?

[pravinpushkar]

@lazyshawn ok, It seems it is related to this code - https://github.com/dapr/dapr/blob/master/utils/ports.go. As a workaround can you also pass this parameter --dapr-internal-grpc-port with some free port in this command - ~/.dapr/bin/daprd --app-id test --dapr-http-port 3500 --dapr-internal-grpc-port 52355

@mukundansundar @ItalyPaleAle I think issue should be transferred to runtime.

[lazyshawn]

@pravinpushkar hi, dapr cli starts now. I don't know whether it works correctly when running without an application, at least it works, like below,

~                                                                                                                                                                    16:54:51
❯ ~/.dapr/bin/daprd --app-id test --dapr-http-port 3500 --dapr-internal-grpc-port 52355
INFO[0000] starting Dapr Runtime -- version 1.11.1 -- commit 204163e75fcfd4d56b6cdbb849265b223372aab0  app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] log level set to: info                        app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] metrics server started on :9090/              app_id=test instance=shawn scope=dapr.metrics type=log ver=1.11.1
INFO[0000] loading default configuration                 app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] Resiliency configuration loaded               app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] standalone mode configured                    app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] app id: test                                  app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] mTLS is disabled. Skipping certificate request and tls validation  app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] Dapr trace sampler initialized: DaprTraceSampler(P=0.000100)  app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] local service entry announced: test -> 192.168.60.111:52355  app_id=test component="mdns (nameResolution/v1)" instance=shawn scope=dapr.contrib type=log ver=1.11.1
INFO[0000] Initialized name resolution to mdns           app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] Loading components…                           app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] Waiting for all outstanding components to be processed  app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] All outstanding components processed          app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] Loading endpoints                             app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] Waiting for all outstanding http endpoints to be processed  app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] All outstanding http endpoints processed      app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] gRPC proxy enabled                            app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] gRPC server listening on TCP address: :50001  app_id=test instance=shawn scope=dapr.runtime.grpc.api type=log ver=1.11.1
INFO[0000] Enabled gRPC tracing middleware               app_id=test instance=shawn scope=dapr.runtime.grpc.api type=log ver=1.11.1
INFO[0000] Enabled gRPC metrics middleware               app_id=test instance=shawn scope=dapr.runtime.grpc.api type=log ver=1.11.1
INFO[0000] configuring workflow engine gRPC endpoint     app_id=test instance=shawn scope=dapr.runtime.wfengine type=log ver=1.11.1
INFO[0000] API gRPC server is running on port 50001      app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] enabled metrics http middleware               app_id=test instance=shawn scope=dapr.runtime.http type=log ver=1.11.1
INFO[0000] enabled tracing http middleware               app_id=test instance=shawn scope=dapr.runtime.http type=log ver=1.11.1
INFO[0000] HTTP server listening on TCP address: :3500   app_id=test instance=shawn scope=dapr.runtime.http type=log ver=1.11.1
INFO[0000] http server is running on port 3500           app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] The request body size parameter is: 4         app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] gRPC server listening on TCP address: :52355  app_id=test instance=shawn scope=dapr.runtime.grpc.internal type=log ver=1.11.1
INFO[0000] Enabled gRPC tracing middleware               app_id=test instance=shawn scope=dapr.runtime.grpc.internal type=log ver=1.11.1
INFO[0000] Enabled gRPC metrics middleware               app_id=test instance=shawn scope=dapr.runtime.grpc.internal type=log ver=1.11.1
INFO[0000] internal gRPC server is running on port 52355  app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
WARN[0000] App channel is not initialized. Did you configure an app-port?  app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
WARN[0000] failed to read from bindings: app channel not initialized   app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] dapr initialized. Status: Running. Init Elapsed 2ms  app_id=test instance=shawn scope=dapr.runtime type=log ver=1.11.1

Unfortunately, it's still not working as expected when running with an application. I tried to bring up an application, a server listening on 8088, daprd keeps waiting for app to listen on port 8088, while app seems to be blocked. p.s. the app is correct, and I can dapr run it when using personal wifi. Here are the logs,

~/Desktop/test                                                                                                                                                   45s 16:20:36
❯ ~/.dapr/bin/daprd --app-id hello-dapr --app-port 8088 --dapr-http-port 8089 --dapr-internal-grpc-port 52355 ~/Desktop/test/build/mytest
INFO[0000] starting Dapr Runtime -- version 1.11.1 -- commit 204163e75fcfd4d56b6cdbb849265b223372aab0  app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] log level set to: info                        app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] metrics server started on :9090/              app_id=hello-dapr instance=shawn scope=dapr.metrics type=log ver=1.11.1
INFO[0000] loading default configuration                 app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] Resiliency configuration loaded               app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] standalone mode configured                    app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] app id: hello-dapr                            app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] mTLS is disabled. Skipping certificate request and tls validation  app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] Dapr trace sampler initialized: DaprTraceSampler(P=0.000100)  app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] local service entry announced: hello-dapr -> 192.168.137.26:52355  app_id=hello-dapr component="mdns (nameResolution/v1)" instance=shawn scope=dapr.contrib type=log ver=1.11.1
INFO[0000] Initialized name resolution to mdns           app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] Loading components…                           app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] Waiting for all outstanding components to be processed  app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] All outstanding components processed          app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] Loading endpoints                             app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] Waiting for all outstanding http endpoints to be processed  app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] All outstanding http endpoints processed      app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] gRPC proxy enabled                            app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] gRPC server listening on TCP address: :50001  app_id=hello-dapr instance=shawn scope=dapr.runtime.grpc.api type=log ver=1.11.1
INFO[0000] Enabled gRPC tracing middleware               app_id=hello-dapr instance=shawn scope=dapr.runtime.grpc.api type=log ver=1.11.1
INFO[0000] Enabled gRPC metrics middleware               app_id=hello-dapr instance=shawn scope=dapr.runtime.grpc.api type=log ver=1.11.1
INFO[0000] configuring workflow engine gRPC endpoint     app_id=hello-dapr instance=shawn scope=dapr.runtime.wfengine type=log ver=1.11.1
INFO[0000] API gRPC server is running on port 50001      app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] enabled metrics http middleware               app_id=hello-dapr instance=shawn scope=dapr.runtime.http type=log ver=1.11.1
INFO[0000] enabled tracing http middleware               app_id=hello-dapr instance=shawn scope=dapr.runtime.http type=log ver=1.11.1
INFO[0000] HTTP server listening on TCP address: :8089   app_id=hello-dapr instance=shawn scope=dapr.runtime.http type=log ver=1.11.1
INFO[0000] http server is running on port 8089           app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] The request body size parameter is: 4         app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] gRPC server listening on TCP address: :52355  app_id=hello-dapr instance=shawn scope=dapr.runtime.grpc.internal type=log ver=1.11.1
INFO[0000] Enabled gRPC tracing middleware               app_id=hello-dapr instance=shawn scope=dapr.runtime.grpc.internal type=log ver=1.11.1
INFO[0000] Enabled gRPC metrics middleware               app_id=hello-dapr instance=shawn scope=dapr.runtime.grpc.internal type=log ver=1.11.1
INFO[0000] internal gRPC server is running on port 52355  app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1
INFO[0000] application protocol: http. waiting on port 8088.  This will block until the app is listening on that port.  app_id=hello-dapr instance=shawn scope=dapr.runtime type=log ver=1.11.1

[pravinpushkar]

@lazyshawn ok. somehow the automatic free port resolution logic is not working on your setup.

[ItalyPaleAle]

It seems to be related to the network you’re using. Perhaps the corporate network is disabling certain things like mDNS?

[lazyshawn]

ok, i get it. I agree that my network or setup gets things wrong. Fortunately, dapr does work without the corporate network. I'll try to figure it out later. Thanks for your help.

[dapr-bot]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

[dapr-bot]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue, help wanted or triaged/resolved. Thank you for your contributions.