Scheduler service is not started in self-hosted mode

mikeee commented 1 week ago

Expected Behavior

The scheduler should be started when using dapr init.

Actual Behavior

C:\Users\#>dapr init --runtime-version 1.14.0-rc.1
Making the jump to hyperspace...
Container images will be pulled from Docker Hub
Installing runtime version 1.14.0-rc.1
Downloading binaries and setting up components...
Downloaded binaries and completed components set up.
daprd binary has been installed to C:\Users\#\.dapr\bin.
dapr_placement container is running.
dapr_redis container is running.
dapr_zipkin container is running.
Use `docker ps` to check running containers.
Success! Dapr is up and running. To get started, go here: https://aka.ms/dapr-getting-started

C:\Users\#>docker ps -a
CONTAINER ID   IMAGE                      COMMAND          CREATED         STATUS                     PORTS
                           NAMES
ee713c674604   kindest/node               ""               6 days ago      Exited (0) 292 years ago   80/tcp, 443/tcp, 127.0.0.1:52648->6443/tcp   kind-cluster-control-plane
03a0b3703797   daprio/dapr:1.14.0-rc.1    ""               9 minutes ago   Up 9 minutes               8080/tcp, 9090/tcp, 50005/tcp                dapr_placement
38979748ab7f   openzipkin/zipkin:latest   ""               9 minutes ago   Up 9 minutes               9411/tcp
                           dapr_zipkin
f82bcb0d8866   redis:6                    "redis-server"   9 minutes ago   Up 9 minutes               6379/tcp
                           dapr_redis

C:\Users\#>dapr uninstall --all
Removing Dapr from your machine...
Removing directory: C:\Users\#\.dapr\bin
Removing container: dapr_placement
WARNING: dapr_scheduler container does not exist
Removing container: dapr_redis
Removing container: dapr_zipkin
Removing directory: C:\Users\#\.dapr
Dapr has been removed successfully

Steps to Reproduce the Problem

Mac/Windows - Podman

Release Note

RELEASE NOTE:

mikeee commented 1 week ago

dapr init --runtime-version 1.14.0-rc.2
⌛  Making the jump to hyperspace...
ℹ️  Container images will be pulled from Docker Hub
ℹ️  Installing runtime version 1.14.0-rc.2
❌  Downloading binaries and setting up components...
❌  Error: statfs /Users/#/.dapr/scheduler/data/0: no such file or directory

@artursouza

v1.14.0-rc.2 CLI fails to initialise the scheduler - everything else seems to have been initialised. Running on Podman

olitomlinson commented 1 week ago

@artursouza @mikeee

I used brew to install rc.2 successfully! Thanks both!

dapr init --runtime-version 1.14.0-rc.2
⌛  Making the jump to hyperspace...
ℹ️  Container images will be pulled from Docker Hub
ℹ️  Installing runtime version 1.14.0-rc.2
↓  Downloading binaries and setting up components...
Dapr runtime installed to /Users/olivertomlinson/.dapr/bin, you may run the following to add it to your path if you want to run daprd directly:
    export PATH=$PATH:/Users/olivertomlinson/.dapr/bin
✅  Downloading binaries and setting up components...
✅  Downloaded binaries and completed components set up.
ℹ️  daprd binary has been installed to /Users/olivertomlinson/.dapr/bin.
ℹ️  dapr_placement container is running.
ℹ️  dapr_redis container is running.
ℹ️  dapr_zipkin container is running.
ℹ️  dapr_scheduler container is running.
ℹ️  Use `docker ps` to check running containers.
✅  Success! Dapr is up and running. To get started, go here: https://aka.ms/dapr-getting-started

philliphoff commented 1 week ago

Just an observation, but I had several CI builds for the .NET SDK fail due to the Dapr installation step failing due to the dapr_scheduler is not running error. It seems sensitive to timing--re-run it enough and you'll eventually get a success.

mikeee commented 1 week ago

I think my issue is now moving towards it being podman-specific now. Possibly an issue with the default mounts

mikeee commented 6 days ago

{"level":"warn","ts":"2024-07-10T16:11:53.454626Z","caller":"embed/config.go:679","msg":"Running http and grpc server on single port. This is not recommended for production."}
time="2024-07-10T16:11:53.453650444Z" level=info msg="Starting Dapr Scheduler Service -- version 1.14.0-rc.2 -- commit 3d30a6f3ae2191125ed9d3dcf0761077772e5de2" instance=48a60b6b1ed4 scope=dapr.scheduler type=log ver=1.14.0-rc.2
{"level":"info","ts":"2024-07-10T16:11:53.454714Z","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
time="2024-07-10T16:11:53.453786398Z" level=info msg="Log level set to: info" instance=48a60b6b1ed4 scope=dapr.scheduler type=log ver=1.14.0-rc.2
{"level":"info","ts":"2024-07-10T16:11:53.455061Z","caller":"embed/etcd.go:135","msg":"configuring client listeners","listen-client-urls":["http://localhost:2379"]}
time="2024-07-10T16:11:53.453933081Z" level=warning msg="mTLS is disabled. Skipping certificate request and tls validation" instance=48a60b6b1ed4 scope=dapr.runtime.security type=log ver=1.14.0-rc.2
{"level":"info","ts":"2024-07-10T16:11:53.455299Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.14","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.22.4","go-os":"linux","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":4,"member-initialized":false,"name":"dapr-scheduler-server-0","data-dir":"./data-default-dapr-scheduler-server-0","wal-dir":"","wal-dir-dedicated":"","member-dir":"data-default-dapr-scheduler-server-0/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"],"listen-client-urls":["http://localhost:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"dapr-scheduler-server-0=http://localhost:2380","initial-cluster-state":"new","initial-cluster-token":"etcd-cluster","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"periodic","auto-compaction-retention":"24h0m0s","auto-compaction-interval":"24h0m0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"warn","ts":"2024-07-10T16:11:53.455399Z","caller":"fileutil/fileutil.go:53","msg":"check file permission","error":"directory \"./data-default-dapr-scheduler-server-0\" exist, but the permission is \"drwxr-xr-x\". The recommended permission is \"-rwx------\" to prevent possible unprivileged access to the data"}
{"level":"info","ts":"2024-07-10T16:11:53.455487Z","caller":"embed/etcd.go:375","msg":"closing etcd server","name":"dapr-scheduler-server-0","data-dir":"./data-default-dapr-scheduler-server-0","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2024-07-10T16:11:53.455561Z","caller":"embed/etcd.go:377","msg":"closed etcd server","name":"dapr-scheduler-server-0","data-dir":"./data-default-dapr-scheduler-server-0","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
time="2024-07-10T16:11:53.454099166Z" level=warning msg="etcd client http ports not set. This is not recommended for production." instance=48a60b6b1ed4 scope=dapr.scheduler.server type=log ver=1.14.0-rc.2
time="2024-07-10T16:11:53.454288709Z" level=info msg="Dapr Scheduler is starting..." instance=48a60b6b1ed4 scope=dapr.scheduler.server type=log ver=1.14.0-rc.2
time="2024-07-10T16:11:53.454311377Z" level=info msg="metrics server started on :9090/" instance=48a60b6b1ed4 scope=dapr.scheduler type=log ver=1.14.0-rc.2
time="2024-07-10T16:11:53.454482075Z" level=info msg="Starting etcd" instance=48a60b6b1ed4 scope=dapr.scheduler.server type=log ver=1.14.0-rc.2
time="2024-07-10T16:11:53.454438544Z" level=info msg="Healthz server is listening on [::]:8080" instance=48a60b6b1ed4 scope=dapr.scheduler type=log ver=1.14.0-rc.2
time="2024-07-10T16:11:53.45566[18](https://github.com/dapr/go-sdk/actions/runs/9877355870/job/27280015154#step:9:19)16Z" level=info msg="Dapr Scheduler listening on: :50006" instance=48a60b6b1ed4 scope=dapr.scheduler.server type=log ver=1.14.0-rc.2
time="2024-07-10T16:11:53.455747345Z" level=info msg="Scheduler GRPC server stopped" instance=48a60b6b1ed4 scope=dapr.scheduler.server type=log ver=1.14.0-rc.2
time="2024-07-10T16:11:53.455775893Z" level=info msg="Running gRPC server on port 50006" instance=48a60b6b1ed4 scope=dapr.scheduler.server type=log ver=1.14.0-rc.2
time="2024-07-10T16:11:53.456109[19](https://github.com/dapr/go-sdk/actions/runs/9877355870/job/27280015154#step:9:20)5Z" level=info msg="Healthz server is shutting down" instance=48a60b6b1ed4 scope=dapr.scheduler type=log ver=1.14.0-rc.2
time="2024-07-10T16:11:53.456193842Z" level=fatal msg="error running scheduler: cannot access data directory: open /data-default-dapr-scheduler-server-0/.touch: permission denied\nfailed to serve: grpc: the server has been stopped" instance=48a60b6b1ed4 scope=dapr.scheduler type=log ver=1.14.0-rc.2

Seems to be a container mount permissions issue

dapr / cli