After a while ( about an hour). The system becomes unstable.
In the controller log:
[2024-05-21T13:00:09.119Z] [ERROR] [#tid_kDcLqa1uLqzR7GhLLdejqhGeS3mbeHQo] [] Failed to recreate queue for dataspace/ncarpark/carpark@0.0.1, no scheduler endpoint available
Also seeing this in the controller log:
[2024-05-21T13:00:06.174Z] [WARN] [#tid_kDcLqa1uLqzR7GhLLdejqhGeS3mbeHQo] [] The whisk/queue/dataspace/dataspace/carpark/carpark/leader is deleted from ETCD, but there are still unhandled activations for this action, try to create a new queue
In the scheduler log I am seeing this:
[2024-05-21T13:00:09.876Z] [WARN] [#tid_sid_unknown] [EtcdWorker] a lease is expired while registering an initial data whisk/queue/dataspace/dataspace/carpark/carpark/leader, reissue it: io.grpc.StatusRuntimeException: NOT_FOUND: etcdserver: requested lease not found
And also:
[2024-05-21T13:00:10.195Z] [WARN] [#tid_sid_unknown] [EtcdWorker] a lease is expired while registering an initial data whisk/scheduler/0, reissue it: io.grpc.StatusRuntimeException: NOT_FOUND: etcdserver: requested lease not found
Environment details:
Steps to reproduce the issue:
Provide the expected results and outputs:
Provide the actual results and outputs:
After a while ( about an hour). The system becomes unstable.
In the controller log:
Also seeing this in the controller log:
In the scheduler log I am seeing this:
And also:
Not sure where to look for this issue