The requester node attempts to schedule work on disconnected nodes resulting in the job never running

[frrist]

Due to changes here:

• https://github.com/bacalhau-project/bacalhau/pull/3768/files#diff-16bad1556b6ffdd42e5c57a235f78726a613fc46759a999fb23703ce93f92ecbR161-R163
• https://github.com/bacalhau-project/bacalhau/pull/3768/files#diff-3034b567889a6a7f1ea37dcf47d446e1fbf1a2928894a4fb7d9d2cfbafc857ecR93
• https://github.com/bacalhau-project/bacalhau/pull/3768/files#diff-6552869d10da40039d76eca490cf89bb20a2e4eb860c892ac1569b7a0eb88bebR110

Current proposal is to:

• make auto-approve work via a fix for: https://github.com/bacalhau-project/bacalhau/issues/3783
• modify the aforementioned code to only schedule on connected nodes.

cc @rossjones & @wdbaruni to weigh in on how the new event system introduced in https://github.com/bacalhau-project/bacalhau/pull/3772 can be used to force scheduling of executions when offline compute nodes come online again.

[wdbaruni]

What is the proposal here? I believe the default option now is to auto-approve nodes, and only schedule on approved and connected nodes. Is any of that still missing?

[frrist]

Yeah the only scheduling on connected and approved nodes is missing. Currently we schedule on disconnected node for some job types and ignore their approval state for other job types. frankly it's a bit of a mess:

• we need to address this TODO: https://github.com/bacalhau-project/bacalhau/blob/main/pkg/orchestrator/scheduler/batch_service_job.go#L158
• modify this section: https://github.com/bacalhau-project/bacalhau/blob/main/pkg/orchestrator/scheduler/daemon_job.go#L93
• modify this section: https://github.com/bacalhau-project/bacalhau/blob/main/pkg/orchestrator/scheduler/ops_job.go#L110

Or rather than modify, allow these aspects of scheduling to be configured.

Further we need to ensure that worked scheduled on an offline node runs when the node comes back online which we will need https://github.com/bacalhau-project/bacalhau/pull/3772 to do. e.g. the orchestrator could listen for connected events and create an evaluation to execute the work.

[frrist]

Another point to consider: How can we allow users to define different scheduling heuristics for compute nodes. e.g. nodes in a data center ought to have a more strict requirement on connectedness than nodes that are expected to go offline for longer periods of time (e.g. submarine compute nodes)

[rossjones]

@wdbaruni previously suggested adding another (a third) timeout in future which allows nodes to be offline for that long before being considered dead.