Workers should ignore unrecognized workflows and activities instead of closing them as failed

Copied from https://github.com/cschleiden/go-workflows/issues/251#issuecomment-2091237841 and https://github.com/cschleiden/go-workflows/issues/321#issuecomment-1961937528, I figured it should be its own issue.

Does it make sense for workers to intentionally skip workflows they don't know about themselves? Right now, the workflow is marked as failed. But if you had two deployments, one meant to handle workflows Foo and Bar, and another meant to execute workflow Baz, you'd want each deployment to ignore workflows intended for the other.

Also, supposing there are multiple worker instances in a scaled deployment, and one deploys to a new version minutes before the other. (blue-green deploys or merely slow instance rollout) You might have a time where the new version is creating (sub)workflow instance that the old version polls for but doesn't recognize. I believe the old version would immediately mark these unrecognized workflows into an error state; they'd never be executed.

If, due to some operational hiccup, the worker believes it cannot execute a workflow, that workflow should be left pending, right? Ideally testing would catch something like this from ever reaching prod, but it still seems risky for a durable execution tool to close the workflows without doing whatever work is meant to happen.

cschleiden / go-workflows

Workers should ignore unrecognized workflows and activities instead of closing them as failed #340