When scheduling an activity (and presumably in many cases), we create the command before we try to serialize the parameters. This can cause an issue where if it fails, the command is still in the activations commands list but incomplete. This is usually not a big deal since it's supposed to be a task failure which does not send commands and waits for code fix. But some users are not allowing these failures to be task failures and therefore you get a half-complete command going to Core.
To replicate, simply convert a serialization error for a parameter to an activity to an application error.
There are a couple of approaches to fix. We can take the more drastic approach of not creating the command until we have built it (changes a lot since we are doing a proto-lib-encouraged approach of creating the thing before populating it today). Or we can just make sure to just do things that can fail before creating the command.
Describe the bug
When scheduling an activity (and presumably in many cases), we create the command before we try to serialize the parameters. This can cause an issue where if it fails, the command is still in the activations commands list but incomplete. This is usually not a big deal since it's supposed to be a task failure which does not send commands and waits for code fix. But some users are not allowing these failures to be task failures and therefore you get a half-complete command going to Core.
To replicate, simply convert a serialization error for a parameter to an activity to an application error.
There are a couple of approaches to fix. We can take the more drastic approach of not creating the command until we have built it (changes a lot since we are doing a proto-lib-encouraged approach of creating the thing before populating it today). Or we can just make sure to just do things that can fail before creating the command.
Relates to and closes https://github.com/temporalio/sdk-python/issues/540.