jkglasbrenner
commented
2 months ago I pushed a couple of updates. The first adds the missing Resource.is_deleted == False filter when fetching the plugin parameter types to build the job YAML. The second adds more detailed error handling to the worker execution scripts, and ensures that the worker will attempt to update the job status in the API no matter where the failure happens. In addition, all YAML validation errors should now be emitted as log messages in the worker, as opposed to just saying "Job YAML is invalid!", which should make it easier to debug things.
Request: Please do not squash, rebase and merge to dev only. The commits are different enough in scope and purpose that squashing will obscure the reasons for the various updates.
Summary of changes
The first commit adds support for the "boolean", "integer", "list", and "mapping" entrypoint parameter types that are already supported by the task execution engine. In addition, the
/workflows/jobFilesDownloadendpoint service has been updated to handle all supported types when creating the parameters.json and task engine YAML files. The "path" and "uri" types have been removed since they were treated the same as strings.The second commit adds the
env_varscontextmanager to the Dioptra SDK, which can be used to temporarily update environment variables in a safe and reversible way.The third commit updates the Worker execution logic to set a
__JOB_IDenvironment variable when executing task engine YAML so that its possible to access the job ID using plugins.The fourth commit adds a workaround to export the full list of registered plugin parameter types to the task engine YAML. THIS IS A WORKAROUND THAT VIOLATES IDEMPOTENCE/REPRODUCIBILITY! This workaround allows users to create and use parameter types that are only used indirectly, such as when defining a structured parameter type. This is a "hack" because the indirect parameter types are the latest available snapshots, not the snapshots that were associated with the plugins when the entrypoint was saved/updated. This is in contrast with the parameter types explicitly registered to the plugin task input and output parameters, which are linked to the entrypoint and job by their snapshot id instead of the resource id. This difference means that the task engine YAML files are not 100% reproducible, as any changes to an "indirect" plugin parameter type will be immediately reflected in subsequent download requests made to the job files workflow.
The fifth commit improves the error handling in the Dioptra workers in 2 ways. First, every function that has a possibility of encountering an error has been wrapped in a try/except block, which better ensures that any errors are logged and the job status is properly updated in the Dioptra API. Second, the YAML validation step has been modified to emit warning and error log messages instead of swallowing them silently, and in addition raise a ValueError exception when errors are encountered with a message that details the errors that were encountered. This update implements the request in #604.
The sixth commit is a fix to ensure that the exported job YAML sorts the input and output parameters for the plugin tasks according to their position index.