✨ Updated workflow documentations as per issue #47
feat: Updated workflows documentation for issue #47 (#51):
Removed nested_task and added subworkflow
Added methods into examples so that user doesn't have to search for them
Updated CI with -y flag on job: unit-conda step: Install built package to save some seconds on awaiting input
📚 docs: Added issue templates
docs: Added issue templates. (#50):
Bug report
Feature request
Improvement of existing functionality
🛠 Fix/31 return tuple
Fix/31 return tuple (#49):
This PR enhances the tuple[float, float] support in Dewret, initial implementation issues encountered during testing.
Changes
Introduced tuple[float, float] support to Dewret.
Encountered unforeseen errors during test implementation.
📚 Feat/33 documentation tidy up
Feat/33 documentation tidy up (#44):
This PR improves the Quickstart.md documentation for Dewret as discussed in issue #33. The update creates an introduction to Dewret and attempts to answer the question Why use Dewret?
Changes
Introduction: Added a detailed description of Dewret, highlighting its purpose, features, and supported renderers (Snakemake and CWL).
What are Workflows?: Explained the concept of workflows.
Highlighted Dewret's unique features.
Installation for Pure Users
🛠 Readded snakemake with fixed mypy issues.
Readded snakemake with fixed mypy issues. (#43):
This Pull Request adds a custom snakemake renderer that produces a workflow in snakemake. This will be an example renderer for a tutorial on how to create a custom renderer in dewret. The renderer is tested with most of snakemakes quirks to produce a workflow with minimal tweaks to be executable. I've added # TODOs inside the renderer for possible features which I will also list here
Changes
Added SnakemakeRenderer class to examples as it needs approval - if approved I can move it to renderers dewrete.renderers.
Split each snakemake rule into separated blocks in:
InputDefinition: Which contains the input and parameters blocks. I've combined the params and input blocks into InputDefinition to eliminate redundancy. The input block in Snakemake specifies the required input files for a rule to run, while the params block defines the actual arguments for the rule. Both blocks required similar processing, so merging them seemed more efficient.
OutputDefinition: Which defines the output block of snakemake - Output block defines what files need to exist in order for the rule to be considered finished.
RunDefinition: Which defines the run block of snakemake - Run block defines what processes are needed to run (e.g. dewret Tasks).
Added and example using the snakemake renderer.
Motivation
To be used as an example custom renderer for a tutorial on how custom renderers are created. The Snakemake language is widely used in the bioinformatics community for creating reproducible and scalable workflows so the snakemake renderer was a good choice for both usability and an example.
Possible improvements
Feature: add a rule all: In snakemake if you want to run multiple connected rules you have a rule all with the input definition being the last task's output to make sure all of the rules run.
Feature: add tests.
Refactor: better way to handle the types. to_snakemake_type methods conversion of types could be imrpoved.
Refactor: introduce better error handling.
Refactor: better way to handle input/output files.
Issues
pre-commit hooks don't take into account my local environment and show a variety of unfixable errors.
task() annotations of a workflow must be removed before executing snakemake generated workflow.
🛠 Revert "Example snakemake renderer (1-48)
Revert "Example snakemake renderer (1-48)" (#42):
Reverts flaxandteal/dewret#32 as this breaks the build - from this CI job it looks like the branch needed rebased onto release/0.9.1 from the last 3 weeks, and mypy issues fixed.
In particular, RunDefinition and StepDefinition signatures need fixed to match - then re-PR please and we can put it back in once all green.
🛠 Fix subworkflows and globals
Fix subworkflows and globals (#41):
What does this PR do?
Fixes the ability of subworkflows to use global factories and constants.
🛠 Nested tasks should not get globals as arguments:
Nested tasks should not get globals as arguments (#40):
What does this PR do?
Only passes real arguments, not globals, to a nested task / subworkflow when it is called.
✨ Add factories
Add factories (#38):
What does this PR do?
Adds an initial implementation of input factories.
Pulling this in, on the basis that we can follow up with a second iteration if needs be.
🛠 Fix: minor type issues using raw or compound types
Fix: minor type issues using raw or compound types (#37):
What does this PR do?
Solves minor issues when subworkflows handle types that are not an instance of type
🛠 Hug not installable
Hug not installable (#36):
What does this PR do?
Removes any dependency on hug which is breaking CI.
✨ Example snakemake renderer
Example snakemake renderer (1-48) (#32):
Summary
This Pull Request adds a custom snakemake renderer that produces a workflow in snakemake. This will be an example renderer for a tutorial on how to create a custom renderer in dewret. The renderer is tested with most of snakemakes quirks to produce a workflow with minimal tweaks to be executable. I've added # TODOs inside the renderer for possible features which I will also list here
Changes
Added SnakemakeRenderer class to examples as it needs approval - if approved I can move it to renderers dewrete.renderers.
Split each snakemake rule into separated blocks in:
InputDefinition: Which contains the input and parameters blocks. I've combined the params and input blocks into InputDefinition to eliminate redundancy. The input block in Snakemake specifies the required input files for a rule to run, while the params block defines the actual arguments for the rule. Both blocks required similar processing, so merging them seemed more efficient.
OutputDefinition: Which defines the output block of snakemake - Output block defines what files need to exist in order for the rule to be considered finished.
RunDefinition: Which defines the run block of snakemake - Run block defines what processes are needed to run (e.g. dewret Tasks).
Added and example using the snakemake renderer.
Motivation
To be used as an example custom renderer for a tutorial on how custom renderers are created. The Snakemake language is widely used in the bioinformatics community for creating reproducible and scalable workflows so the snakemake renderer was a good choice for both usability and an example.
Possible improvements
Feature: add a rule all: In snakemake if you want to run multiple connected rules you have a rule all with the input definition being the last task's output to make sure all of the rules run.
Feature: add tests.
Refactor: better way to handle the types. to_snakemake_type methods conversion of types could be imrpoved.
Refactor: introduce better error handling.
Refactor: better way to handle input/output files.
Issues
task() annotations of a workflow must be removed before executing snakemake generated workflow.
📚 Clarify input behaviour
Clarify input behaviour (#27):
What does this PR do?
The fundamental problem it solves is that, as per feedback, when a raw value is passed as an argument to a function, it should not simply be captured as a default value for that function call, but be turned into a global parameter. However, this behaviour does not make sense for raw (literal) variables inside tasks/nested tasks. In practice, this turned out to be quite a major change with a number of corner cases.
Consequently, this PR includes the following:
subworkflows: nested tasks that are not transparent to the outside workflow, effectively having a scope and preventing literal values in nested tasks (when marked as "subworkflows" from bubbling up to the global parameters).
parameter-naming: unnamed parameters are given names that should be predictable, but will remain distinct from each other. A potentially undesirable sideeffect is that two tasks called with the same raw variable (i.e. one not declared as a parameter) will always have separate defaults, even if they were originally the same static variable. There does not appear to be a straightforward (non-convoluted/brittle) way around this.
the renderers can return multiple (related) workflows if subworkflows are discovered.
📚 Add glossary
Add glossary (#25):
Begins a glossary.
📚 Explain globals input
Explain globals input (1-22) (#23):
Adds a brief explanation around the parameters section to the documentation, explaining how context is handled.
📚 Documentation refinements
Documentation refinements (#18):
This PR implements a number of suggested improvements to the documentation. It also modifies the publication of docs to occur only on push of tags rather than on push for any branch.
✨ Better error messages
Better error messages (#15):
Implements several closely related features:
error messages show location of call, not lazy-evaluation, if possible
custom type errors are more specific
examples in the custom error messages
✨ Can we make dewret not attrs-only?
Can we make dewret not attrs-only? (#14):
Can we swap attrs for something else, if desired?
Following the discussion in #10, rather than doing a general purpose layer for returned types, this implements an either/or approach, where either dataclasses or attrs may be used to specify a multi-part result. It does not touch attrs in the dewret internals, and so (for now) it remains a dependency.
✨ Check argument list is complete
Check argument list is complete [1-25] (#13):
Closes #11 - provides basic Python-valid checking of task signatures in calls. Both directly, and within a nested task.
This also adds a series of minor CI fixes that arose during implementation, to ensure tests and static checking is thorough and passes.
Changelog
✨ Updated workflow documentations as per issue #47
feat: Updated workflows documentation for issue #47 (#51):
📚 docs: Added issue templates
docs: Added issue templates. (#50):
🛠 Fix/31 return tuple
Fix/31 return tuple (#49): This PR enhances the tuple[float, float] support in Dewret, initial implementation issues encountered during testing.
Changes
📚 Feat/33 documentation tidy up
Feat/33 documentation tidy up (#44):
This PR improves the Quickstart.md documentation for Dewret as discussed in issue #33. The update creates an introduction to Dewret and attempts to answer the question Why use Dewret?
Changes
🛠 Readded snakemake with fixed mypy issues.
Readded snakemake with fixed mypy issues. (#43):
This Pull Request adds a custom snakemake renderer that produces a workflow in snakemake. This will be an example renderer for a tutorial on how to create a custom renderer in dewret. The renderer is tested with most of snakemakes quirks to produce a workflow with minimal tweaks to be executable. I've added # TODOs inside the renderer for possible features which I will also list here
Changes
SnakemakeRendererclass toexamplesas it needs approval - if approved I can move it to renderers dewrete.renderers.Motivation
To be used as an example custom renderer for a tutorial on how custom renderers are created. The Snakemake language is widely used in the bioinformatics community for creating reproducible and scalable workflows so the snakemake renderer was a good choice for both usability and an example.
Possible improvements
to_snakemake_typemethods conversion of types could be imrpoved.Issues
🛠 Revert "Example snakemake renderer (1-48)
Revert "Example snakemake renderer (1-48)" (#42): Reverts flaxandteal/dewret#32 as this breaks the build - from this CI job it looks like the branch needed rebased onto
release/0.9.1from the last 3 weeks, and mypy issues fixed.In particular,
RunDefinitionandStepDefinitionsignatures need fixed to match - then re-PR please and we can put it back in once all green.🛠 Fix subworkflows and globals
Fix subworkflows and globals (#41):
What does this PR do?
Fixes the ability of subworkflows to use global factories and constants.
🛠 Nested tasks should not get globals as arguments:
Nested tasks should not get globals as arguments (#40):
What does this PR do?
Only passes real arguments, not globals, to a nested task / subworkflow when it is called.
✨ Add factories
Add factories (#38):
What does this PR do?
Adds an initial implementation of input factories.
Pulling this in, on the basis that we can follow up with a second iteration if needs be.
🛠 Fix: minor type issues using raw or compound types
Fix: minor type issues using raw or compound types (#37):
What does this PR do?
Solves minor issues when subworkflows handle types that are not an instance of
type🛠 Hug not installable
Hug not installable (#36):
What does this PR do?
Removes any dependency on
hugwhich is breaking CI.✨ Example snakemake renderer
Example snakemake renderer (1-48) (#32):
Summary
This Pull Request adds a custom snakemake renderer that produces a workflow in snakemake. This will be an example renderer for a tutorial on how to create a custom renderer in dewret. The renderer is tested with most of snakemakes quirks to produce a workflow with minimal tweaks to be executable. I've added # TODOs inside the renderer for possible features which I will also list here
Changes
SnakemakeRendererclass toexamplesas it needs approval - if approved I can move it to renderers dewrete.renderers.Motivation
To be used as an example custom renderer for a tutorial on how custom renderers are created. The Snakemake language is widely used in the bioinformatics community for creating reproducible and scalable workflows so the snakemake renderer was a good choice for both usability and an example.
Possible improvements
to_snakemake_typemethods conversion of types could be imrpoved.Issues
📚 Clarify input behaviour
Clarify input behaviour (#27):
What does this PR do?
The fundamental problem it solves is that, as per feedback, when a raw value is passed as an argument to a function, it should not simply be captured as a default value for that function call, but be turned into a global parameter. However, this behaviour does not make sense for raw (literal) variables inside tasks/nested tasks. In practice, this turned out to be quite a major change with a number of corner cases.
Consequently, this PR includes the following:
📚 Add glossary
Add glossary (#25): Begins a glossary.
📚 Explain globals input
Explain globals input (1-22) (#23): Adds a brief explanation around the parameters section to the documentation, explaining how context is handled.
📚 Documentation refinements
Documentation refinements (#18): This PR implements a number of suggested improvements to the documentation. It also modifies the publication of docs to occur only on push of tags rather than on push for any branch.
✨ Better error messages
Better error messages (#15): Implements several closely related features:
error messages show location of call, not lazy-evaluation, if possiblecustom type errors are more specificexamples in the custom error messages✨ Can we make dewret not attrs-only?
Can we make dewret not attrs-only? (#14): Can we swap attrs for something else, if desired?
Following the discussion in #10, rather than doing a general purpose layer for returned types, this implements an either/or approach, where either
dataclassesorattrsmay be used to specify a multi-part result. It does not touchattrsin the dewret internals, and so (for now) it remains a dependency.✨ Check argument list is complete
Check argument list is complete [1-25] (#13): Closes #11 - provides basic Python-valid checking of task signatures in calls. Both directly, and within a nested task.
This also adds a series of minor CI fixes that arose during implementation, to ensure tests and static checking is thorough and passes.