flow step filter based on an expression aka grep

[brendanheywood]

eg any series of expression which results in a booleon to decide if each row should be kept or not

eg lets filter out any where user.suspended = 1

[brendanheywood]

I've just realized this step is technically redundant because anything you could do with a single filtering expression you can do with the case step with multiple filtering expressions. So what does everyone reckon?:

1) do we still do this to make it more obvious to flow authors when choosing a step 2) or ditch it and treat this as documentation task in the case step, and also tweak case so that it is valid to only have a single output

I'm leaning towards just improving case step

[keevan]

I prefer having a separate filter step:

• It's very clear from the step type, what the intentions are (even if you could replicate the behaviour using the case step).
• You could potentially extend the case step class (or move this core flow logic behaviour handling up to a common base class), adding filter specific rules / behaviours and it should just work. For instance, a filter would be a one-to-one connection instead of a one-to-many (case), and things that don't "pass the filter" will be ignored.. Which means you'd only expect one input (an expression)
• instructions are tailored around this expected behaviour / use case.
• Imagine having to explain to a client that something is actually not a bug, but instead was configured not in a way that would give you the expected outcome.
• It's easy to reason about since it's very common across many programming languages (JS, PHP, Python, etc.)
• Rules and validation specific to this step

It's similar reasoning to why you would split up large unwieldy methods into smaller portions, so they can be more easily understood and other parts reused for different purposes. I also feel some urge to change the terminology of "case step" to be a switch or branch step instead.

The alternative of course, is to pack only the functionality that dataflows need, because everything else can be done from core steps. This would fall into the territory of YAGNI and kind of where Golang stems from, in that the core language features are only those that are needed. Which is great for keeping things simple, but you end up writing a lot of the same thing to do more complex things, more expressive things. The cost being the tech debt to maintain the expressive alternatives involved.

Maybe later down the track - ideal, more polished product - I would like to see some sort of filter for all step types, that return steps relevant to the phrase or keyword chosen, taking into account the place the user wants the step to be added, which could be initially implemented simply using a list of defined tags for a step type (switch, case, filter, s3, copy) which each step author should use properly and to highlight relevance to use case.

[brendanheywood]

+1 for rename to switch https://github.com/catalyst/moodle-tool_dataflows/issues/399

I think from the authors perspective the discoverability is the main thing. If they can see a step called either grep or filter it's pretty obvious. So agree lets make it two steps