Pipeline command context chaining.

[tatarize]

Overview.

Vpype commands should have a context. Commands should have an input-context and an output-context. Commands should only work within their defined contexts. If an identically named command exists in a different context, this would cause no issues.

Each accepted context would have a defined associated datatype and could lead to several different output contexts. In many cases the contexts would stack on top of each other. This like with block-processors, but rather than only have nested vector-data contexts, each item in the stack could have a different context. This would work like a different set of vpype command acting on a different datatype.

Assumptions

I will be assuming that the default context for all commands is vpype that that is the default underlying context for all commands and the input and output context for all the current commands.

What we gain.

What this gives us is a much more generic pipeline architecture which can work for arbitrary document types.

First, commands only work from a given context. So let's say we add a command called 'image' which switches the context from vpype to image which is to say it has an input context of vpype and an output context of image. Note that if we called read on an image file the resulting context would be image by default. The data type for this context is pil image data within different layers. I can then provide a large series of commands which have input-contexts and output-contexts of image context. For example brightness, flip, mirror, invert, cymk-channelize, in effect we would have vpype but if vpype was written for image data rather than vector data. This would work seamlessly alongside the rest of vpype simply by loading an image rather than an svg.

We could also have a command like potrace (vector polygon tracer) which would take an input-context of image and an output-context of vpype taking the data we processed and manipulated with the image and exporting vpype data. Likewise rather than call image we could also have perhaps called render which takes in vpype data and rasterizes it to PIL image data.

And importantly we could have a write command that would only write our image data for this our context here.

Why this matters.

"But, wait, vpype is just for plotter stuff this would make the entire architecture super-generic and that is not needed for plotter-stuff!"

The importance here is that because our commands would all be context-relative we transition some compound-commands into their own mini-vpype context. We could have context-sensitive versions of write ( #116) and read. We could more easily solve the nasty habit within some earlier vpype commands to conform to the architecture with extra options and arguments to satisfy all the things you might want to do here within a single command. Most options could be implemented as commands within their own context (if needed).

Pseudo-Contexts already within Vpype

l- prefixed.

We already have some things like this, but it's hacky. Why are there lmove, lcopy and ldelete, and proposals for things like lreverse, or lshuffle? This is because the layer ordering is a context in its own right. We are making sloppy commands prefixed with l- because these commands are in the context of layer-order.

If we had layer-order as context we could call begin layers swap 2 5 copy 7 new where color=blue delete sort --travel end. I'm not sure of the best implementation here so I'm using begin and end and layers like a context-defining block. But, note this is actually going to modify the document at the layer-level. Here we have a series of layers commands:

• swap layers 2 and 5.
• copy layer 7 to a new layer
• we create a sub-context matching all layers (where) that fit the given statement (color=blue)
  • we call delete on the matching sub-context created by the where.
• The sort would be the layer-context version of linesort rather than sort within each layer we sort the layers themselves. We optionally do with with a --travel option which means we are optimizing the minimize travel between layers.

Write Types (and lack thereof)

We have implicit context within write you will notice the options for write have [SVG only] and [HPGL only] and add to this the large number of exporting formats I've written: DXF, Gcode, Embroidery formats. If write is only ever the same write command this will hobble forwards compatibility. There were previously some options here that got moved into their own commands. But, there's only some examples where this is a useful. There are several other write options that only become relevant within a context.

Implementation note. The write contexts here would be defined with the write filename.ext where we would delegate our write based on the extension result in an output-context based on what write specified. The options then would be after the write command. While the write stuff would be preserved for backwards compatibility write out9.svg would take in vpype (or maybe optionally a curved-paths) context and delegate the output context to write-svg (by detecting the filetype) we could then set flags with actual commands so color-mode layer would be a command and not an option, or paths and poly would switch between paths and polygons/polyline writing, etc. You could then specify a lot of different commands in the write-svg context and importantly plugins would also have access to this since they can define commands.

The actual write would only happen at the end of the context. write out9.svg paths ppi 72 grouplayers commit where write would detect the mime-type and then change the context to write-svg and only at the commit would the actual writing happen. This might be called by default when popping off the contexts from the context-stack as the default way to go from write-svg context to vpype.

Single-Layer contexts.

Generators have a single-layer context. You have to specify -layer # or --layer new to set this context. It tends to propagate between layers and you need to overtly set this again if you're dealing with a different such context.

Specifying a layer to interact with is specify a context. The context is the individual layer. Generators all work on within the context of a single layer. While this carries forward with preserving the layer selected this is just a method of contextualizing the commands. layer 3 line 1cm 1cm circle 0.5cm 0.5cm 0.3cm makes a lot more obvious what's going on. The command line and circle work on the single-layer context to add that data.

Implementation Note: Single layer context would be multi-layer contexts with 1 layer.

Multi-Layer contexts.

Likewise some contexts can have multiple layers these are typically things like layer processors. Commands like crop and translate can work within the context of multiple layers. Likewise we can easily see how a command like rect could be applied to a multi-layer context. Specifically the command adds the specified rect to all layers. So we add the same rectangle to each layer.

Block-processors like grid work within the scheme by defining a context, and pushing it to the stack. You'd actually have grid define a group of new layers in a multi-layer context and each command like line, circle or rect would add the circle the multi-layer context, which would be a grid. Also, start and end define a fresh vpype context which is then merged with the lower context. We could can also see how grid could be used to take an existing layer and split that layer into many layers with the given offset. circle 1cm 1cm grid --offset 2cm 2cm 3 4 could be made to take the context of the circle and fork that into a grid of values, rather than block process the each element of a grid. (this could also be done with a generator that reaches down in the stack and pulls up a layer from lower in the stack).

Curved-line context.

The plugin vpype-dxf had to define its own loader by converting all the commands over to a svgelement elements and then reuse commands like --quantization (see: https://github.com/tatarize/vpype-dxf/blob/8cd1595c83bebe07a1b1d4dc7e3393895ea272af/vpype_dxf/dread.py#L23) and keep in mind that that's not exposed API, I literally imported a _ function to hack that in there. With contexts quantization would convert a curved-line context to a standard vpype context. This would also allow some loaders that might also produce shapes like the ability to reuse internal code. Also commands like circle and ellipse also perform quantization which does the same thing but is also added as an option.

Open Questions

The transitions between contexts are not always obvious or necessarily implied. Clearly a command like layer 3,4,5,6 or where color=blue select could define a multi-layer context, and converts our current context into a context that consists of these layers. Just as grid would but it's not obvious how you would denote the ending of a block, implicitly. And if we must define these explicitly, it would be hard to maintain backwards compatibility.

In the example for how write would work with contexts it required a commit command. But, we could certainly imply this from the fact that we have a write context. Clearly the end of the code means we should close the context stack. Likewise commands like ellipse and read would produce curved-context objects, and while it's clear that quantization as a command would transition from curved-path contexts to straight-lined contexts and merged back with the vpype document. It's not clear if we could omit the command and simply imply quantization. vpype read mydocument.svg quantization 0.001mm layer 3 would be pretty obvious that quantization would set the quantization for the given curved-path context and clearly all of this data would get sent to the target layer 3 which would be our next context after that command, we could likewise not specify a layer and get a multi-layer context of loaded file data, but if we called vpype read mydocument.svg quantization 0.1mm layer 1,2,3 we would load the document, quantize into straight lines and put a copy of those lines into each of those layers. We can see how we could load an svg document into a grid and create a grid of copies of the loaded lines. But, could we do vpype read mydocument.svg layer 1,2,3? Obviously we could make the layer command that gets some curved-path data (what loading svgs would yield) put that into 3 different layers, but is there an obvious way to imply the quantization?

• It would seem like the if our current context does not have the given command that we should apply whatever terminal operations we have and merge the current context with the lower-context and pop this off. If both contexts have the same command we would need to do this explicitly.

While the backwards compatible idea of making vpype its own context is fine, there are clear places where this would get extremely kludgy like line --layer 3 sorts of things where it's really clear layer as a command defining the contexts of the layers within the document makes much more sense, we'd take --layer 3 and make that command more obviously the same as layer 3 line. We'd have a lot of these eyesores as there's clearly some rather easy simplifications with this scheme.

How exactly does splitting a context into a series of subcontexts work? For example if we used read to open different files. read *.svg this could, in theory, give us a multi-document context rather than single-document (global processor). Now, we could then do something akin to how layer-processor do the same commands on a layer-by-layer basis at a document-by-document process. Running every command for these contexts. And it's clear this would be really powerful, it's not clear how that works with regard to the context stack. Is there a way to imply the sort of for each when we are switching contexts?

• It would be obvious that we could call a command like split which could perform the same sort of process each of these contexts, and since it could be context sensitive, multi-document context split would work different than multi-layer context of split. This could take --sequential or --parallel --threads 4 since any split context would be basically done like functional programming.

Would this be possible with click or would click need to be peeled off and recoded? I coded my click-like parser when I implemented something similar in meerk40t, it's not clearly obvious if we could non-overlapping command contexts here.

[tatarize]

Conceptually the core idea behind block processes would be sort of expanded since it's the chief stacking of contexts. Though in that case we can see that begin pushes a fresh vpype context on the stack. The blockprocessor splits the context into a the relevant layer contexts and the end pops that context off the stack and merges the current context with the parent. If this was more generalized we could see how this would work in a highly generic sense.

We have a stack of contextualized data-types and commands can push and pop these from the stack in a block of commands the particular function would apply to all of them. This is basically a program. We have different commands based on the object we are manipulating. A command block which requires a blockprocessor is very much like a function call being applied to a series of subitems within the current context. If a particular command does not match any command in our current context, that context is popped and merged until we reach a context that does have that context.

We could then see that we load a default vpype context at the base level. read will usually push the new context of our loaded data to the stack. So read itself would give us a read context, which when merged with the lower level adds whatever data we loaded to our vpype context.. So if we call write we will get a resulting context of that particular write command. And in these defined contexts the pop-merge which would happen by default at the end of our usage, will do the saving.

So write file.svg paths nogroups the write command from vpype would load up the write-svg context within which paths and nogroups are commands. these could also be done as options in the write command but would generalize as calling those same commands within the context and could be done for backwards compatibility-sake.

So the command ldelete --layers 4,5,6 would be an in place equal to layer 4,5,6 delete where layer 4,5,6 would create a multi-layer context of those 3 layers. That context would have a delete command and we're good. Likewise we could do some generator commands to add a rect to each of those layers, etc. But, the same core procedure would be happening. Our overall actions have a stack of contexts. Some commands will push a new context to the stack, we then get other commands that are registered with that context. At the end we pop that context off and it is merged with the layer below it, much as the initial push to the stack either took in some of that data or didn't. Some commands would overtly pop a context off our stack, whereas other commands that do not match a registered for the command for a particular context would start popping contexts off the stack until it correctly matches a command for that context.

We could then have a command like circle which will give us a context of a single curved-path circle. The curve-path context would accept quantization as a command and maybe other things if plugins supported them. And then when that context is popped off because we get another command like line which is not a command within curved-paths, it will pop and merge that context by linearization of the circle into the layers of the current context. And execute the next command.

Split would also be a sort of special operation on the pypeline. It wouldn't be command but a stack operation, in that it would subdivide the current context into objects, this is sort of how a layer processor command works the command being run on that layer by layer, or how a block processor works with the block itself being applied each context.

This seems like it's solid enough to fully implement all the core api stuff in vpype, And you could easily expand it so that, for example, the read command could be used to load a dozen files, then split that documents parallel subcontexts of document which all the regular vpype commands would work. But, this sort of splitting you do with blockprocessors is kind of like apply this function to all objects within this object. After which pop_and_merge adding them to the context that was originally used to create that command.

This seems like it would work with a data-agnostic version of the pipeline vpype uses without being tied to the data. You'd have push, pop, and split to switch between contexts within the context stack, and these context switches could be performed by whatever command needs to do them, with the various datatypes defined elsewhere. A particular context-type would need to know how to push (what data gets accepted and how it uses that), how to split and into what would that context-type's data turn into, and pop which would remove the context from the stack merging it down with the other layer-type.

Then we define the different contexts and the transitions between the contexts. Then commands are registered on a per-context basis. So you could apply the same pypeline stuff to images and even transition between image and vectors sometimes, but we'd have say image as a base context and vpype as a sub-context so if we called brightness which would be an image command it would pop off the vector context. If we then called write we could only get the image write. Rather than writing in the topmost context that supports the write command which would often be at the document level. You could also see how you could have a write_all command for a documents outer context which would split into documents and write each of them sequentially.

We could also see how we could work with things like segments, curves, graphs, and images, and even deal with the interactions between these types. We could do something like crop circle 3mm 3mm 2mm it'd make a cropping context that would get a circle merged into, which would define the clippath and perform the clipping. The sky's the limit there. And it would be reasonable enough to implement the standard vpype api in the more generic pypeline.