Classify lines by colour into layers (`read`)

[ithinkido]

It would be very useful to be able to sort lines by a range of colours and/or brightness into layers. My end goal would then be to use the sorted layers to apply different hatching to the shapes so that the overlaid shapes create nice shading effects. At the moment it I have not been able to find a way of selecting lines or objects with similar colour / brightness thresholds so that the number of layers generated could be controlled to match the number of pens or effects desired In my case the shapes are separate closed polygons, but are layered on top of each other to achieve a final colour / shade

Updated: block_test.zip

[duncangeere]

I would also find this very helpful.

[abey79]

If I get it correctly, the idea would be to have a flag in read such that, instead of considering top-level SVG group, it would sort geometries by their styling attribute. How should attribute be pooled then? Each unique stroke colour matched to a different layer (disregarding anything else, stroke width, alpha, etc.)?

It feels tricky to do right because either you have control on how the SVG is made (and then why not just use top level groups). Or you don't. But then the possibilities are endless. Case in point: @ithinkido's file doesn't even have stroke styles (fills are styled instead). For a tool that is all about lines, it would be weird to consider fills.

A dirty hack could be to pool by unique content of style attribute (completely disregarding its actually content, nor the styling inherited from parent elements). This would work with that file, and possibly other situation, but it's really dirty and will fail in other situation (e.g inherited style, etc.)

[ithinkido]

I was imagining something like read group -- <option> --<threshold> Option would be a singular attribute and could include things like colour, brightness or style as these would be the most used values in plotting, and a treshold to control how many groups are outputed. In my case disregarding stroke width , alpha etc would not be and issue, and I suspect the majority of users would continue to modify the produced layers to suit thier needs I agree that my example file is not the best test candidate, but it helps to visualy understand the idea of overlaid lines or shapes that become difficult to group, I only see now that there is no stroke style assigned which is not ideal for a test case, but I think the idea is clear enough.

[abey79]

It would be useful to have some specific examples of workflows where using SVG's <g> in the first place is not an option.

[ithinkido]

Not quite sure I understand "... using SVG's in the first place is not an option."

I have updated my test file , the fill and stroke attributes are now both included. My specific workflow in this case would be to separate out the shapes using their stroke attributes into groups of similar lightness ( these could be put into layers ) . Once that would be done , I would later add hatching to the shapes based on the lightness. What I can also imagine is that others who would generate multi colour plots might like to confine the number of different pens used by controlled grouping of lines based on their colour attributes to layers.

There are two general selection tool ideas that I have found so far. Inkscape can select objects with identical attributes, Gimp has a magic wand function that can select by a range of attributes defined by thresholds but can not differentiate if objects are overlaid to create a new colour. The ideal would be a combination with the ability to select lines based on their attributes within a threshold. This could be used when plotting with different colours to achieve a combined colour effect. In the simplest form - Blue box over Yellow box = Green box. My plotter has 8 colours, Axi has theoretically unlimited, but realistically most users would be trying to use a limited number of pen options, so controlling grouping of the lines to match how many pens you have or want to use is the goal.

[abey79]

Not quite sure I understand "... using SVG's in the first place is not an option."

Sorry, the <g> disappeared because of formatting.

What I fail to understand: if you generate the SVG yourself and are able to add arbitrary formatting to your SVG, why can't you directly group them in the SVG and use the existing layer support?

[ithinkido]

I do not have the control over the generated svg in this case, and I have not been able to find a way of grouping similar ( not only identical) objects or lines or colours.

[abey79]

I do not have the control over the generated svg in this case

How do you set/change styling attributes then?

[ithinkido]

There is an inkscape extension to add fill attributes to stroke.

[tatarize]

You need to store the color of each individual layer. So that all layers have a color. This is nothing to do with -read that's overloading that command with stuff it doesn't need to do, but only it can do because only it has access to the colors.

If each layer has a color, and some metadata just a dictionary of some string stuff that gets ignored by like 99% of everything. In fact if you wanted you could dump the .values of the svg shape you read into there, maybe something down the line would care you set teddy_bear="True", on your path even though it's generally meaningless to basically everything (though fill='red') might come in handy..

In any event, you'd have a color on the layers. And you'd have a command to sort those layers. Tada! If you wanted to sort them based on the how similar they are to a defined color, you could easily test the color distance of from one color to the next color.

The nearest color in those layers get sorted first: lsort --color "black" --ascending

The svgelements code includes a really helpful function for this https://github.com/meerk40t/svgelements/blob/a30578b136f2c06dfdef0a39e2870de07e427b53/svgelements/svgelements.py#L1623 Basically if you define any two colors in svgelements you can request the distance. You also can get the distance_sq which is fine since it'll sort the same. There's also a bunch of properties like lightness, saturation, intensity, brightness, hue, blackness, luminance, luma and obvious ones like green, blue, red, and alpha. There's subtle differences between many those values in color theory.

There'd also be obvious sorting criteria like number of points, and stroke/pen-width.

[tatarize]

Some of this may be residual prior to svgelements when svg was a bigger and more daunting thing and you'd need to be more strict. But, really a grab bag of values() to shove all your metadata that just generally gets ignored, and some properties with defined types like your stroke, and pen_width defined on the layers and you just add in your lsort command and you're good.

[abey79]

As you suggest, adding a bunch of per layer metadata in vpype.Document would be no big deal, be it structured (color, penWidth, etc.) or not (values map). Further, as initially requested in the present issue, read would have to offer a choice between structuring layers by color in addition to top-level (as is currently the case). In this context, colour "closeness" could be useful.

Obvious consequences:

• show would use layer colours when colouring by layer (per path colouring is useful eg. to check linemerge results
• write would use layer colours for export
• a new color command should be added to manually set a layer's colour
• write --layer-label should have a formatter for colour

Open questions:

• What's the default colour for a layer created by anything else than read? (e.g. vpype circle 0 0 10 write output.svg)
• Why only colour? Stroke width could also be defendable criterion for layer sorting. Are there others? We'd need to make sure the complexity remains in check.
• What would be a repeatable way to map colours to layer ID, in a way that?
• What's a suitable CLI UX to deal with colours (color and write --layer-label)?

[tatarize]

• The default color of the primaries would be None. That is their current color. All other code should do with what they currently do. Unless that color is set to something in which case the functionality is overridden. If they want to specify something else they'd use a --color option or something. The other alternatives "black" gives them information they currently do not possess namely you couldn't tell black from default black which leads to random output in show. The other other alternative is directly random from the start, this is also an issue since it's conveying on the layer information they didn't already have, and you can't tell this random from a none which would result in random.

• You can sort by anything. You might even be able to shorthand a string that does sorting <color or >color. <red. <luma >pen_width, >pen_width,<color to sort ascending by pen width but subsort by color within that.

  • I think the complexity is only being increased by making read and write a sort of god-object since they are the only function with access to the initial shapes. And if I wanted to add in dxf reading and writing I'd basically be duplicating like 90% of the SVG functionality, and all the load commands. And then making a sort of strange dread dwrite functions to do that. The thing about the suggestion here is that we are sorting in another command. We aren't sorting in the read function because only the read function knows what colors things are. You can sort based on whatever and have different lsort plugins as needed.

• In theory the colors would be part of the layer that has colors. They would get their colors from their existence within that layer. If you moved them they wouldn't have that color anymore. They'd have LineCollections with colors, the blocks within that wouldn't map to anything.

• You could have a lot of other commands that work on the vp.ColorType like basic commands in darken or desaturate or assign_real_random_colors (bad name).

[abey79]

I think the complexity is only being increased by making read and write a sort of god-object since they are the only function with access to the initial shapes .[...] We aren't sorting in the read function because only the read function knows what colors things are. You can sort based on whatever and have different lsort plugins as needed.

This is both a very insightful comment, and contradicts the premises that colour (and other metadata) is attached to layer as opposed to individual paths.

I now see two approaches.

1. Metadata is attached at layer-level.

This is what I had initially in mind.

Pros:

• Much lower impact on the existing code-base (metadata added to vp.Document, can be ignored by most code so long a new doc are create with doc.empty_copy() to preserve said metadata (already happening now for page size).
• No impact on vp.LineCollection().

Cons:

• This doesn't capture the depth of, eg. SVG's possibilities, so the process is lossy.
• Metadata-based layer grouping must append in read, making its interface and implementation more complex.

2. Meta data is attached at path-level

Pros:

• This much simplifies the implementation of read (just attach meta data to each path and stick to the "standard" <g>-based layering).
• Metadata-based layer grouping can be implemented in a separate lsort command whose implementation and UX would be relatively easy to come up with.

Cons:

• Very significant changes in vp.LineCollection with widespread impacts throughout the codebase (including third-party projects).
• Adds complexity to all generators (primitives, etc.) with metadata-related UX.
• Layer-based meta data would still be required to be used as default for e.g. primitives.

---

Although the second approach's pros are very attractive, I don't think we should pursue this approach for the time being. The main reason is that the problem we are trying to solve (namely match SVG's path color to specific plotter's pen) is relatively niche. In 90% of the case, this is dealt with by the grouping structure of the SVG, and the first approach would be OK in the remaining cases. From this perspective, the widespread impact on both UX and code-base isn't warranted. Much of the success of vpype has to do with the simplicity of its data model, and I'm reluctant to make it more complex unless there are strong reasons to do so.

[tatarize]

I was and am advocating for 1. If you opted for 2 it simplifies things while complicating them. You don't need groups at all if you opted for 2. You could flat file the entire thing and have metadata laden lines.

I don't see what the point of the LineCollection is if not to hold the metadata about the lines they contain. The sort would not necessarily result in the same number of layers. You would sort everything as a flat file and group them according to adjacent metadata. So if you had 10 line blocks in 3 layers and you sorted them based on their pen width or their color you'd get the same 3 layers sorted based on that criteria since it wouldn't change anything.

However, if you, however sorted based on the number of points, you could end up with 30 different layers, with 1 line block each, since the layers would be broken up. You'd have layers with identical metadata stacked such that they preserve their order, at the cost of dividing up the layers.

I don't think the lines should be complicated. I think all metadata goes to the layers, with some of it could be called as a type. In elements I parse a few smaller things like fill and stroke and transform and put those as solved values on the object but, when most things get tossed into values. If you wanted them you could fiddle around in there and get out any dash array that might exist or whatever, but it certainly wouldn't be required.

1. Cons
   • This doesn't capture the depth of, eg. SVG's possibilities, so the process is lossy.
     • The process is already lossy. And it's likely a good thing. You're not dealing with shapes etc. And you're getting better results out of constraining the types. So you have documents which are individual files. They might have some file level metadata what the original file was, what the command being issued was, whether this has already been loaded or not, etc. And layer level metadata that gives specifics that are true for all the objects it contains. Most importantly pen-width and color. Then you have your lines which are simple pointlist structures that merely contain a list of points in sequence. You can ensure everything using these classes would have at most 4 levels. Assuming you have pipeline level view of multiple documents, document level objects, layer level objects, and lines. And pipeline isn't really a thing, I just sort of want it to be. Everything is well defined and gives it a proper level. SVG allows metadata on everything, lines, layers, documents, this sort of limited nested hierarchy tends to capture a lot of different file types.

• Metadata-based layer grouping must append in read, making its interface and implementation more complex.
  • You could make 1 layer for 1 object and be kinda simple that way. SVG lets any number of levels exist, embroidery is a series of commands one of which tends to be change the thread and the series of points and colors that way is a sort of increased hierarchy. You don't need to do the harder or weirder grouping.

Pros:

• This much simplifies the implementation of read (just attach meta data to each path and stick to the "standard" -based layering).
  • SVG allows a lot of stuff to be put on groups and nested groups all the properties are configurable at each layer. Svgelements just gives you the list of relevant values because it's much easier than parsing all that stuff yourself. You could still just put a path in a group and flag that group with the metadata you would have put on the group. The only reasons this would matter is joining groups together based on their colors and pen_width.
• Metadata-based layer grouping can be implemented in a separate lsort command whose implementation and UX would be relatively easy to come up with
  • You could sort at the layer level too, you'd just need to split and combine layers more than you currently do. The same is true for the line-block level one too. You'd just basically treat all objects flat and then make copies of their groups if they end up splitting, etc.

[abey79]

Oh, I think there is a big misunderstanding here, which is now obvious from the bad wording of this issue's title.

OP's goal is not to reorder existing layers according to some criterion (which would indeed be very easy as soon as some metadata is attached to layers), but rather to create layers (and distribute paths within them) according to some criterion. For example, for a read on a SVG with some red paths, some blue paths and some black paths, regardless of how they are structured in the SVG, all red paths should end up in layer 1, all blue paths should end up in layer 2 and all black paths should end up in layer 3. That would have to happen in read, or would require strategy (2) above. Again, desirable feature would be to group similar colour in the same layer according to some threshold, or use entirely different criterion like stroke width, etc. Or combination thereof.

[tatarize]

You could absolutely do that in 1. You would load all the metadata layers and anytime you have a change in the metadata along the color or pen_width/stroke_width value you would create a new layer and put that pointseries in that different layer with different metadata. Making layers as needed. Then you might well create 10 layers or so with each one containing a couple different pointseries objects. You then sort that based on the color, which would give you 3 different layers and only break the initially loaded ordering when specified specifically to do so. And not in the read command itself.

[tatarize]

The program I wrote that does the embroidery drawing works exactly like 1. It has ColorLayers and PointLayers (pardon the camelcase, it's Java). And will organize things into these the layers. If a Pointlayer is moved from one color to another, it'll change color. But the order is given exactly in depth first order. So the first PointLayer in the first ColorLayer is first. And if you need to change that color, you would add in another ColorLayer in the 0th position and insert that point within that ColorLayer. It's not too hard to track things, you can recolor entire sets of lines in a single command, and changing the color of a single line layer would just bifircate the color layer and insert another one, between them. And you can also rather easily merge these if that is needed. And most of that wouldn't be needed. All you actually need is to give layers colors and pen_width, and maybe do some stuff to help reorganizing the layers if you're going doing something fancy but those don't usually come and it's not hard when they do.

[tatarize]

Metadata-based layer grouping must append in read, making its interface and implementation more complex.

I misread that the first time, yeah, you couldn't just dump everything in the same line collection, you'd make a series of line collections in a read operation each one classified based on their color and pen_width already. Since that's how they'd have that information. If you wanted you could do the current method and put them all into a single LineCollection with None for pen_width and color. You would need to do something slightly different than throwing all the colors and and pen_widths out the window.

[abey79]

Work-around possible with the following inkscape plug-in: https://wiki.fablabchemnitz.de/display/IFM/Styles+To+Layers