Import Commands - Githubissues

hey @tartarize! In the Ink/Stitch project there was a question raised, that I would like to transfer to you, since it also might be interesting for other use cases of pyembroidery.

When we import stitch files as color blocks, we get all the commands as they are written in the embroidery file. Now, that we want to transfer them into the svg representation, we are very interested into a file specific "translation" of the commands (e.g. jump jump jump = trim). I found the 'get_pattern_interpolate_trim' function, which is already very helpful to translate most of the trim commands, but I was wondering, if pyembroidery could deliver a more file-specific result? It would also be nice to have recognition for other commands such as the stop command, etc.

All the commands come off the files perfectly intact, or as good as I know how to get them. I think you've just advanced to the point that a one-size fit all solution like importing as color blocks isn't as helpful to you anymore. It's great to get people running right out of the gates, but it lacks some finesse. It might actually be reasonable and correct for you to strongly consider importing sequins, or caring whether the boundary between one color block and the next was a needle-change or a stop (in a format that can't tell stops from color_changes).

I saved all the data, the problem is you might want to actually loop through the data yourself and extract more of the information it saved. It doesn't confuse stops with color changes when they aren't ambiguous in the format itself.

inkstitch/pyembroidery#38 for example was asking this very thing. How much effort is needed to try and build low level stuff into higher level stuff. Even things like inkstitch/pyembroidery#22 is basically that. You could figure out from file that a frame eject is being performed there, if it jumps out to some place, then stops there, then jumps back into the design. That's a rare but totally something that could be figured out.

The stop commands are all preserved in place, and maintain whatever meaning they functionally have. The split with inkscape's version comes a before needle-changes were fully implemented but basically every bit of data is saved whether the stop literally means stop, color_change, or change_needle to some specific needle. The data all gets saved.

Most formats don't do that great of a job preserving stops that are not unambiguous with color changes. Some of the ones that do are the needle-change formats. So 10o, Barudan DAT, DSB, DSZ, KSM, TBF, U01.

I think there are other methods like PES does the same color twice which usually will cause a stop, but most of the Brother machines are single needles and need a switch each time anyway. So there's very little to interpret there. But, you might be able to delete the duplicate color and switch the color change for a stop in that instance. Which isn't something that is currently done. But, it seems like it would only apply to PES and should just be an option for the PES reader or something.

I'm not sure what else you think pyembroidery would need. As is, it saves every bit of data it can. And if there was a way to tell a stop from a color change it should have been done. I'm more than happy to work with you. But, when there was some fine-grained file-specific data, I saved that data. It's the reason why there's a bunch of differences between needle_change and color_change, largely because some formats actually gave additional information as to what needle you were changing to, and so I needed to preserve that information too.

Hey there! Thank you for taking the time for an answer.

I didn't mean the files are broken or deliver the wrong commands. Sorry, if that was the message I brought across.

It's just, that I thought it'd be nice for higher level programms to directly receive a "translated" version of the raw information that the input files are delivering.

I only spend very few time to understand a tiny bit of the way how embroidery file formats are written. So excuse me, if I have a wrong understanding of what is possible or not.

Let's take a little example... (dst,jef info taken from the edutech wiki)

Trim DST: 3-5 jumps in a row JEF: The trim command is a JUMP with 0 distance x or y. PES/PEC actually delivers trim commands from pyembroidery without modification etc.

Now we can easily get away by telling pyembroidery to interpolate trims whenever there are more than three jumps in a row. This might be sufficient for most users and all we need.

Concerning the stops I only had a very quick look at the pes format. It seems to have a "jump trim jump trim" combination to indicate the stop. I had no further investigation on this topic so far and I might be wrong by this observation.

But from what I am understanding, each format has it's own way to combine commands to express an other command. That was raising my thought, that it'd be nice if pyembroidery could have a function to translate the information of "multiple-command-commands". So there would be no need to make the interpretation of the incoming data at a later time?

I didn't mean the files are broken or deliver the wrong commands. Sorry, if that was the message I brought across.

No need for walking on eggshells with me. If I messed up, I messed up. If I didn't understand your point, I didn't understand. No worries. More than happy to hash this out.

The translated version of the raw information would certainly make sense and seems like it should be provided. If the machine would take 3 jumps as a trim, it's generally going to be correct to fix that. However, from Wilcom's software I got the distinct impression that this was a settable option. It seemed like it ranged from 2 to 5 jumps before invoking the trim. Also, and this is worth noting, both pyembroidery and several other software suites do a series of jumps namely (+2 +2), (-4, -4), (+2,+2) which clearly indicates a jump. This isn't however interpreted anywhere and seems like some low hanging fruit.

Let's take a little example... (dst,jef info taken from the edutech wiki)

Seems reasonable. I tried to make that edutech stuff as accurate as I could.

The JEF section seems like it might need some updating. Lex seemed to think three jumps of 0 distance would do it. Or rather 3 commands of any type and put it into the forked version that it should export 3 jumps of 0 distance to make sure it consistently trims. I haven't bothered to recheck his work yet. But, he initially accepted my interpretation then removed my change which was as follows:

        if ctrl == 0x02:
            if x == 0 and y == 0:
                out.trim()
            else:
                out.move(x, y)
            continue

However, he didn't then turn around and also correct the JEF reader for his interpretation. Namely you'd need to count the control sequence and when it was a sequence greater than 3 add in a trim. He added the trim to the writer, but no trim is able to be read from the reader.

I didn't get around to rechecking his work and updating the JEF code if accurate. My current code will work correctly if JEF really is a single move of 0 distance, for reads and writes.

This leaves DST. Which seems like it actually should come out of the reader with trims if every machine and program knows that some number of jumps is a trim. It is however configurable on the machine itself. So if you really need three non-trimmed jumps in a row you can set that value on the machine up to 4. And while it's true that get_pattern_interpolate_trim covers this remaining hole, it might actually be proper to add this to the reader itself. Especially for the easy interpolated jumps like (+2,+2), (-4,-4), (+2,+2) where it can't really be doing anything other than a trim.

And if Lex is right about JEF trims properly being 3 commands in a row (and 0 distance jumps are a padding command), then I should update my code there. Since three trims in a row isn't actually a thing, and the intent is that it should have 1 trim. And more properly if it really is just command operations as such, I should rather insert a trim at the same position the machine would have inserted a trim.

So rather than higher level interpretations and parsers these format specific trim commands should be corrected.

The DST reader should properly permit an option for how many jumps forces a trim, and default that value to 3. This would natively work like get_pattern_interpolate_trim just for DST files. Since the trim interpolation there is basically hardcoded in. Obvious padding jumps like (+2,+2), (-4,-4), (+2,+2) should be removed where it resulted in a trim operation.
The JEF reader should be double-checked if 3+ command sequence causes trims. Duplicate 0 movement jumps should not all be treated as trims. And triple command sequences even lacking a 0-movement jump might need a trim directly interpolated. And if the correction checks out, I should update the edutech wiki with that information. If the interpretation doesn't check out, my current code regarding JEF is correct and LEX simply removed the trim from the reader.

Other formats like PES, U01, have trims, but these are directly and explicitly given within the format, so they aren't a concern. But, the strong preference should be on the formats themselves interpolating the trim commands if they are appropriate. The mandate isn't for exact preservation of the raw data, but maximum preservation of information. That the machine that runs this format would call these events a trim, is information which is, at present, being lost. And as such is a bug, even if I gave tools enough to correct the problem.

Unless there's something else I'm missing.

Alright, checked and I'm pretty sure Lex is right. Jef Trims are caused by sequence of command elements when the machine is set to Manual with read trim commands on.

However this isn't strictly speaking enough information to solve whether a trim should exist or not. If you were doing something like fluff and wanted a really really long stitch it would exceed the number of commands in a row and cause the trim, however if the setting would have been off for that design rather than manual, commands=on it would have taken that correctly to be a bunch of long jumps without any trimming.

This tends to mean trim is a bit interpretation for Jef and there should be loading options. DST always tends to do that trim stuff. However, even there it's some degree of interpretation since the number of jumps is able to be configured.

I haven't uploaded it yet, but I have DST done. Gets rid of the extra stitches for invoking a trim and physically adds trims where they are correct to add. Also, lead me to fix a couple other things one of which is a perhaps a breaking change. I'll finish up with all the jef options with "auto" being the default.

However this isn't strictly speaking enough information to solve whether a trim should exist or not. If you were doing something like fluff and wanted a really really long stitch it would exceed the number of commands in a row and cause the trim,

Typically when something like this is an issue for puff (for instance a stitch length is greater then 12.7mm (on most machines, some are less)) there is a setting on the machine itself that tells the machine to ignore all trim functions (and that is throughout the design, not just the areas of puff).

Unless that is the simpler low hanging fruit option or it isn't able to be done within the file itself, I dunno.

The problem is the same file means two different things to the machine when it's loaded, depending on the settings in the machine. Since it's the same file, I can't tell those two situations apart without also including in the read parser a similar set of settings that the machine would have. It might be possible to post-process the file so you convert 3 commands in a row to a trim in this instance, but don't in that. But inkstitch doesn't have any of that.

So generally I just need some loading settings that would work like the machines default settings. For DST this means trimming at about 3 jump commands and JEF's auto trim option is any jump larger than 3mm by default. I don't have a good selection of the various machines to look at but, that is my understanding.

I think I recall you mentioning some machines have similar distance based settings for causing trims. So maybe I should add that to the DST files too, and just default it to None.

Since there's some trim interpolation done on the files, if my intent is to load them such that they look like they would look like when they sew out, I need some trim interpolation for the reading. Many of the formats have overt trims. Those that don't need to have the trims interpolated.

As is, if you don't run the function to get the interpolated trim commands you end up with some untrimmed files. Properly these should have the kind of default understanding of the machines that will run them, rather than all basically default to no-trim interpolation or requiring a reprocessing of the pattern that's loaded in a format that is already properly trimmed.

I've added in a good faith guess at the criteria most machines would use to interpolate as trims, and options to make that configurable. So rather than just the information in file I also do some default interpolation with regard to how the machine would do that. @kaalleen is right that I'm wasn't taking the machines into account enough, especially with regard to some of the interpolated stitch patterns. Now it should do that. I also fixed a bug Lex noted as an aside, namely that you can have two similar but different colors pigeon-hole into the same palette value in Jef and have the color change ignored.

I also added a versions element to the formats list. And simplified those broken options to be versions. So you if it has a writer and a 'versions' then you can ask for a particular version. This makes requesting all the types of PES outputs easier and extended headers for DST files. But moreover it would make exporting half a dozen different versions of gcode or exporting a file specifically crafted towards a particular machine much easier. And kinda standardizes the whole thing. I also corrected a dumb oversight I made having extension as a 1:1 thing. So the program calls .u01 the extension but doesn't know these could be .u02 or .u00 etc.

@kaalleen if you have any other specific requests for things it messes up without taking the machine into account I'd be happy to address those. The trims in .jef and .dst were the clearest oversight that fit your inquiry though. These now call an interpolate_trim() function in EmbPattern for the sorts of default interpolations we'd expect from an embroidery machine. While not perfect, this will catch those glaring cases. And it provides a solid methodology for doing other things like deciding that if you do a series of jumps to a distant spot, add a stop/color_change and then jump somewhere else, that that was likely a frame_eject.

I'd think there's some other solvable issues. Like PES/PEC stops are actually color_changes to the same color. While I dutifully load the colors and the color changes. I could add in an interpolation that says if I color change to the same color, delete the extra color and replace that color_change correctly with a stop. And double check that the reverse properly happens as well. That stops written to pes actually encode as double-color color changes.

In a number of formats like U01 and JEF these color changes to the same color does nothing and the machine simply ignores them. So this interpolation is specific to the filetype being loaded and should be added to the reader.

@kaalleen 1.4.1 does the interpolation for PES/PEC for stops. These are implemented in Brother by duplicating the color. Since the Brother machine stops and doesn't need a color change, it's a stop. Other like JEF, U01 won't stop there. So it's not acceptable for the general case, but is on the per-machine case. This should make it so that stop commands are saved and loaded in PES. They will come out in the file as double-color changes. But, will load up again as a stop command. And work on PES reading machines like they are stops. I'm not sure of any other cases where this happens. I implemented interpolate_frame_eject() but I wouldn't really put it anywhere. It searches for jump, stop, jump patterns and jump, stop, end patterns to call them frame_ejects.

@kaalleen, btw inkstitch/inkstitch#484 is actually caused by #7 in pyembroidery. Namely in pyembroidery the write routine (PyEmbroidery.py line ~554 has a bit that reads):

    if isinstance(stream, str):
        with open(stream, "wb") as stream:
            writer.write(pattern, stream, settings)

Which is a bug. Python 3, uses str as basestring and doesn't have a object called basestring. So when you check against str, it excludes unicode in python 2. It therefore fails that check and thinks the unicode string it was given is a stream and tries writing to it, but it's a string so it can't, therefore the error.

With a function something like:

def is_str(obj):
    try:
        return isinstance(obj, basestring)
    except NameError:
        return isinstance(obj, str)

And the offending bit of code.

    if is_str(stream):
        with open(stream, "wb") as stream:
            writer.write(pattern, stream, settings)

You'd also need it corrected in the read routine, or you won't be able to read perfectly valid unicode named file names.

Your fix in inkscape/inkscape#485 actually just patches over the issue. There's literally nothing wrong with unicode filenames. They should be used. Forcing them into UTF8 fixes the issue, but only through a kludge. The real bug is a pyembroidery one. Unicode filenames are valid and it doesn't save them. Which I identified back on Sep 5, 2018, but was permabanned around the Aug. 16th.

You'll hit the same issue on reads and it's kinda pointless, files with unicode characters are necessarily valid. Pyembroidery didn't process them correctly. They are valid ways to save files. Unless you want somebody chinese saving things in ascii, that sort of needs to be correctly rectified.

Thank you for having a look into the issue and taking your time to update pyembroidery. I think this might be helpful for everyone who is using pyembroidery for their projects. Also for the hint about the encoding issues we were having in Ink/Stitch. I didn't think about pyembroidery as the source of it. I'll inform lexelby about your update.

EmbroidePy / pyembroidery

Import Commands #62