Min. Embroidery Formats for Stock Store

[wwderw]

When I do LC size designs the following are formats that I like to have for sell in my stock design store:

1. DST
2. PES
3. VP3
4. JEF
5. EXP

The 2 formats that I mentioned being out of whack in my comment for the PR, those were with bringing them into EmbroiderModder and into TruSizer.

The others appeared as they should, but I do not know if they stitched out as they should as I didn't test that.

[lexelby]

PR: #198

Awesome, thanks!

VP3 is pretty close but some of the color blocks seem to be offset in my test design. JEF is a horror show. EXP is bad but may not be too hard to fix.

[lexelby]

EXP: #203 JEF: #204 VP3: #205

[X3msnake]

Awsome work you are doing here guys. :)

[tatarize]

I might be finishing up my current project pretty soon and was wondering which way you wanted to go for these? Did you want an audit of the libembroidery code to fix up the errors there. Or to actually just code these formats into python?

Most of the serious functional awesomeness of the libembroidery work is simply by establishing the read and write of the files it does a good job outlining the actual format. But, often checking various parts require a bunch of rather tedious work comparing various hex elements for various outputs, and proper files found in the wild.

I recently wrote out a pretty good cursory first draft overview of many of those formats on the edutech wiki (the PES info is a copy of the wiki on fno7's libpes work I strongly contributed to). https://edutechwiki.unige.ch/en/Embroidery_format_DST https://edutechwiki.unige.ch/en/Embroidery_format_EXP https://edutechwiki.unige.ch/en/Embroidery_format_JEF https://edutechwiki.unige.ch/en/Embroidery_format_PES

A quick look will properly show you that the formats aren't actually wildly complex (reading such a file is often downright trivial). Most of the work is ensuring strong compatibility with something that is ultimately kind of jiggered from understanding various export utilities and files kicking around. There's not a lot of code that needs transcoding. There's just a strong need to test that code against itself and keep it as simple and reasonable as possible. But, once the reverse engineering has been done, coding this up in other formats is not actually that hard.

I do think that the MobileViewer java code is better than libembroidery code, as reading a file into move(), trim(), end(), color_change(), and stop() has a lot of notable benefits, and using the pixel-space to embroidery space at 1:1 is clearly beneficial as well as implementing coordinate system y-flips etc, during the reads and writes is generally more proper.

[tatarize]

DstReader.py

def getbit(b, pos):
    return (b >> pos) & 1;

def decodedx(b0, b1, b2):
    x = 0;
    x += getbit(b2, 2) * (+81);
    x += getbit(b2, 3) * (-81);
    x += getbit(b1, 2) * (+27);
    x += getbit(b1, 3) * (-27);
    x += getbit(b0, 2) * (+9);
    x += getbit(b0, 3) * (-9);
    x += getbit(b1, 0) * (+3);
    x += getbit(b1, 1) * (-3);
    x += getbit(b0, 0) * (+1);
    x += getbit(b0, 1) * (-1);
    return x;

def decodedy(b0, b1, b2):
    y = 0;
    y += getbit(b2, 5) * (+81);
    y += getbit(b2, 4) * (-81);
    y += getbit(b1, 5) * (+27);
    y += getbit(b1, 4) * (-27);
    y += getbit(b0, 5) * (+9);
    y += getbit(b0, 4) * (-9);
    y += getbit(b1, 7) * (+3);
    y += getbit(b1, 6) * (-3);
    y += getbit(b0, 7) * (+1);
    y += getbit(b0, 6) * (-1);
    return -y;

def stop():
    print("STOP")

def changeColor():
    print("CHANGE_COLOR")

def move(dx, dy):
    print("JUMP", dx, " ", dy)

def stitch(dx, dy):
    print("STITCH", dx, " ", dy)

def read(file):
    with open(file, "rb") as f:
        f.seek(512)
        while True:
            byte = f.read(3)
            if len(byte) != 3:
                break;
            dx = decodedx(byte[0], byte[1], byte[2]);
            dy = decodedy(byte[0], byte[1], byte[2]);
            if ((byte[2] & 0b11110011) == 0b11110011):
                stop();
            elif ((byte[2] & 0b11000011) == 0b11000011):
                changeColor();
            elif ((byte[2] & 0b10000011) == 0b10000011):
                move(dx, dy);
            else:
                stitch(dx, dy);

read("C:\\Users\\Tat\\PycharmProjects\\pyembroidery\\tree.dst")

Obviously such things needs a bit more stuff like a datastructure to actually hold all the data we read in triggering the stop(), colorchange(), move(), stitch(), some metadata and some colors, some threads, and some thread metadata. And there's a lot of important work done in transcoding files into a stabilizing middleformat. (DST for example calls 3x JUMP and if so needs a TRIM, and a bunch of jumps in a row doesn't actually matter, you can ignore the jumps and jump to the final destination.) And there might be some more preferred Pythonic way of doing such things that I wouldn't know. And some of these things might get progressively easier with lambda functions.

[tatarize]

It also might allow for good readability and modifications. For example, here's the only open source DST reader that registers sequins.

def getbit(b, pos):
    return (b >> pos) & 1;

def decodedx(b0, b1, b2):
    x = 0;
    x += getbit(b2, 2) * (+81);
    x += getbit(b2, 3) * (-81);
    x += getbit(b1, 2) * (+27);
    x += getbit(b1, 3) * (-27);
    x += getbit(b0, 2) * (+9);
    x += getbit(b0, 3) * (-9);
    x += getbit(b1, 0) * (+3);
    x += getbit(b1, 1) * (-3);
    x += getbit(b0, 0) * (+1);
    x += getbit(b0, 1) * (-1);
    return x;

def decodedy(b0, b1, b2):
    y = 0;
    y += getbit(b2, 5) * (+81);
    y += getbit(b2, 4) * (-81);
    y += getbit(b1, 5) * (+27);
    y += getbit(b1, 4) * (-27);
    y += getbit(b0, 5) * (+9);
    y += getbit(b0, 4) * (-9);
    y += getbit(b1, 7) * (+3);
    y += getbit(b1, 6) * (-3);
    y += getbit(b0, 7) * (+1);
    y += getbit(b0, 6) * (-1);
    return -y;

def stop():
    print("STOP")

def changeColor():
    print("CHANGE_COLOR")

def move(dx, dy):
    print("JUMP", dx, " ", dy)

def sequin(dx, dy):
    print("SEQUIN", dx, " ", dy)

def stitch(dx, dy):
    print("STITCH", dx, " ", dy)

def read(file):
    with open(file, "rb") as f:
        f.seek(512)
        sequinMode = False;
        while True:
            byte = f.read(3)
            if len(byte) != 3:
                break;
            dx = decodedx(byte[0], byte[1], byte[2]);
            dy = decodedy(byte[0], byte[1], byte[2]);
            if ((byte[2] & 0b11110011) == 0b11110011):
                stop();
            elif ((byte[2] & 0b11000011) == 0b11000011):
                changeColor();
            elif ((byte[2] & 0b01000011) == 0b01000011):
                sequinMode = not sequinMode;
            elif ((byte[2] & 0b10000011) == 0b10000011):
                if sequinMode:
                    sequin(dx,dy);
                else:
                    move(dx, dy);
            else:
                stitch(dx, dy);

read("C:\\Users\\Tat\\PycharmProjects\\pyembroidery\\sequin.dst")

[tatarize]

Similarly here would be the reader for .exp files. Note the need to transcode java signed byte from the unsigned version, and the reuse of the same call functions that would in theory be calls to something that can record those events property.

def stop():
    print("STOP")

def changeColor():
    print("CHANGE_COLOR")

def move(dx, dy):
    print("JUMP", dx, " ", dy)

def stitch(dx, dy):
    print("STITCH", dx, " ", dy)

def signed(b):
    if b > 127:
        return -256 + b
    else:
        return b

def read(file):
    with open(file, "rb") as f:
        while True:
            b = f.read(2)
            if len(b) != 2:
                break
            if ((b[0] & 0xFF) == 0x80):
                if (b[1] == 0x80):
                    b = f.read(2);
                    if len(b) != 2:
                        break
                    stop()
                elif b[1] == 0x02:
                    stitch(signed(b[0]), -(signed(b[1])))
                elif b[1] == 0x04:
                    b = f.read(2)
                    if len(b) != 2:
                        break
                    move(signed(b[0]), signed(b[1]))
                else:
                    if ((b[1] & 1) != 0):
                        b = f.read(2)
                        if len(b) != 2:
                            break
                        changeColor()
                        move(signed(b[0]), signed(b[1]))
                    else:
                        b = f.read(2)
                        if len(b) != 2:
                            break
                        stop();
                        move(signed(b[0]), signed(b[1]))
            else:
                stitch(signed(b[0]), -signed(b[1]));

read("C:\\Users\\Tat\\PycharmProjects\\pyembroidery\\venv\\Include\\BN00883_A.EXP")

[lexelby]

Whoa there, buddy, I can see you think quite quickly! Please hold off for a bit until I have a chance to respond. Should be later today. :)

[lexelby]

I might be finishing up my current project pretty soon and was wondering which way you wanted to go for these?

Oh, are you interested in contributing? That'd be great!

Did you want an audit of the libembroidery code to fix up the errors there. Or to actually just code these formats into python?

I have to admit I'm a bit torn, but I'm leaning in favor of translating to Python. On the one hand, libembroidery was huge in helping me get my start, and I want to contribute back to that project. On the other hand, having the library in C is causing problems for Mac (#181) and makes the Windows build difficult (I have yet to figure out how to automate the libembroidery build for Windows, so right now it's manual). Ultimately I think we can iterate much faster if we port to Python.

The next question is: should the python libembroidery port live in Ink/Stitch, or should it be its own library? I'd lean toward the latter. I see you already named your directory pyembroidery. It'd be really cool if we could get that posted to PyPI and just put pyembroidery in Ink/Stitch's requirements.txt.

You seem to have a lot of interest in and experience with embroidery file formats. Do you feel like being the maintainer for a pyembroidery repo? I've been toying with the idea of creating an inkstitch organization and moving this repo into it. pyembroidery could live there too.

Obviously such things needs a bit more stuff like a datastructure to actually hold all the data we read in triggering the stop(), colorchange(), move(), stitch(), some metadata and some colors, some threads, and some thread metadata.

Agreed. It makes sense to me for pyembroidery to have a set of classes to abstract those concepts, much like libembroidery has. This also gives us a chance to properly differentiate STOP from COLORCHANGE.

Regarding stop, trim, sequin, etc: this is an area where Ink/Stitch needs some improvement. Right now, Ink/Stitch renders a TRIM instruction as "TRIM, JUMP, JUMP". The idea is that libembroidery's DST converts TRIM to JUMP, so this sequence will result in three JUMPs in DST. All other formats will have TRIM + two redundant jumps, which is fine.

It really makes more sense for the file format code to handle the concept of "three JUMPs = TRIM" for formats that require that (just DST?). I'd like to be able to tell pyembroidery "trim()" and have it do the right thing for the file format: native TRIM for formats that support it and convert to JUMPs for those that need it.

[lexelby]

Oh, one more thing: let's keep code implementations out of comments here because it just makes the thread too long. That's what repos and pull requests are for :)

[tatarize]

Any apprehension I have certainly doesn't apply to converting the formats from and to a couple different set embroidery files. The strong preference there would be on a stand alone project. Even if you would be the only one actually using it, obviously it would be completely independently useful. After all libembroidery is the most useful part of embroidermodder.

So yeah, I'd contribute this.

Part of your issues there at the end is that converting the files to jumps and trims and stops is actually a bit too high level. Trying to directly code those commands to coax it into working property is a problem in itself. And some of the metaformat encoding like three jumps equals a trim, or a format requiring a color_change, jump, jump vs. trim, color_change, jump, jump. Or even simple things like breaking a really long stitch into a smaller stitches that a format can actually encode. All of these things are lossy. And the hope should be preserving as much as you can. To do that you need to take in data as close to the data you produce and let the conversion-read-write stuff maximally know which parts you care about and which you don't.

All of the low level stuff and slightly higher level stuff needs to be handled by the library, as it's a bit more complicated than simply reading and writing data in and out of the files. But, not by much.

In theory the interfacing would be done the same way it's done for Touch Embroidery, and Scribble (my photoembroidery program), EmbroideryViewer which is generally writing it into the embroidery pattern from layers to a format that matches 1:1 the points you use internally (and doing nothing else with them), then you call the internal converter to transcode that into a format that fits writing out, then writing that out.

I used, STITCH, STITCH_NEW_LOCATION, STITCH_NEW_COLOR, STITCH_FINAL_LOCATION, STITCH_FINAL_COLOR, for these. When you internally use layers these are easy to flag and write each point properly, and since you're trying to do more than that, this can convert this to a coherently known set of commands as might be needed for the particular format being exported.

If you don't leave these decisions up to the conversion library you actually well end up with a series of different issues, like loading from DSTs never overtly having a trim command because the format doesn't encode a trim, then saving to a different format that expressly needs a trim command so it knows it needs to trim. Suddenly there's a bunch of super-long lines all over the connecting various items and it categorically did exactly what you should have expected of it. Hence the need for a sort of middleformat that captures all the data losslessly but isn't hard to read and write into, and a couple internal converters into and out of that generally stable form.

I already did that work in Java, at least somewhat reasonably. And if you consider these problems and realize you might need to tweak some aspects of them, you simply wouldn't have that option without the code actually in python.

You would actually have the converter handle the three jumps = trim thing. Because the raw set of commands you load from the file will still need some processing done on them. Or to read them into stitch blocks. In python it would be something like a generator of coherent uninterrupted stitch groups. So load the format then get every group of "STITCH" commands, and the thread the format says should have been used for that. And while you'd have all the color changes, trims, etc, you'd just generally ignore reading the stuff you don't want to read, but have a bunch of useful helper functions to access the data.

So yeah, I daresay there's a clear need for a pyembroidery, I am not a python programmer. I did learn it yesterday (and code in enough languages to rapidly pick up new ones in a couple days). So I could certainly transcode my work from java and produce totally functional code, it might need some reviews from some pythonistas and maybe some refactoring, if I did something overly javaish. But, I could quite certainly get some of the easier formats working for reading and writing running.

The bulk of the work is and was always done in Embroidermodder, namely parsing out what the formats are actually doing, so anyway you slice it, libembroidery would still be huge in getting such things done so don't worry about loyalty there. But, all the formats ww suggested could certainly be done and be done properly in pure python code. With the exception of VP3 which I am not actually an expert in, the other ones I know enough to be kinda straight forward.

I'm convinced enough, to do this project either way. I'll convert the read-write stuff I have working in java into javaish python, and do dst and exp, likely pec (with blank graphics), and hacked-pes. So as to have at least functional work to review. It's a pretty concrete project with overt achievable goals. And maybe somebody would have need for a python script able to convert one embroidery file format to another. I could see other people needing that. And give it an MIT license or whatnot so maybe somebody could smooth over the less than pythonic edges.

[lexelby]

Thank you, that sounds wonderful!

I hear you on the lossiness and the mismatch between various formats versus what my design actually wants to do. I'm actually totally fine with it if writing a machine embroidery file is a lossy procedure. My users won't be round-tripping to DST as a matter of normal operation. The SVG is the master file. I think of writing a DST or PES as a process of "rendering" the design.

Ditto for reading. Ink/Stitch recently gained the ability for Inkscape to natively open embroidery files, and it converts it to an Ink/Stitch SVG with "manual stitch" blocks. At the moment I don't even think it properly reads and inserts TRIM instructions. I see this as a way to tinker with purchased design files rather than as a way to open files Ink/Stitch itself has produced.

Or to read them into stitch blocks. In python it would be something like a generator of coherent uninterrupted stitch groups. So load the format then get every group of "STITCH" commands, and the thread the format says should have been used for that.

That sounds a lot like Ink/Stitch's ColorBlock class:

https://github.com/lexelby/inkstitch/blob/master/lib/stitch_plan/stitch_plan.py#L141

A ColorBlock is a set of stitches all with the same color. A StitchPlan (another class defined in the same file) contains multiple ColorBlock objects. Perhaps it might even make sense to relocate those class definitions into pyembroidery?

I'm convinced enough, to do this project either way. I'll convert the read-write stuff I have working in java into javaish python, [...] maybe somebody could smooth over the less than pythonic edges.

That sounds fine to me. Python's got all the basics you'll expect, and you're primarily going to be writing simple procedural bits-n-bytes manipulation code anyway. The few classes you may need to write shouldn't be particularly complicated. I'd be happy to review your Python code and would even be willing to create PRs, if you're interested.

Last time I wrote Java was in college in my computer architecture class, and I seem to recall it not having an unsigned integer type, which made bitwise manipulations significantly more difficult...

[tatarize]

Okay, really basic early version is done. (Reading DST, EXP writing EXP)

https://github.com/tatarize/pyembroidery

It has the same main core elements as the java version. Some bits aren't really hooked up though (for example none of the formats actually load a EmbThread object so that code never got tested.

There's bound to be some better ways to work out a useful api. And I coded up exporting in .exp because of the general it required the smallest amount of piping. Also, I gotta figure out some more coherent way to do a damned switch statement.

And some aspects are clearly going to need to be changed like, I shouldn't actually be doing the file manipulations inside the readers and writers, rather than just deal with the streams, and remain agnostic as to where those streams come from.

Josh standardly has some other stuff like reading and writing based on extensions. Though it suffers some problems like if you want to save out a .PES file, you can't indicate whether you'd like .PES v1 or .PES v6 (v6 has encodings for overt threads rather than predefined threads with singular byte index).

Also, I just called pass rather than actually code up the StitchBlock and ColorBlock sort of generation code. The stitch_plan code there is clearly doing some of the same things as my write encoder. It's necessary work to coax it into working properly with a format, it's critical to keeping the formats as stable as possible. The way I write the data is entirely expecting the stuff to come from a datastructure kinda like that. It's just very useful to think of large groups of stitches as a unit. You just iterate over all the data and flag the points that are part of a transition and send it to the encoder to do the heavy lifting.

Internally between a few different programs I have a lot of different actual data structures. Including things like ColorBlock or StitchPlan or my RootLayer class in Touch Embroidery, or my Layer class in Scribble, or StitchBlock in MobileViewer always kind of fits well, but, for maximum interoperability I think it's best to just give users highly functional and useful iterators (in python generators) so that the data is as decoupled as possible.

Each time I've tried using the internal datastructure the actual project uses, it always works fine. But, then when I try to port the changes etc in the code away from that, it gets a bunch of tendrils wrapped in everything and is a giant pain in the butt to decouple it, and doing such things to save what might be a couple milliseconds quite simply ends badly, so I've sworn it off. Better to give you the very best and most general converters into whatever you use internally. So things like generators to give you each uninterrupted set of stitches, and their color. Or each uninterrupted set stitch commands of a particular color.

[tatarize]

Added in .DST writing. And .DST writing with extended headers, even though I've never actually seen a .DST file with extended headers, AU, CP, TC for author, copyright, threadcolor respectively. I'll try rounding out the rest of @wwderw's suggested formats. Though I don't know VP3. This should, already, in theory fix the .EXP write errors. And handles the processing of DSTs with sequins.

[lexelby]

Wow, nice work!

I haven't had a chance to go through this in depth yet, but a couple of things stand out. First, it looks like you've written this in Python 3, but unfortunately Ink/Stitch is stuck on Python 2. :( That's because inkex.py, the extension framework for Inkscape, is in Python 2 only.

The other thing is that your methods are in CamelCase. It's not a huge deal, but Python actually has a standard style guide and it's useful if all code follows it. The standard is PEP8 and there's a tool, https://github.com/hhatto/autopep8, that can fix your code automatically.

[tatarize]

Sadly the autopep8 doesn't fix that because CamelCase is permitted generally if it fits with the style of stuff already there, and might end up making breaking changes. It fixed some other stuff though. I'll have to change the method calls myself or just let them fester.

I added PEC (blank graphics) and PES (truncated). These might have some other issues I don't know about though, haven't gone over them with a fine tooth comb and they are fairly big. The truncated PES will run fine on everything except Brother's own software for the format, as they try to load the data in the PES section and I said it was completely blank. Though anything else will load the PEC block there for the stitches and won't realize anything is off. It really needs proper PEC graphics and to actually write the PES blocks rather than totally cheating on that bit (but so easy and only PE-Design would notice).

I didn't add the readers for PEC or PES yet, but those are easier than the writers.

I'm not at all informed enough to know how to convert to python 2. I assume most of it kinda fits but some idioms are invalid. But, again, dunno. Learned the language like two days ago.

[lexelby]

No problem, it's really quite easy to convert to Python 2. I'll throw together a PR when I get a chance.