Contribute to implementation

[JoshVarga]

I am the developer primarily in charge of formats on the Embroidermodder project. We have a simple implementation of writing PES files. Recently I worked to understand the entire PES format primarily version 1 and 7. I have an application written in C that parses nearly all of the data properly. I plan to make it open source, currently embroidermodder is not prepared to handle all the design information so I wanted to understand your plans for implementation. Does it make sense to you for me to contribute to this project or just spin up a new repository?

Thanks, JoshVarga

[treveradams]

Currently, I wish to document as much as possible. My longer term goals were then to create a libpes, much like libpng or libjpeg, in that it allowed easy access (read-only at first then read-write) to all data in the file. I was hoping that eventually there could be similar libraries for the other common embroidery formats. I was hoping, if I was involved in such, that there could be a similar API for these libraries (as much as is possible).

At first, I was hoping to extend Inkscape to be a gui for these libraries. Short term this may work, but I am not sure it will work as well as I had originally hoped long term.

What do you think?

[JoshVarga]

I think if I contributed what I have right now it might help with the documentation. I believe there is quite a bit that I discovered that is not currently in your documentation. Some of the more difficult portions would be more easily decoded if this program did the majority of the decoding for you. Following the libpng and libjpeg API, sounds like a good idea. We could evaluate what I have now and see how can modify it to more closely follow the API of libpng and libjpeg while still making sure that it makes sense for embroidery formats.

Currently in embroidermodder we have a portion of code named libEmbroidery that provides a common api for many different formats including pes (read and write) but it doesn't support many of the "designer" file portions of pes. Only simple designer code with the additional "machine" portion (PEC).

-------- Original message -------- From: Trever Adams notifications@github.com Date:02/08/2014 2:17 PM (GMT-06:00) To: treveradams/libpes libpes@noreply.github.com Cc: Josh Varga vargaj2002@yahoo.com Subject: Re: [libpes] Contribute to implementation (#1)

Currently, I wish to document as much as possible. My longer term goals were then to create a libpes, much like libpng or libjpeg, in that it allowed easy access (read-only at first then read-write) to all data in the file. I was hoping that eventually there could be similar libraries for the other common embroidery formats. I was hoping, if I was involved in such, that there could be a similar API for these libraries (as much as is possible).

At first, I was hoping to extend Inkscape to be a gui for these libraries. Short term this may work, but I am not sure it will work as well as I had originally hoped long term.

What do you think?

— Reply to this email directly or view it on GitHub.

[treveradams]

I am sorry. I wasn't clear enough. I meant in concept like libpng/libjpeg in that they were fairly easy to use and could be used by anyone (LGPL or similar).

Is the license on libEmbroidery such that we could start an API from there (not the code necessarily, just the API)? We could then just extend it for the designer portions of those formats that have it.

Thank you for your contributions and interest!

[redteam316]

@treveradams Yes, libembroidery can be used by anyone for making plugins, other apps or other uses. It is zlib licensed so there shouldn't be any concerns about license conflicts. If you are wanting to use it with Inkscape, python bindings would need to be completed first. libembroidery is written in C89 but is a long term goal of ours to provide bindings for many other languages eventually and I have been looking at SWIG to do this with since it covers many languages. If you look at the Embroidermodder repo, you will see that there are experimental Delphie/FreePascal bindings contributed by another contributor(just not using SWIG since it doesn't support Delphie/FreePascal bindings yet).

We can generate shared and static libraries. If you are using GCC/G++ you would use -lembroidery for linking. The biggest thing you should probably note right now is that since we have not locked down the API yet, upgrading to a newer version in the future may break your code somewhat.

[frno7]

Hello @JoshVarga, @treveradams and @redteam316. I’ve written a PES library that does decoding, encoding and transcoding of the PES format and a prototype version of an SVG based embroidery format. Eventually I’d like to make the library support all PES versions and all features, but I’m lacking some documentation and sample files to test with. Are any of you actively involved using PES these days?

[treveradams]

On 01/06/2017 04:36 AM, frno7 wrote:

Hello @JoshVarga https://github.com/JoshVarga, @treveradams https://github.com/treveradams and @redteam316 https://github.com/redteam316. I’ve written a PES library https://github.com/frno7/libpes that does decoding, encoding and transcoding of the PES format https://github.com/frno7/libpes/blob/master/PES-format.md and a prototype version of an SVG based embroidery format. Eventually I’d like to make the library support all PES versions and all features, but I’m lacking some documentation and sample files to test with. Are any of you actively involved using PES these days?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/treveradams/libpes/issues/1#issuecomment-270887329, or mute the thread https://github.com/notifications/unsubscribe-auth/AA-I_eerCfRJVwBKyDYdAROf98AgKpBdks5rPidWgaJpZM4BgWJO.

I have had input from an individual that adds many/much of what is missing. Unfortunately, I haven't had a chance to review it, reply and/or commit. I will be trying to do these next week. I would love to work with you going forward.

Thank you,

Trever

[JoshVarga]

All,

A while back I wrote code that supported almost every version and had almost every data type at least 90% understood. I will have to dig up that code. I was planning on integrating that into the "libembroidery" portion of embroidermodder.

Josh

On Jan 6, 2017 7:56 AM, "Trever Adams" notifications@github.com wrote:

On 01/06/2017 04:36 AM, frno7 wrote:

Hello @JoshVarga https://github.com/JoshVarga, @treveradams https://github.com/treveradams and @redteam316 https://github.com/redteam316. I’ve written a PES library https://github.com/frno7/libpes that does decoding, encoding and transcoding of the PES format https://github.com/frno7/libpes/blob/master/PES-format.md and a prototype version of an SVG based embroidery format. Eventually I’d like to make the library support all PES versions and all features, but I’m lacking some documentation and sample files to test with. Are any of you actively involved using PES these days?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/treveradams/libpes/issues/1#issuecomment-270887329, or mute the thread https://github.com/notifications/unsubscribe-auth/AA-I_ eerCfRJVwBKyDYdAROf98AgKpBdks5rPidWgaJpZM4BgWJO.

I have had input from an individual that adds many/much of what is missing. Unfortunately, I haven't had a chance to review it, reply and/or commit. I will be trying to do these next week. I would love to work with you going forward.

Thank you,

Trever

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/treveradams/libpes/issues/1#issuecomment-270907665, or mute the thread https://github.com/notifications/unsubscribe-auth/AAYcnIhNDr4YvoWyf61YMoHim3kGAPcgks5rPkglgaJpZM4BgWJO .

[treveradams]

I just updated the documentation with the information I have received. It makes a huge amount of sense. I have not yet worked all of it into the existing format of the document, but what isn't is still included at the end.

[frno7]

Fantastic! @treveradams, thanks for your quick update! I’ll look through the details of it. At a glance, I believe I have a more complete description of e.g. “PES Block v6” in the “version 6 header section”, especially the “color list subsection”. Do you have files containing “CEmbCirc”, “CEmbRect”, “CEmbLine”, etc.? None of files I have contain these.

@JoshVarga, I have a few hundered files of PES versions 1 and 6, and a handful of versions 4 and 5. The files were supplied by Brothers Industries Ltd. and Bernina International AG so I assume they are properly encoded. Unfortunately the files cannot be distributed as they have restrictive licences. It would be great to have a set of known-to-be-correct files as a test suite. Would you happen to have PES files that could be used for format testing?

By the way, what would be your opinion of using a very small subset of SVG as an open and vendor neutral format for embroidery designs?

[treveradams]

On 01/06/2017 11:23 AM, frno7 wrote:

Fantastic! @treveradams https://github.com/treveradams, thanks for your quick update! I’ll look through the details of it. At a glance, I believe I have a more complete description of e.g. “PES Block v6” in the “version 6 header section https://github.com/frno7/libpes/blob/master/PES-format.md#version-6-header-section”, especially the “color list subsection https://github.com/frno7/libpes/blob/master/PES-format.md#color-list-subsection”. Do you have files containing “CEmbCirc”, “CEmbRect”, “CEmbLine”, etc.? None of files I have contain these.

I believe I may have CEmbRect and CEmbLine. They are in PES files that I have under license and cannot share.

I would love having and all information you have and are willing ot share.

@JoshVarga https://github.com/JoshVarga, I have a few hundered files of PES versions 1 and 6, and a handful of versions 4 and 5. The files were supplied by Brothers Industries Ltd. and Bernina International AG so I assume they are properly encoded. Unfortunately the files cannot be distributed as they have restrictive licences. It would be great to have a set of known-to-be-correct files as a test suite. Would you happen to have PES files that could be used for format testing?

By the way, what would be your opinion of using a very small subset of SVG as an open and vendor neutral format for embroidery designs?

When I first started this project I had a few goals in mind. My main ones were to create a standard API that could be used to write several of the different embroidery formats. Some only have stitch, some have stitches and colors, some stitches colors and shapes. So the API would need to be able to define these.

The API would need at least three data structures. Colors (name, RGB representation, maybe vendor/vender part#) Stitches(start and stop of stitch, array likely) Shapes (not sure).

The functions/methods would simply be: Read Write

For formats that didn't support various parts, a standard UTF-8 text file could be created listing things. Some formats may only support part of the color information, for example, all of it could be written to the text file.

This way a program could load these data structures and call a write function, the read would be the reverse.

I was hoping to make a small change to Inkscape, or similar SVG editor, that would allow SVG to be used a "lossless" format.

If I remember correctly there were a few attributes which would have to be added: 1) A way of defining stitch direction for the primitive instance (poly, line, arc, etc.). This may seem unnecessary, but proper stitch direction can make all the difference in how an embroidered piece looks. 2) A way of defining stitch length, again for the given primitive instance. (In documentation or for libpes, or what not, good minimums and maximums would be good to know as well as a good standard.) 3) Stitch density for primitive instance. Again, this can make all the difference. In some designs it may even be desirable to have some parts more dense than others. (Again good defaults documented, and recommended minimum/maximum would be nice.) 4) Use gradients for the colors. Why gradients? This has more to do with thumbnails. Some embroidery thread has multiple colors. A gradient could show a rough idea of this in the SVG editor as well as in a preview image. (This last one would not require changes to Inkscape or what not, but would just require that designers follow some standard in how they created/used such gradients. Not following it may not change any functionality, just preview results.)

I never got to this as I got stuck on figuring out the PES format and my time was demanded by another project.

The final piece would be the stitch <-> SVG converter. Stitch to SVG might be relatively straight forward. The SVG to stitch has to be "smart" enough to avoid potential damage to the machine due to knotting (stitches being too close, etc.). It would need to read the 4 items added to SVG and use them to adjust settings.

A preview might also do something like Gimp's clothify, adjusting each parameter automatically, to each part of the image based on stitch direction, length and density. It could also use the gradients for multi color thread. We would not (and probably don't want to anyway) attempt to simulate each stitch, just a rough representation such as clothify does.

I would love to have feed back on this API idea and the SVG idea.

Thank you. Trever

[treveradams]

Caveat: I have never used any of the embroidery programs, so pardon my ignorance. I never go that far.

Further ideas on SVG subset. I think the path stroke and fill colors are important. Again, the use of a gradient if the thread is multi color. Stroke style. We should support whatever primitives that SVG supports that would commonly be used: circles, squares, ellipses, lines, arcs, letters, etc.

Was I missing attributes needed beyond stitch density, stitch length, stitch density?

@frno7 would you mind if I merged information from your document into mine and maybe even copy the format? (Get it into md out of odt.)

[treveradams]

@frno7 , It seems in version 6, you are skipping a field before "Length of following string". Another individual has that as being the # of stitchgroups. I saw this same field but didn't know what it was. Are you sure you have it complete?

As I am looking at this maybe I have been a bit naive. I saw a demo of one of the software pieces once upon a time, it showed simply drawing objects, setting stitch direction, density and length and it did the work.

[frno7]

@treveradams, many thanks for your comments! Your ideas on a high-level “generic” embroidery API sound good. In addition, I believe format specific low-level APIs (specifically for PES etc.) are useful as well, with the generic API implemented in terms of the low-level APIs. My libpes implementation is mostly low-level except, perhaps, for the SVG transcoder prototype. One design choice made is to avoid direct file operations in the library and instead rely on a set of callbacks, as this is much more flexible.

Some form of Inkscape integration, as you suggest, would be excellent! Similarly to the API, I think the SVG embroidery format could be split into two parts: one fairly low-level part mostly involving stitching instructions (PEC type instructions), and another high-level part concerning rendering of shapes, typefaces, etc. Details like SVG namespaces etc. need to be thought out as well.

My libpes repo is GPL-3.0 so feel free to clone and modify the documentation etc. I’m happy to accept patches as well.

Regarding the PES version 6 header, I‘ve been unable to make sense of the fields (the 4 bytes u16 + u8 + u8 with typical values of 1, 0x30 and 0x31 or 0x32) just before the description string. Is some field missing? Not sure how any of them could be the number of stitch groups? [ The hoop width and height are given in millimeters after the string so why would those be encoded in reduced form in the u16 before the string, as suggested in your ODT? ] One way to try out the specification is to run the pes-info tool that comes with the library. Does it work properly with all your PES files?

[treveradams]

@frno7, I have been looking around today. I think what I am looking for (if I could make it work) is somewhere between https://github.com/lexelby/inkscape-embroidery and https://github.com/sergstetsuk/inkscape-embroidery/tree/shapesplit. I cannot make it work because it complains about self intersection or holes.

I am not entirely liking the algorithm results for stitches with it (the samples shown). I like the idea of all the stitch length, density, direction (maybe stitch fill type such as those in Wilcom Hatch) being attached as SVG/XML attributes to the object. Your SVG format seems to convert each stitch to a path, which is NOT bad and allows for things to be manually edited. Would it be possible to mark each such SVG object you create with some attribute so that modified versions of inkscape-embroidery wouldn't try to restitch it, but would dump it directly? Or am I missing how your pes to svg conversion works? I do work with Python, but I am having a difficult time figuring out python shapely and why it throws the self intersection error. I also am also not sure what to do about the lack of deal with holes properly. (https://en.wikipedia.org/wiki/File:Flag_of_the_United_States.svg has self intersection, http://www.freepik.com/free-vector/coloured-eagle-design_952146.htm#term=eagle&page=1&position=13 will need eps to svg conversion by Inkscape, it causes the holes problem).

As for the hoop, that was provided by someone else. It does seem to match what I am seeing. It also fills in a gap, that at least in my copy of v6 and v1 headers, you don't even seem to have a place holder for.

I will try to better compare my documentation, yours and the other contributor's. Unfortunately, I have pressing time requirements for at least the next 2 weeks related to work and home obligations. So, my work here will be sporadic until those are met.

[frno7]

@treveradams, yes, the SVG embroidery format I’ve been prototyping is a simple almost direct stitch-for-stitch transcoding for the PES format. I think it could be improved using attributes, as you mention, especially the technical parts such as handling of jumps and trims.

I imagine at least two kinds of workflow. Firstly, one can to transcode PES into SVG in order to adjust a design (in a very technical and precise way using for example Inkscape, manually or otherwise), and then transcode the modified SVG back into PES again (preferably losslessly). Secondly, one can author original designs in SVG and have them transcoded into PES. In this case it helps to work with shapes in SVG and then translate those shapes into SVG paths before transcoding into PES. I’ve made a few proof-of-concept designs using Inkscape and my own tools in this way (SVG shapes → SVG paths → PES, so SVG has dual purpose here).

I work mainly with C/C++ and haven’t tried the Python tools you mention. I believe a well rounded set of extensions to Inkscape could certainly improve usability of the workflows.

[JoshVarga]

I haven't had time to document the format, however I just published some code that should be able to answer questions about many missing details. Take a look at the code here: https://github.com/JoshVarga/libpes

It should help to get an understanding of the difficulty level of making lossless conversion between PES and SVG.

If anyone has time they may be able to extract much of the code into the documentation.

It covers: Version 1 - 8 Affine Transforms Embedded patterns Circles D Text Object groups differences in the header of the files regular letters Lines Motif patterns M Text N Text Rectangles Satin stitches Special shapes Stamps S Text Tile lines

[treveradams]

@frno7 yes, I think you and are have the same work flow in mind. SVG -> PES being the major part. @JoshVarga, PES -> SVG may lose information. While I am interested in that, it isn't my main concern. A complete documentation of the PES format and a way to use SVG, say through Inkscape with extensions/modifications, is what I am interested in. My work flow as I am working to get going uses SVG for many things and it would be nice to be able to use it for the emroidery part as well for consistency.

[frno7]

Thanks @JoshVarga! I’ve updated the PES format document with many of the findings in the headers especially.

[frno7]

@treveradams, my Brother Innovis 955 machine seems to only make use of the stitches in the PEC, ignoring most parts of the PES. If this holds true for embroidery machines in general then the PES encoding format can be quite minimal in any SVG path → PES transcoding workflow.

The problem of translating SVG shapes (polygons, curves, text, etc.) into suitable SVG paths could then be treated separately, independently of PES and possibly any other embroidery format. Translating shapes into paths is a difficult problem but at least things simplify if one can ignore PES.

@JoshVarga, do you have any comments on these thoughts?

[treveradams]

@frno7, Mine also uses the color table (names, rgb values). As for translating shapes into stitches, that is why I am looking at the python projects I mentioned. Inkscape can convert just about any SVG shapes into paths, once there, the python extension (barring holes and self intersection problems) solves the rest for me (then manual editing to fixes mistakes, etc.)

[treveradams]

@frno7, would you mine chiming in on my suggestions for the SVG related stuff in https://github.com/stesie/inkscape-embroidery/pull/4?

[JoshVarga]

PEC is the machine format, I don't know of a machine that uses the PES portion. PES is for designing only as far as I know. Embroidermodder is capable of writing version 1 PES files that are accepted by every machine that I have tested on. It's possible to just use that code. In the libEmbroidery portion of Embroidermodder most of this is supported, SVG is partially supported but is definitely lacking.

On Mon, Jan 16, 2017, 1:32 PM frno7 notifications@github.com wrote:

@treveradams https://github.com/treveradams, my Brother Innovis 955 machine seems to only make use of the stitches in the PEC, ignoring most parts of the PES. If this holds true for embroidery machines in general then the PES encoding format can be quite minimal in any SVG path → PES transcoding workflow.

The problem of translating SVG shapes (polygons, curves, text, etc.) into suitable SVG paths could then be treated separately, independently of PES and possibly any other embroidery format. Translating shapes into paths is a difficult problem but at least things simplify if one can ignore PES.

@JoshVarga https://github.com/JoshVarga, do you have any comments on these thoughts?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/treveradams/libpes/issues/1#issuecomment-272945134, or mute the thread https://github.com/notifications/unsubscribe-auth/AAYcnN5xnyF5XuSl4I73_IoC12ELesVxks5rS8XKgaJpZM4BgWJO .

[frno7]

@treveradams, about the min/max stitch lengths etc. represented in SVG? It sounds interesting but unfortunately I don’t have much to say at the moment since I haven’t worked with those kinds of designs. So far I’ve simply programmed my own algorithms for stitch patterns, etc.

By the explanation of @JoshVarga (thanks!) I favour as a basis having a vendor neutral “machine format” representation in SVG (equivalent to PEC), essentially made up of SVG paths and perhaps a few necessary attributes but not much else.

As for “digitizing” general designs (for example SVG shapes) into machine format (assuming this is what you’re trying to do here?), there are obviously many degrees of freedom, and it’s a quite interesting problem. It seems likely that this too could be defined within SVG in some sense.

[tatarize]

All the SVG shapes can be converted into SVG paths, and in theory PES would support those regardless. And much of these are generated within Brother's PE Design program. So in theory some parts will be necessarily lossy. For example, there's no stopping and starting points in SVG because there's nothing to be done there. There's no angle of the stitch, or stitch densities or any qualifiers on how the stitches are laid down. But, both formats certainly share shapes, not just the line segments EMBSewSeg but the other shapes would certainly be able to be directly transcoded. Even without explicitly specifying what the sew-commands are. You'd end up with a bunch of shapes as such, that can be loaded in Brother's PE Design and then that program could make the stitches then save it out, in the default way it would go about filling those shapes.

You'd only be feeding in the basic shapes and hoping Design PE fills them in somehow coherent. But, note this is similar to how a lot of modern Embroidery Software goes with them being addons to Corel Draw or Illustrator. They simply jump to the program after all the vector shapes are in, and tweak those settings that aren't relevant to vector shapes.

[tatarize]

@frno7 As for the format neutral general design format, the degrees of freedom and the interesting problem it reveals. I've actually dedicated a very considerable amount of time to address it. There are expressly a wide range of different freedoms here. And we need to capture them all. What actually is needed is a format that can properly express the simple as well as the complex and do so just as easily. So you can express not just the shape that needs to be rendered but every way which it can be filled with stitches. It is not sufficient to simply list these ways, and start implementing what will end up being a massive program with every single way to articulate stitches, and every feature of every program has ever had. It needs to be done in a much more coherent fashion from what amounts to first principles.

A programming language for example, is useful because you can express anything in very useful and expressive syntax and explain large amounts what you're doing in a coherent fashion, while ultimately compiling down to bitcode/machine-code that a simple machine use. We have the same problem here with embroidery. Where you need to be able to express something complex like take this the letters TAT and express them as stitches. Each layer could well have added degrees of freedom, and power.

• "embroidered word TAT at location m"
• The letter T ending at the bottom, A ending at the bottom right, T starting from the bottom and ending where-ever.
• the glyph data of the various shapes with the starting and ending points without knowing they are letters.
• The boundaries of the glyph data, the satin stitch information, the underpathing information, the jump stitch information, the data for the next letter etc. *The physical x,y change location of the stitches, cut/jump etc.

But, what these actually make is a nested hierarchy, where you can make the lower level items from the expression of the higher level items. If you have the stitch information making the physical X,Y changes result in BAT rather than TAT is basically a non-starter. Each step you go up in the hierarchy it becomes more and more easy to make this change. There are also really crazy expressive vector line things like guilloches that could be expressed in embroidery but nobody does, as they don't at their core work directly on shapes. And so you need a format which can properly express all higher level and lower level things to do, and express the interrelationship of all the elements. Without, also limiting the format, to just those methods.

I've solved the intermediate issue. And can express a really useful format that is much more expressive than stitches while not being as limiting as, fill this particular shape. It basically amounts to a rendering pipeline of particular line types. Where we can express not just simple stitches, cuts, color-change, and jumps, but we can also express things like fill stitches and satin stitches without regard to the boundaries of the shapes that we're using to limit those geometric ideas. After all, a fill stitch relates directly with the geometric idea of monotony, equally there are satin stitches which relate to the idea of sweepability, and radials with have direct correlation with star-shaped polygons and all the maths therein. But, all of these can ignore the actual property of the shape they are filling and simply render in a ray-traced fashion (so you can fill a non-montone polygon as if it were montone and just miss a section. So you can take complex things like the algorithm for a tatsumi fill and break it down into intermediate steps. Thereby describing all the different quirks with all the different tatsumi fill algorithms that differ greatly between various programs. Or things like start and stop points within a shape and the complex patterns that describes or the underlay and underpathing as their own thing. We can break down the complex (fill this shape, with these start and end points) into the simple (draw this sequence of lines, but a few of these lines ray-trace to the edge of the shape they might be filling).

The ultimate goal of all of this is to figure out a language that is as expressive as stitches themselves, while also being simpler to express some superset of predefined fill patterns. And we can do this by understanding everything as a path the sewing machine generally takes and perturbations of these paths. Thus resulting in the simplest and most expressive language for embroidery. It's also, consequently nothing like how most modern embroidery is done, but is entirely able to express all of those forms of embroidery and a huge amount of forms that are basically missed. At the highest level you can express that you'd like to fill this shape starting at point p and ending at point q, but we need to break that down into the specific very expressive steps taking into account underlay, underpathing, the density of the thread, randomness at the edges, and all the huge number of potential options and turn that into an intermediate form then into the final stitch form. Since the intermediate form could well be final stitch form, it would rightly be, by definition, properly expressive. And properly with pathing and the core underlying geometry we can make considerably more expressive forms that still following paths, but because these forms are still fully free they can properly express the work of all higher level commands and all embroidery programs.

Mostly this is required because when you're faced with something new that doesn't fit into the classic colorforms style of modern embroidery, you're screwed. Say you want to add a spirograph. to your embroidery program. How does one possibly articulate that? What about if one program makes the needlepoints within a fill fit a particular pattern? What about if the fill they use includes a particular pattern itself? Or if you want a flat fill but having it curve up a little bit at the very end thus filling a non-monotone shape in an interesting way? How about if you're doing a series of different fills in the same direction, and plotting out all the underpathing, and end up needing make thousands of objects? What if somebody develops a better more interesting way of doing that same thing that basically amounts to finger painting monotone shapes in vorono diagrams shapes? There's way more things in embroidery than any modern embroidery program can express (except as a very naive running stitch), and a significant amount of these shortfalls is because of the assumption that embroidery is about shapes and how to fill them, rather than how can we quickly put down a lot of stitches in this interesting way. Modern embroidery is just one way of doing these things, and the trick is to develop something that can easily capture modern embroidery without limiting it. Much as modern programming languages allow you to express very complex ideas easily without losing the power of what machine code can do.

It's a very interesting problem and requires a lot of thought and work. But, it's well beyond the SVG spec. Even if we account for foreignObjects and switch statements and extending the format a lot in some reasonable and consistent ways. While expressing vector objects as svg paths would be fine and such, you cannot express the basically endless ways of filling that shape in any coherent fashion.

[frno7]

@tatarize, thanks for your thoughtful comments! SVG accommodates foreign namespaces and private data which I believe would be appropriate to handle many if not all of the embroidery stops, angles, densities, qualifiers, etc. that you mention.

As regards to an SVG based embroidery design format, I agree there are many degrees of freedom. For this reason I’ve decided to concentrate on the embroidery machine format (PEC in this case), which in analogy with your programming language example would be equivalent to the assembly/byte code. I’m optimistic we can capture all aspects of PEC within SVG in a reasonable manner. This simplifies tooling in as much that designs can be transcoded into SVG instead of having to deal with PEC directly.

One way to resolve the design complexity is to accept that there are several design formats for different purposes, some perhaps based on SVG, much like no single programming language is able to handle all desirable abstractions.

[tatarize]

Even if we ignore the command level commands of PEC we could easily view what following those commands would result in as a completed embroidery very largely just say the command sequences are just line plots. Though this is admittedly lossy with regard to stops. It is valid to stop the machine in places where the actual resulting embroidery would not have any effect generally for things like applique. These also record the machine's work product and not how it did the jumps. It's rare that it's ever important but the embroidery machines really will jump around in a circle a bunch of times if you tell them to. Actually PEC blocks also contain the graphics data which cannot be represented in SVG though one assumes it to not contain much of anything.

There's a lot to PES that is specific to the Brother embroidery software used to produce it. Things like background colors, and hoops, and the handles for the objects. I tried breaking down EmbRect a bit, and determined that the block stores all the rectangles end to end, but also stores not just the 4 things you'd need but a bunch stuff like the position of the actual handles used to move the rectangle within that data. So you can within Brother define the rectangle in one place and the bit that moves the rectangle as being in another place. These are part of what PES actually is.

There are also things like how shapes are filled that aren't recorded anywhere save in the Software used to fill the area. Even if you are going to fill a "U" shape for example with a density of 4 lines per mm, and do a monotone fill from top to bottom going purely horizontal. There's still a huge number of ways to fill that. You can lay down an underlay first, to right side or left side first. Attach the bottom as part of the right or part of the left likely with some underpathing. To make this worse PES doesn't record that as a shape. It literally calls it a letter object (EmbLetter).

Assuming we could go through all the work to make a PES to SVG converter we could certainly do a lot of that. We'd need to realize most of the shapes perhaps the basic ones and skip over text, and get that working. And maybe there's some interesting ways to convert SVG equally into some of the core PES objects assuming they are mapped out well enough to write them. The only thing this would do is allow Brother PE to edit this stuff and then also use things like inkscape then convert it to PES. And then have brother generate the core stitches. It's entirely possible. In fact, I can currently do that with just the line objects themselves. Read the lines, write the svg lines, and back again. That's really easy. It wouldn't be super-hard to add to that some ability to do rectangles.

I program Touch Embroidery for the Android and one of its core formats is SVG. I know the standard well enough to know its power, and its limitations. But, the biggest issue isn't with SVG, it's in the heart of the computational embroidery: it's not vectors.

The lines, sure those work. And shapes and filling the shapes is a great way to lay down a lot of stitches very quickly, but the fact that the shapes are being filled with stitches means we're not really dealing with vector shapes. We're dealing with fills constrained by vector shapes. Yeah, if you want to convert shapes in PES to shapes in SVG and back again, that can be done. It might even be very nicely useful. You could equally do as a number of high end embroidery programs do and take a bunch of vector shapes and apply a series of algorithms to those vectors and get an embroidery design that you laid down a bunch of underpaths, fill stitches, underlay, and used some pretty solid metrics for how to do that. -- My objection to this is if you wanted to how would you define a spirograph or a satin fill with a curved element in the center.

It's certainly not needed for this task. Honestly one could hack out circles, rectangles and likely punches fairly quickly and perhaps get them working both ways in the PES<->SVG arena, in a week or so. At least provisionally working even if you don't plot out all the fills (after all the SVG doesn't have them, and your SVG file would be the only thing that cares). You might even just do punches for the same reason in SVGs you can get away with only doing paths. You can represent all the shapes with just that one element. The differences are that they are edited in different ways, you can change a circle in different ways than 4 bezier or 2 arcs that basically perfectly represent a circle.

Really I've explored SVG as embroidery rather extensively and I keep coming back to the core problem: embroidery isn't vectors. And when you address the proper problem it requires asking mathematically what is embroidery, like, as a thing. And secondly, what can you do to make that actually usable so that a puncher can put down all the stitches they need to put down in the shortest most efficient ways, which basically brings dozens of different solutions to that to mind, a goodly number of them that cannot be properly captured as vectors.

[frno7]

@tatarize, my personal workflow is to generate whatever layouts I want programmatically. So if a spirograph is desired, I’d make a program to output machine format SVG, and when the look is finalised transcode that to PEC using libpes.

So essentially I sidestep the embroidery shapes format complexities. Sometimes I block out a base design using Inkscape SVG, and then programmatically fill in the missing parts to obtain machine format SVG for conversion to PEC.

As for “EmbLetter”, I suppose FreeType would be a good start for rendering? Admittedly, it’s more difficult to layout stitches than pixels.

[tatarize]

Yeah, but then if you wanted to adjust the design any, you have it in a bunch of line segments and can no longer modify it after saving. Part of the problem with the embroidery as vector ideas is that all this other stuff about how the shapes are filled must be saved. They simply are many times more important than the shape itself is. Basically unless you have all the parts you used to generate the stitches, you can't edit the stitches. Because really what needs to happen is that the stitches rendering elements need to change and the stitches need to be rerendered.

If it came down to getting the letters right, SVG does have text. It might be possible to simply have a text object there and match the letters as such. So you can get the shape more or less. But, in a lot of embroidery they have internal fonts and they are embroidery fonts which differ from actual fonts in the same way that embroidery differs from vectors, namely they have a bunch of information on how to fill the particular font with stitches.

The shapes might entirely be doable in a short period of time. The stitches are doable now. It's also worth noting that within PES it's often correct to use the PEC commands for stitches simply because if the PES has a higher level object like a circle or text or rectangle it will produce resulting stitches but those won't be found in the EmbOne/EmbSewSeg block. That only governs sew segments. Like any proper embroidery format PES stores the workers and the rendered stitches even though they are equal after you use PE Design to render them.

But, it seems like these things could be incrementally added. Figure out how to read and write EmbRect to Svg rect, and whatever scheme PES is using to combine two rectangles, it doesn't use a 1:1 ratio of blocks to shapes, it puts all the rectangles in the same place, so a rectangle, then circle, then rectangle will produce a EmbRect block with both rectangles, then a EmbCirc block with the circle, and somehow not screw up the ordering. It isn't too tall of an order to convert a number of simple shapes into the right class of shapes, or convert them all into a punch object (which would fill with stitches but be technically wrong in a lot of ways). Then if this is correct enough that Brother PE Design can load the file (it crashes a lot when you give it weird stuff) then that one high priced program could convert those shapes into stitches and save out a different PES object with the PEC blocks too.

The very lowest level parts of that conversion would certainly work. PES only has so many shapes. And while the info on how to fill them with stitches would be lost going to SVG and from SVG we'd have to throw on default settings it would be entirely plausible to do this and moreover wouldn't take too long. If I knew the precise ins and outs of what it's doing I'd toss it into my rather giant breakdown of PES.

https://jsfiddle.net/Tatarize/4hyx4e06/

Which reads a number of the blocks that it can, that are understood and simply hex dumps the bits it doesn't know yet. But, if things like EmbRect and EmbCirc were a bit better figured out it wouldn't be too hard to simply write that to the svg like I do for all the stitches. We'd simply need to ignore the parts of SVG that don't convert like things like gradients or higher level definitions etc. In theory going from PES to SVG would be easier, (reading is always easier than writing an unknown format), and that would be doable to even make the text, rectangles, punch -> path, etc. All with notable lossy nature losing the information of how those shapes are filled with stitches. But, that project would be entirely doable. Though I'm not sure how many people have fairly intricate PES files that they actually made entire vector pieces out of and would want a fairly worthless SVG representation of vectors used in the project.

[tatarize]

The conversion, even if we used the SVG markup which I cannot guarantee the SVG editing software will maintain or that they will make any sense after the modifications is made, will generally be lossy. We will lose things in the conversion, even if they are commented what they were originally.

In the conversion from proper SVG to PES things are definitely lossy, things like gradients, paths with arcs, stroke widths etc, are all going to be lost. Assuming we breakdown any curves with small line segments, and arcs with small line segments, and maybe just use punch for everything and convert SVGs transforms either directly into actual physical shapes of those positions. An svg rect can be rotated by 45° and so can the EmbRect and all that would need to be accounted for, including guessing where PE Design would put the handles for the object. And to get stitches it would need to be loaded in the proprietary and expensive program "Brother PE design" and saved again to generate stitches.

There's a need for a format that can accurately work as a coherent open source object type that can capture all the intricacies of any embroidery with the least amount of information loss, but certainly SVG is not that format. Because embroidery is not vectors. And I seriously believe they are doing massive harm to embroidery as a whole, because clipart colorforms like drawing is not maximally expressive even if it does have the advantage of being quick and allowing re-editing.

[frno7]

@tatarize, as a simplification I’m happy to ignore interoperability with Brother PE-Design, and thus ignore PES designs, for the following reasons: it is expensive, Windows only, proprietary, unspecified, and there are as far as I know no PES designs of interest if one primarily wants to create original content. In contrast to this, compatibility with embroidery machines is very important. Hence, PEC machine format compatibility is required.

Granted, interoperability with Brother PE-Design may be desirable for some workflows for various reasons but for everyone not planning to obtain that particular application there seems to be no reason to bother with it.

While the idea that vector designs are unsuitable to express all kinds of embroidery may be true, the relationship isn’t mutually exclusive. A broad range of vector designs can be converted into perfectly fine embroidery. For these kinds of designs, SVG seems to be the best choice given the availability of capable applications such as Inkscape.

[tatarize]

Some brother machines have quirky things with PEC as a stand alone format. And it's not out of the realm to support the minimum PES functionalities. This means writing the baseline PES blocks namely the header, the EmbOne/EmbSewSeg blocks. Doing this is well mapped out. I even bothered to write out the step-by-step requirements to do this in the PES document within this project. It's done for EmbroiderModder and Touch Embroidery and fairly easy to do. And a solid plan. Treat that format as it has nothing beyond segments of any real use, just so machines can load them without issue in a proven valid formating (we just omit all shapes and say what we have is explicitly line segments).

Now the missing piece is all of embroidery. For what it's worth, it's the right answer to not bother with PES. While there might be some utility to a PES -> SVG converter at least at the shape level, and it's a solid plan perhaps worth doing. If the idea is original embroidery then really what you're back to is how do you fill shapes with stitches and how to we best represent these methods. And that's an amazingly good question. While the idea of using SVG to do the shapes is fine, great even, especially with all the fantastic tools for editing those shapes, that part is certainly done. And when it comes to stitching, we can simply convert to a bunch of end of the line formats to be read by the actual embroidery machines, so that part is also good. -- So what about that middle piece for how we go from shapes to stitches? Or more generally, how we go from something to stitches (as the over-reliance on shapes as such is in error).

For this, there's the traditional embroidery ideas. You have fills, some of which vary and are coaxed into something the embroiderer wants through a bunch of tricks. Largely the best are tatsumi and satin. Tatsumi is to say an algorithm for filling in a shape to be cut into montone shapes. The algorithm here isn't actually that hard. It's a bit a slog but entirely doable, and I fear most implementations screw it up. You can do it in n log(n) time. Just make the shape outline a bunch of segments (really you could do bezier curves directly, or anything (as long as they are monotone)) and sort a series of points into a list then just keep an active list, run across the sorted list of line segments in order, determining the active edges at any point, and iterating through the list. Touch Embroidery has this for the fill tool and can do it in real time even for very large series of points. You also match the various split and merge points to cut the non-monotone shape into monotone pieces. And then do some other algorithm for filling shapes with predefined start and end points, usually underpath to the top of the shape, the uppermost split node (as things are called in monotone polygons) then when you reach the end node you underpath to the bottom-most merge node (as things are called in monotone geometry) and then fill back up to the desired end point and finish the shape. There's also things like split-lines and breaklines in mainstream embroidery software. Which are used to coax things into what the embroiderer wants. -- I think most of this is entirely wrong.

I think properly one should treat shapes as such as an environment that everything is a sequence of lines and some of which can be very special, and render into larger sequences of lines. That one should consider the entire embroidery project as a series of line segments, including special ones like jumps, cuts, color changes, fills, satin fills, patterned sequences, etc. Though it's not something we tend to consider, it's not impossible to think of a flat fill as having an absolute position within the fill area. So if you have the letter Y and you try to draw it out, you can rightly do so by saying you have a line that does a monotone fill that goes down one side or the other. Or have satin fills which operate such that the fill acts perpendicular to the virtual line segments with in. You can ultimately break down all actions of embroidery into special lines which when they are compiled/rendered within a given environment into lower level objects interact and produce all the results we expect of embroidery (after all, we could have just had normal lines, so it certainly is as powerful as embroidery can be, while being simplified).

When we do this we properly view embroidery as a sequence, we can simply define things like fill in this circle from the top to the bottom as jump to the top of this circle and do a monotone fill to the bottom (with these properties). Then we're at the bottom and do the next thing. If we change the shape, what the montone fill does but doesn't change, only the environment. We need not lock these things together. This provides, at a minimum, a type of embroidery between the high level fill algorithms and the super-basic shapes, and does nothing to harm the possibilities. Since the only real limits on embroidery is graph theory, there's an infinite array of different fills that are never done and can't be defined within mainstream embroidery, but could be defined at this intermediate level.

The idea would be generally to have higher level commands like here's a circle, fill it, at this angle. Then have that commands produce a sequence that requires a jump to the topmost point, a fill sequence to the bottom, with the given properties, and then process the next item in the sequence. So you could implement something like a tatsumi fill, but it simply breaks down to the sequence of special fill sequences, underlay, underpaths, etc, then have that break down to stitches. So if you don't like how the program does the tatsumi fill, you can alter it at the stitch level or better yet, at the pre-rendered sequence level in a coherent fashion. This intermediate step would allow all the embroidery power without having to specify everything. -- Such things would create a sort of nested hierarchy each level renders the level below it. Where each rung produces perfectly valid code for the next rung, so you could specify any task at any level of desired complexity. So you could have the it lay down a series of flat fills. Or simply make some shapes and fill them in some custom but rather brilliant way. Or allow interesting ideas that simply aren't found anywhere. Like subdividing a shape via a voronoi diagram, and define a series of rather fuzzy edged fills to create, for example, the scales of a fish. This would be totally possible as a tool, and could well allow an embroiderer or an embroidery-program creator to make a sort of finger-painting fill tool. Or any of the other very reasonable things people should be able to do but generally can't.

It would also mean that you could run the resulting filetype with other embroidery programs and simply discard the higher level implementations that you didn't implement. And since everything would be very step-wise in the resulting complexity, you could always just use the stitches at the bottom (and convert to whatever machine read format you want), or edit the intermediate productions at a level or two up. It would be able to capture all the power of embroidery, and be able to subsume all high level mainstream embroidery, and allow for innovation and ideas that haven't been seen as of yet, and otherwise weren't possible. And could capture the simple and the complex, all while using the same core open format. -- At least that's my plan. I'm pretty sure a lot of the troubles of opensource embroidery software is because they keep using the flawed paradigm of traditional machine embroidery.

And my idea here has some big question marks like patterns in the needle points of fills, those seem like they are part of the environment of a fill, but would be defined as raster. And how to properly express that this a sequence and this part can also be broken down to these elements. And those elements could also be broken down, so how does one define a sequence and which parts of the super-sequence are the same as the sub-sequence and a few more open questions, like defining raster elements as part of environment like fill this with this set pattern in the needle depressions etc. And which parts could be shared with which programs and how openly define new elements without stepping on any toes. It could at least capture all the mainstream elements as well as doing things like saying, here's some SVG shapes, fill them with thread of this color (and then defining nothing else about) then getting stitches out of that as long as those ideas are rendered in a program with all the needed intermediate steps.

[frno7]

@tatarize, true, it’s easy to support PEC embedded within elementary PES. Like you mention, several tools and libraries do that, including my own libpes. Would you be able to share a link to your PES document? I’ve written a PES format description as well, with some details missing.

Regarding handling of shapes, I’ve created some designs using Inkscape by (manually) dividing complex shapes into monotone polygons made up of pairwise opposing line segments, which are easy for a script to fill with satin stitches. Within Inkscape, which includes an XML editor for extra fine tuning, one can precisely control line orientation and thereby starts and stops, ordering, grouping, layers, etc. for various effects.

Admittedly, this is labour-intensive, time-consuming and somewhat fiddly. Inkscape extensions could potentially make things a lot more efficient. I’m not aware of any other freely available editor where this kind of work would be substantially easier, without losing out on design.

I very much like your ideas of being able to do custom fills, while keeping an eye on traditional embroidery and opening up to new ways of laying out machine embroidery.

[tatarize]

The document I contributed is in the docs section of this project largely appended to the end of the typical document. There, but it does appear to have some corrections over your description. I really wish there were a wiki or something to properly collaborate. But, things like:

CEmbOne Coordinate transformation matrix, etc. CSewSeg Stitch sewing coordinates, etc.

Is actually wrong, it turns out the CEmbOne block is a headerblock for the stitch group object. Namely each group of stitches that you move around within Design PE is it's own object and there needs to be at least one if there are any CSewSeg objects. So the first one is given it's own block. All future ones are appended. If you have more than one Stitchgroup object it appends exactly the same format to the end of the CEmbOne object data. It's highly relevant for the more PE specific file formats. The header:

char 4 "#PES" Identification char 4 "0001" Version 1 u32 4 Absolute PEC section byte offset u16 2 0 | 1 | ? Unknown u16 2 1 (typical) Unknown u16 2 1 (typical) Unknown

After the PEC offset, we have hoop, scale to window, and that final 1 there, is actually the number of StitchBlock objects. If it is 1 then there's only the EmbOne and EmbSewSeg. If it's more than 1. It's the number of independent objects within PE that are being moved around, but it also means the EmbSewSeg block gets the additional StitchBlock headers appended inline. The JSFiddle I posted above will read these types of objects and read all the EmbOne/EmbSewSeg blocks, and actually appends the EmbOne block in front of the EmbSewSeg block because that's really what they are. Given that if there's more SewSeg blocks there they get appended it natural to treat those blocks the same way. Also, the EmbOne/SewSeg-header number there is the number of stitch objects within that block which is not exactly the CSewSeg block count, but the blocks that follow the given header.

The posted Fiddle ( https://jsfiddle.net/Tatarize/4hyx4e06/ ) is actually largely a better duplication of the document. It takes a given file, and parses it, even the parts that are basically redundant and reads out exactly what, to the best of my knowledge, those bits do. Including accurately parsing all the sew-seg blocks and listing out what some of the previous unknowns in Version 1 header do. A lot of these were figured out by hex editing a valid PES and loading it in PE Design and figuring out what changed. Things like knowing the hoop size is toggled by the first Unknown there basically involved fiddling with it. There's also stuff in later bits that do things like change the background color of the viewing board which is utterly pointless outside of PE design.

I was planning on doing things like making an SVG for each parsed elements of the PES file and parsing some of the shapes. But, honestly the way Wilcom and Embroidermodder do it is correct, I also layout exactly what to do use that in the document and what it means (which you mostly already know) if you have such output working already. The Fiddle was just such a slog to basically parse all the elements, even the ones I didn't care about, that I burnt out on it.

---

In addition to the document I emailed Trevor some points of errata:

These aren't in the document, and you seem to have properly accounted for them on your side initially, but they should have been included in Trevor's document and weren't. My initial rundown contained some oversights and errors.

---

1) The color change log after the SegmentsBlock is 2 bytes, size (not 4) and the elements go first-segment-used, color (not color, last segment used).

This error is because these two radically different descriptions are the same in the non-professional manner, saved out by OESD, Wilcom, Touch Embroidery, Embroidmodder, etc.Because the first segment position that you use the first color is always segment 0. So the start bytes 10 00 00 00 cc cc xx xx is the same as 10 00 XX XX cc cc, when the first index is 0. Then at the end of the CSewSeg block we get 00 00 00 00. These are still unknown, but properly seem more like FF FF 00 00. --- This ordering becomes apparent when we process the multi-segmentsBlock files and we end up eating not an unknown 00 00, but rather the 0x8001 that flags the block's end. Also, it makes coding that that stuff easier. As the log is the first position rather than last position, so you don't need to keep that information around. Just add a new log entry when we see the new color.

2) My previous form of the document suggested that the CEmbOne format had the copies of the dimensions, but the versions in the CSewSeg did not. This is wrong. The CEmbOne format is identical to the CSewSeg block only difference being that the version in the CSewSeg block simply has a the terminator appended (0x8003). To this end my demo reader there clips out the CEmbOne block and gives it to the parser for the CSewSeg block.

---

The termination sequence for the PEC code blocks is not FF 00, but simply FF. PES empty files which is to say a file with no stitches reads, starting from 00 in the PEC block:

00 00 0D 00 00 31 FF F0 E8 03 E8 03 FF 00 00 00 00 00 00 F0 FF FF FF FF 0F 08 00 00 00 00 10 04 00 00 00 00 20 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 02 00 00 00 00 40 04 00 00 00 00 20 08 00 00 00 00 10 F0 FF FF FF FF 0F 00 00 00 00 00 00

You will notice that the file started 31 FF F0, then did a stitch of E8 03, and E8 03. Then terminated FF. You can see there are 6 sets of 00 after the FF. If FF 00 were the terminator Brother's software would have FF followed by 7 sets. The value there is always 00 anyway. But, if you were to read to the terminator assuming FF 00, and then read the graphics, you'd have 5 sets rather than the needed 6. Clearly because reading the graphics jumps via the graphics value, and that value is always 00 anyway (though this might not be a requirement by the actual machines themselves), as they seem to take any and all data and the frames are just candy. The 00 there is not the terminator. Though it would fall in the gap between the stitches and the graphics and be ignored when correctly done. And when incorrectly done, will find a 00 there anyway. And basically nobody reads the graphics but if they did they'd still likely always get a 00 anyway, but that 00 belongs to the graphics and not to the stitches.

[frno7]

@tatarize, it would be very useful to have a test suite of known to be correct PES files showcasing various features and versions, as produced by Brother PE-Design. Would you happen to have appropriately licensed PES files that can be distributed for test purposes?

I have a small set of very simple demo files that came with my machine, but I’m unable to redistribute these due to licence restrictions.

I’ve just created a PES format description wiki and you should be able to edit, if you wish.

[tatarize]

To make the more complex PES files I simply saved something in PE Design (Version 6 is max I can do). I really haven't seen anything that actually uses any of the PES shapes intact. They are basically always just converted to the various forms and distributed like that without any design information for editing. Even the highly copyrighted stuff I have only has the one layer of all the stitches. Nothing is at all broken up into pieces much less shapes like ever. If you open any design in PE and resave it, it will make multiple sewseg layers.

complex.zip

[tatarize]

It's an interesting point. How useful can such a program be if I've never even seen anything in the wild that fits the bill? I just saved the stuff I needed and assumed such files existed, but I'm not really sure they do.

Complex is just a series of shapes, I slapped together (and assert no copyright on / freely license), some with different colors and in different orders so it's not just rectangles in order.

A quick something broken into different line parts of which I own the copyright (made in Touch Embroidery). merrychristmass-brother.zip

[frno7]

@tatarize, interestingly, your file complex.pes didn’t contain a CEmbOne section. I’ve added placeholders for the undescribed sections CEmbCirc, CEmbRect, CEmbLine, CEmbPunch, CSewFigSeg, CLetter and CEmbNText in the sections table of the wiki. Feel free to fill in details and corrections if you know how these things work.

I’m considering how to extend the PES overview, how to best describe the differences between the design and machine (PEC) sections, how to indicate mandatory sections, etc. Of special interest are mandatory sections for a minimal machine interpretable PES file. Also, perhaps the terminology needs to be updated, as I’m unsure what several of these things are really called...

[tatarize]

It must also lack a CSewSeg block too. I added what I thought was a segment list but it must have been a CSewFigSig (I assume Sewing Figure Segments). It is the case that the CEmbOne is really one block in two sections. I've never had one without the other. I even tried to coax it by saying I had zero segment lists in the block in the header and tried to leave a blank header and omit the header. If memory serves both those attempts crashed the program.

I'd figure they'd have fixed that in later versions. I wouldn't be shocked if the first thing in a lot of these blocks is data that is basically identical to CEmbOne appended to the start of the later classes.

But, yeah, unless you have those that class of segment the EmbOne block is always omitted. It's true for simple shapes as well. Here:

Circle, Rect, Circle, Rect, Circle, Circle, Circle, Rect Rect Rect. shapes-CRCRCCCRRR.zip --- It'll have a bunch of stitches but no CSewSeg block.

Here's straight up blank. Opened the program saved the file. With nothing. blank.zip

[frno7]

@tatarize, your file blank.pes is very interesting because it suggests that the header and the PEC are the only required sections. PES files can be significantly simplified and reduced in size if embroidery machines generally accept that CEmbOne and CSewSeg are omitted.

Furthermore, the header doesn’t end with the typical 0x0000FFFF token. This suggests that the token more probably is a preamble rather than a postamble for sections.

[tatarize]

Yeah, the only thing including blanks for all that will do is make Brother's Design PE the only program that cannot open them. You're right that you can totally just omit all that stuff. And what's more, all programs that aren't Brother's own software will load the PEC stitches as nobody's mapped out the other elements. But since there's stitches for all of that, you can always simply get the design from the stitches, regardless of the version or any other consideration. And since nobody seems to release the other elements anyway, you won't miss anything. -- You're totally right. It would make coding PES downright trivial, and the only problem is that the program they are properly intended for, won't be able to read them but everybody else could do so easily. And you could use something like Wilcom to translate the files from PES to PES. Or I guess simplified PES to PES.

[tatarize]

I don't know how many programs would care if you omitted the header too. The machines might, but all the PEC stuff is intended for the machines. The headers are really just details about Design PE and the settings. I mean if we'd be sacrificing Brother's software anyway I would be totally shocked if any software cared if the file started with:

23 50 45 53 30 30 36 30 0C 00 00 00 4C 41 3A 62 6C 61 6E 6B

PES0060....LA:blank

Where 0C is the start of the PEC block, namely 12 bytes in, Wilcom loaded it fine as saying nothing. I appended that as the entire PES block and Wilcom and OESD .ARTsizer opened it fine. I would be surprised if anybody had issues with that:

23 50 45 53 30 30 36 30 0C 00 00 00

Rather than the anything else. Literally the only difference between this and the PEC filetype is that it has a PES block of 0C 00 00 00 which says, the PEC block starts exactly after this number.

[frno7]

@tatarize, I believe the smallest valid PES header is the 22 bytes of version 1. Not sure version 6 can be validly truncated with a short 12 byte PEC offset since a complete header probably is expected, especially since version 6 contains a thread color list subsection that replaces the PEC palette of versions prior to 5. A complete version 1 header can be coded as simply as constant

23504553303030311600000000000100010000000000

regardless of the content of the PEC section that follows. My Brother Innovis 955 sewing and embroidery machine happily accepts and runs circle.pes.gz which is encoded in this way. Perhaps Brother PE-Design will open this PES file as well, at least without crashing? (If not, some bits might need to be flipped in the unknown parts of the header. I replaced the 0xFFFF0000 token with 0x00000000 as suggested by your blank.pes file.)

[tatarize]

Hm. I just considered clipping the header because I'm effectively sure that only Brother's software ever reads anything beyond the seek location to the PEC block that anybody cares about. But, you're right, technically 22 bytes simply as a constant would be a valid PES, and should be read by the machines as such even if a strange one ever looked at the header for some weird reason.

Here's the blanks for versions 1, 2, 3, 4, 5, 6. blankversions.zip

[tatarize]

Circle.pes caused a crash. Problem Event Name: APPCRASH Application Name: Embedit.exe Exception Code: c0000005 Exception Offset: 0007d37b

The issue is your header contained a 1 in the number of segment blocks so it looked for them and they didn't exist. But, see blankv1 header. I changed your two 1s into zeros and it loaded up without incident. It showed nothing in the file (as Brother's software only reads PES and not PEC) but it loaded up as perfectly valid.

[tatarize]

But, yeah, I have to suppose that it would be very helpful to a person to tell them they can get away with calling the entire PES part:

23 50 45 53 30 30 30 31 16 00 00 00 00 00 00 00 00 00 00 00 00 00

In a pinch and it will be perfectly valid. And I checked your 1 at 0x0E crashed it:

23 50 45 53 30 30 30 31 16 00 00 00 00 00 01 00 00 00 00 00 00 00

Is also valid.

8 bytes: "#PES0001" 4 bytes: Location of PEC block. 2 bytes: 1 (scale to fit) 2 bytes: 1 (hoop size, 130mm 180mm) 2 bytes: 1 (we are going to write only one stitchblock object).

23 50 45 53 30 30 30 31 16 00 00 00 00 00 01 00 01 00 00 00 00 00

I take this to mean, we do not scale the diagram, what I took to mean the hoop, but you say there's going to be a stitchblock object, but it categorically cannot seek to the CEmbOne object so it crashes.

The 0x0e block switched between 100mm x 100 mm and 130mm 180mm so I took it to mean hoop. But, here's the two files with that bit set and unset and the only change I could find with blank:

[tatarize]

If those blank boxes are a bit too curious looking, I filled them all in and saved blank files again. (I loaded from a file with 0x0e=1 set)

metadata.zip

[tatarize]

And looked for the setting that 0x0e bit messed with and found it.

[tatarize]

There's still a couple things in this damned v1 header that I don't actually get. But, I googled a v2 manual, so I might actually be able to track down what it could be. Though, if somebody had a copy of PE-DESIGN v1 it would be almost trivial.

http://support.brother.com/g/b/manuallist.aspx?c=us&lang=en&prod=hf_design4eus&flang=English&type3=194&type2=69

[tatarize]

Yeah, 0x0C toggles hoop from 100mm x 100mm to 130mm x 180mm. 0x0E toggles Existing design area. 0x10 is the count of the stitch blocks. If 0 there is no CEmbOne/CSewSeg object. If more than one there are more than one. These are simply scrunched together. 0x12-0x15 are control events. 00 00 00 00 terminates the read of the header.

PE-DESIGN doesn't read that seek address after the version. It doesn't care and doesn't parse the PEC block.

Upping the 0x16 to 0x20, xx xx xx xx 00 00 00 00, adding trash 0x12-0x15 caused a crash. 00 00 00 00 xx xx xx xx, it ignored any trash put in 0x16-0x19.

If I told it I had blocks (0x10) but gave it 00 00 00 00 there it said unexpected file format. If I said I had no blocks (0x10 = 0) and it FF FF 00 00 it said it said unexpected file format.

If I told it I had blocks with a proper FF 00 code, then at the end changed the 00 00 00 00 before the PEC block: FF FF 00 00, said "Failed to open document." 00 00 FF FF, said "Failed to open document." 00 00 00 00 -> 00 00 00 00 FF FF, no problem with document. 00 00 00 00 -> FF FF 00 00 00 00, crashes program.

Any fiddling of the FF FF 00 00 data between the CEmbOne CSsewSeg blocks, crashes.

Saving the complex.pes to v1 only gave it CEmbOne/CSewSeg blocks.

complexv1.zip

---

Completely deleting all aspects of PEC gave a file that opened the file without error with all the relevant line data etc:

complexpecremoved.zip

Seek value was set to 0, and the entire PEC block was removed and it had no issue with this.

There is a preview in the open dialog that does parse the PEC stitches to do a quick preview, and it said unexpected format. Wilcom opened the file, said it was fine and showed me garbage on the screen. Apparently it read the PES stuff since I told it to seek to zero and had no issue with this.

Also, your blank PES idea works fine for that, Circle.pes showed the circle in the preview of the open dialog.

---

Best as I can tell that's a complete description of PESv1.