Wiki Improvements

[tatarize]

I properly classed the control codes. They are not parts of the pes blocks. The 0xFFFF0000 between the sections are control codes, they indicate whether the file is over or if there is more to be read. They are not therefore part of the blocks themselves. They are codes to indicate if there is is something more to read it can properly gauge EOF.

I moved some sections around, and added the block description from Josh. If his data is right, it's certainly possible to parse all the higher level objects and transcode them at least naively to SVG. Even though that goal is largely a bit meh, given that nobody has actual PES files with higher level objects.

[frno7]

@tatarize, great improvements! I’ve collected a couple of notes:

• The “control code” byte sequence FF FF 00 00 is actually 0x0000FFFF which is equal to 0xFFFF (and not 0xFFFF0000) when interpreted as little-endian u32, as defined in the Basic types section.
• The top, left, right and bottom fields of CEmbCirc, are these really u16? It seems s16 would make more sense? The unit of the start and end angles would also be helpful to note. Could it be degree? Same goes for CEmbRect, and its corner radius, CEmbLine, etc.
• Some sections like File format repeat technical details like endianness and header structure that are described elsewhere. Eventually we ought to tighten this up.
• If possible, I believe we should aim for having known to be correct sample files to cover all concepts described. These files can then be used for testing library implementations.
• Some words like section and block seem to be reused for more than one concept which is somewhat confusing. Some rubrics seem better when shortened for clarity, for example “PES CEmbOne” instead of “PES CEmbOne section” where the word ‘section’ seems redundant. Eventually we ought to decide on a naming convention and stick to that throughout the text. Perhaps a clearly defined glossary page would be helpful for this? Technical texts or documentation from Brother would of course be helpful in determining proper terminology.
• I believe it would be slightly more correct to say that PES embeds or is a container for PEC, rather than PES being completely separate from PEC.
• I‘d like to edit the text to be more stylistically coherent (encyclopedia style, if you will). That said, English is a secondary language to me and so my grammar is a bit weak.

[tatarize]

1. Totally right.
2. All positions are integers. It's only the affine transforms that use floats. Everything else is shorts.
3. Yeah, it's a bit loose. I was focusing more on the understandable structure.
4. Test files aren't too hard, although there are some rarer things that for some reason are not trivially produced there is version 2.5 literally #PES0025 and 5.5 and mapping out the parts accordingly should certainly be subsets of 3 and 6 respectively but would take some fiddling. But, they could certainly be done. But, a lot of the header info is basically pointless settings for the program stuff. So you could verify you got it right without trying it.
5. There is a clear terminology issue. It's needs a standard. There's no internal information what those things should be called. There's also PES section and .pes file. So it's a bit cryptic to refer to the 3rd SewSeg block in the SewSeg block in the PES section of the PES file, and note that the number of SewSeg blocks was set in the PES header, but the number of segments within that SewSeg block was set in the header for the third SewSeg block in the SewSeg block which is the second header within the SewSeg block because the SewSeg block's header #1 is in EmbOne. -- I've not seen any technical documents anywhere outside a manual that covered features.
6. Oddly enough there is a PEC format. You can produce .pec files, they are literally the header #PEC001 then the PEC section. No reference code no nothing, seek 8 bytes in and you're now in the PEC block. If PEC has a stand alone component then it is the case that it is actually different and separate.
7. I was stylistically going for understanding. It's the reason why I divided things up into sections that could have been divided up in other ways. And tried to subdivide the sections that occur repeatedly and the elements that we see repeated. And perhaps excessively editorializing why the format would be like that rather than some other way. While I understand that style might not be the best for explaining a technical document, it sometimes certainly requires declaring things in one place and supposing they've been explained and using them.

The fields on the circle, cembrect and cembline are taken from Josh's work on the subject. But, yeah, I realize that that needs a bit better explanation.

The units are 1/10 mm. The angles units are 1 degree. The units within most all embroidery is 1/10mm because that's the minimum you can tell the machine to move. So it gets imbedded in a lot of places.

The explanation that Josh gave on the main unit elements gets the block description, he called it a stitch-block-description and it's actually describing the fill applied to that shape. And yes, it honestly appears to store all the settings for all the types of fills even if that type isn't being used. But, that data is enough to be able to parse the files, and figure out the size by skipping the motifs which can have their size read (if they exist) and get to the core elements within the Circ and Rect and Line objects and transcode those to SVG. If you ignore all that fill data, the actual elements themselves are pretty easy. Circles have, Circle, Arc, Fanshape, Arc and String (I think they mean chord, as that's what it draws).

You can pretty easily turn a good amount of that in the very basic forms into svg. And it's the only big element I hadn't bothered to look into. There's something with motifs still not listed out. There might be some shapes that aren't shown up, but I think that's everything big.

[frno7]

Two 16 bit integers with different semantics are defined: the s16 is a signed integer specifically intended for the stitch coordinate space (0.1 mm unit), whereas the u16 is a generic unsigned integer for all other cases. The only difference is the interpretation. If a generic signed 16 bit integer would be needed, I propose introducing i16, but it does not seem to be necessary? So for example CSewSeg coordinates are s16 but for some oddball reason the hoop size is 1 mm unit and therefore u16, with the 1 mm unit explicitly noted in the description. Having a distinct s16 type saves us from annotating the 0.1 mm unit in lots of places. Whenever we have measurements such as angles, coordinates, lengths, sizes, etc. we ought to use the appropriate types (s16 is a common case) or annotate the units (for uncommon cases not having distinct types, currently basically everything except s16).

I propose using “PES format” and “PES file” (as opposed to “.pes file”). The PEC format seems to be a subset of the PES format. That is, PES subsumes PEC. However, as you note, the converse is not true since stand-alone PEC files also exist. So what is currently said to be the “PES section” is more accurately the “non-PEC parts of PES”. Eventually, I’d like to clarify this in the text as well as in the documentation structure.

Getting accurate content in place is in my opinion clearly more important than the style of writing, which is secondary, but eventually I’d like to edit some wordings. For example, “hijacking” is a colorful way of saying “extending the PES format.” Another example is “the PES part contains the design information ... which only Brother's software reads or cares about” which appears to be overly restrictive given that Bernina Embroidery Software Designer Plus seems to support PES to some extent, and possibly other programs. In this case I’d like to change the wording to say that Brother PE-Design is an example that uses the design data, avoiding the assertion that it is the only program that does this. Similarly, I consider “impossible to do operations” a quite strong wording, and I would prefer “difficult” instead of “impossible.” Another change I’d like to make is to use the proper multiplication sign ‘×’ instead of the asterisk ‘*’.

It’s very convenient to cross-link compound types, such as description_strings which is used as a type in various PES header versions. A problem here is that the type (description_strings) in the tables does not match the heading (“Description strings subsection”) in the document, and I’m not (yet) sure how to improve this.

[tatarize]

Some of that is just an oversight, and yeah, it certainly needs to be more consistent.

For the coords, I didn't check but I'm pretty sure they all fit the pattern-number of coords, (x, y)+. Generally because they would sort of need to do it that way. Which would in theory make a nice object of coord array. But, whether that is helpful depends on the reuse it has in the document.

For simplicity, since the fill settings (block description) is such a recurring element it might be proper to put all the shapes on the same page rather than giving each a different page. They are generally tiny and a lot of the data is generally redundant and they can be described as a whole.

The CSewSeg blocks are certainly different but most of the later blocks (whenever their version came out) are pretty similar, though there is clearly something to putting the CSewSeg block and the CEmbOne block in the same page too. To point out very clearly their strong relationship. The CEmbOne page is basically just an outline and points to the CSewSeg page. It stores the first of potentially several headers within the CSewSeg block.

When the project seemed more unwieldy, a bunch of pages one for everything seemed like a solid idea. But, given Josh's description of the information about the fill and how little else is there. It seems pretty simple, and doesn't need all those pages. The CEmbRect and such are basically completely described. The circle likely needs some information on the types and honestly such a thing would require images.

PEC does the embroidery machine. PES v.1 has EmbOne/SewSeg blocks containing segment data. Later versions (v6 at a minimum) have shapes, which are a big block of data that describes the fill, and the couple things you need to describe a shape. And tada, that's the format! Mapping out that fill description block thing is the very heavy lifting. I'm fairly sure I could change my javascript fiddle to interpret at least the basic shapes and transcode them into SVG.

This would be a solid proof of concept, even though it would be ultimately pointless. Nobody has embroidery files with those shapes in them. If you went SVG -> PES you'd get shapes but no stitches and have to load the file up with PE Design and save them again to generate them. I write embroidery software and I'm still not sure what any of this information is useful for.

Except for that solid blank version 1 header as the entire PES document idea you had. That's actually useful. Which doesn't bode well, given that the most useful thing to realize about PES is that you can easily pretend it doesn't exist.

---

I don't have a Brother embroidery machine to test PEC stuff. But, I did get images of a corrupted version of the graphics from a user of Touch Embroidery who took a picture of her embroidery machine and the crazy bitmap it was displaying. And Josh wrote me up a pec image displayer. So I could test such things for their integrity. Which was a big help, in fixing it. -- But, it does show oddly that the frame around the images isn't really needed, it just looks pretty. But, you might be able to answer questions like what if you said you had one more color and never used that claimed color in the stitching but did include a bitmap of a program logo, would any machine throw an issue at that?

[bungaloretorpedo]

I may be able to add something to this. I've been using PES version 9 as a vehicle for learning python for another job, and I have it probably 90% or more mapped out. I thought someday I'd make the code multi-version compatible, but its been a while getting back to it. My Git-fu is weak, but let's see if this old output file loads...
7thid.pdf

[frno7]

Thanks @bungaloretorpedo!

[tatarize]

"element stitch order" is one of the things we didn't have mapped out that was certainly critical. Most of the stuff is backwards compatible so there's things in the 9 header that look to be number of segments-groups too, that you have blanked out. Or are in the right place given the version 6 header. But, then again, maybe they dropped them, I am unsure why but Wilcom Truesizer will only output sections versions 1-6. And generally won't touch the vector shapes no matter what.

[bungaloretorpedo]

I had a half-dozen or so 'next steps' in the works when I got caught up in other things, V6 header being one of them, as it was the format I digitized in before the new machine and software came home. Depending on how much 6 and 9 have in common in the standard shapes, I think the PDF above takes care of some unknowns. Not obvious because it normally has a zero value, the unknown following the CembCirc block description is the number of 16-byte of data structures to follow before the actual "tail". Those 16 bytes include location and index information for embossings that may have been applied to the CembCirc. Given how long I'd been staring at it, it may never have come to me about the start-end angle fields in the tail. Thank you for that, I've already changed the output display of my own code.

[tatarize]

Yeah, with Josh, and others it's basically mapped out. There might be a couple things left, but I'm pretty sure that's basically all of it. I'd like to see it produced as a better technical document and just published as such. I'm really not sure what else needs to be done. I suppose we could convert the shapes to SVG and convert SVG to the given shapes. But, the software is rare and expensive and I'm not sure the utility therein. As I'm soon going to be writing my own fill these shapes with stitches code, and for the life of me I've looked and never seen any PES with shapes file anywhere other than saving one myself.

[tatarize]

@bungaloretorpedo I fairly recently translated this stuff into python myself for a different effort. But, using the stuff learned here to properly map the formats. While it's mostly intended for version 1 and 6, it would certainly work for other versions. It's mostly for all formats reading and writing. But, has a particularly developed read and write routine even reading the pec graphics.

https://github.com/EmbroidePy/pyembroidery

[tatarize]

And keep in mind the work here and elsewhere also contributed to:

https://edutechwiki.unige.ch/en/Embroidery_format_PES

Which I think is to date, some of the best work on the subject.