PES file version 7 8 9 10 header

[quangnt183]

Could you implement code to read PES file version 7, 8, 9, 10 Header data?

[tatarize]

Probably. Though I think a cursory look into it had very little of any importance. Most of the metadata mapped out well enough already, and the PEC block is where most of the loaded stuff was going to end up. I think it tries to read it, based on the map of the version 6 header and if it does something weird it aborts and just loses that metadata.

[quangnt183]

Yeah. I need to read the colors of thread that saved in the header metadata. I am reading it ok from version 1 -> 6. But using same rules of version 6 for version 10 is not working now.

[tatarize]

Hm. Okay. Some the header stuff is weird stuff like special stitch types with like motif arrays, and a couple bytes unprocessed tends to make it more fail. And that stuff and whatever they added in those versions is likely unneeded. Maybe there's some clever way to read that thread data without processing that stuff. Since it's usually the last items before the other block types. Also, most other software doesn't save anything other than version 1 and version 6. So I didnt map it all out. It might just be a couple bytes to get many of those files working, though if they did something weird it might still have to abort out and not read that data.

[quangnt183]

When I create new design, or export/resize existing one by wilcom E4 software, it is version 10 (it doesnt let me choice the export file version). I have thought about a clever way to read thread color both from start or from end of header but could not found any reliable way. But I think I should try again. Thank a lot for your suggestion.

[tatarize]

Hm. So long as it's E4 and is routinely the same, it shouldn't write any of the fancy stuff to the file. I think just a blanket guess for where in the file you should look would work pretty reasonably. I tried running the reading of 6 into 10 and firstly I had the string as "PES0100" rather than "#PES0100" but it overshot the location by a bit, so apparently stuff was removed from the header. I'd think a reasonable guess at the header should give pretty consistent results.

[tatarize]

In version 10 it looks to be around 0x72-0x73 for the number of threads. Then the threads get read the normal way. So depending on what got cut it should just be going to the right location and read that info. Wilcom would never write anything weird in there, so a seek to that location for that version should just read the data properly.

[tatarize]

def read_pes_header_version_10(f, out, threadlist):
    f.seek(4, 1)
    read_pes_metadata(f, out)
    f.seek(36, 1)  # this is 36 in version 6 and 24 in version 5
    v = read_pes_string(f)
    if v is not None and len(v) > 0:
        out.metadata("image_file", v)
    f.seek(57, 1)
    count_threads = read_int_16le(f)
    for i in range(0, count_threads):
        read_pes_thread(f, threadlist)

[tatarize]

I'm not sure if there might be feather or motif patterns but I could likely change the seek from 36 to up to where it might be valid. Assuming there might be motifs, program fills or feather patterns.

Though there might be landmines somewhere it looks like the readheader for 10 might be:

def read_pes_header_version_10(f, out, threadlist):
    f.seek(4, 1)
    read_pes_metadata(f, out)
    f.seek(36, 1)  # this is 36 in version 6 and 24 in version 5
    v = read_pes_string(f)
    if v is not None and len(v) > 0:
        out.metadata("image_file", v)
    f.seek(51, 1)
    count_programmable_fills = read_int_16le(f)
    if count_programmable_fills != 0:
        return
    count_motifs = read_int_16le(f)
    if count_motifs != 0:
        return
    count_feather_patterns = read_int_16le(f)
    if count_feather_patterns != 0:
        return
    count_threads = read_int_16le(f)
    for i in range(0, count_threads):
        read_pes_thread(f, threadlist)

But, it's kinda hard to say without adding some programmable fills or motifs to see if that is the same end part. But, with it jumping by 51 rather than 24 there it hits the read location correctly and reads that data.

[tatarize]

Tried tossing an image in, and it missed the mark, so the image file position and final seek positions are a bit off. But, really this isn't that hard of work. At least not to get it kinda working. Though Wilcom wouldn't add an image reference to their files either.

[tatarize]

With an image file this is correct:

def read_pes_header_version_10(f, out, threadlist):
    f.seek(4, 1)
    read_pes_metadata(f, out)
    f.seek(53, 1)  # this is 36 in version 6 and 24 in version 5
    v = read_pes_string(f)
    if v is not None and len(v) > 0:
        out.metadata("image_file", v)
    f.seek(34, 1)
    count_programmable_fills = read_int_16le(f)
    if count_programmable_fills != 0:
        return
    count_motifs = read_int_16le(f)
    if count_motifs != 0:
        return
    count_feather_patterns = read_int_16le(f)
    if count_feather_patterns != 0:
        return
    count_threads = read_int_16le(f)
    for i in range(0, count_threads):
        read_pes_thread(f, threadlist)

Seems to hit the right spot and read correctly, I'll check through the other versions and code up something quick to hit the right spots.

[quangnt183]

thanks alot, I am following!

[tatarize]

Okay, mapped them all out. At least I think. There's a jump of a number not divisible by 2 between 8 and 9. Which could be a text element. Since those tend to have a single byte of length followed by text. Which makes it a variable length issue. The jump goes from 38 to 45. The even jumps are going to often be fine, but variable length elements of non-mapped items are a bit spooky. But, I should have the basic mapping with the break for where image file should be and strong assumptions that the start is metadata and end is fills, motifs, and feather patterns.

[tatarize]

https://github.com/EmbroidePy/pyembroidery/tree/tatarize-pes-7-10

That should do it, but the odd byte shift means something. I either need to find anything else that might be variable length in the header and map it out, or be able catch avoid any error if it reads bad data and abort the header read, in case it hits a non-fully mapped file. That branch should do for now though.

[tatarize]

The seeks slightly change between versions, with additional settings. All the V6 stuff was just elements of the program being saved into the header. But, they were all like little endian 16 bits. So even added values like the 36 going to 38 is likely fine, but the shift from 38 to 45 means there's a variable length factor in there and I likely need to find it and map it out, or catch any errors from reading the thread. I can't really merge it until I know it's safe and isn't going to crash on some files it would have previously read. But, for things like Wilcom, where the amount write stuff is going to be static and options won't be included, it will be perfectly fine.

v7: f.seek(36, 1) f.seek(24, 1)

v8: f.seek(38, 1) f.seek(26, 1)

v9: f.seek(45, 1) f.seek(34, 1)

v10: f.seek(53, 1) f.seek(34, 1)

[tatarize]

Checked the programmable fills in version 10. They were where I assumed. So odds are good that code will work fine without any issue, even if there is a variable length item. Odds are good the checks at the end will ensure it will fail gracefully. The added stuff is like "Background Image Density".

[quangnt183]

Could you please check your new code with this file ? https://drive.google.com/file/d/1FtEfTRAXIrYQpt1CHlnrSkBO1Xt2qd4i/view?usp=sharing

[tatarize]

p = PyEmbroidery.read('M Power111.PES') p.threadlist [EmbThread(thread='#0a55a3', description='Blue', catalog_number='2', brand='Brother', chart='Brother'), EmbThread(thread='#0e1f7c', description='Prussian Blue', catalog_number='1', brand='Brother', chart='Brother'), EmbThread(thread='#ed171f', description='Red', catalog_number='5', brand='Brother', chart='Brother'), EmbThread(thread='#134a46', description='Peacock Blue', catalog_number='38', brand='Brother', chart='Brother')]

The code in the file is 2, 1, 5, 38. There appears to be a section that describes the hoop, lemme double check that is a normal metadata thing that's caught.

[tatarize]

Ah, yeah, the hoop description in there is absolutely a variable length object. It got the correct colors from the PEC block rather than PES. It just happened to have simple colors that were readable from both.

[tatarize]

p = PyEmbroidery.read('M Power111.PES') p.threadlist [EmbThread(thread='#0a55a3', catalog_number='002', brand='PES'), EmbThread(thread='#0e1f7c', catalog_number='001', brand='PES'), EmbThread(thread='#ed171f', catalog_number='005', brand='PES'), EmbThread(thread='#134a46', catalog_number='038', brand='PES')]

With that commit it should now catch that metadata spot. I'll check back if that data saves into older formats and try to fix those too.

https://github.com/EmbroidePy/pyembroidery/commit/b7ae68b1ebbc8d9b31323d422a73dd88621d6b1e

[quangnt183]

Thank you, the latest code working great with version 10, I can get correct thread color now. I have not had chances to test with version 7 8 9 yet.

[tatarize]

Yeah, used the version in 10 to saveas in 9 and 8. hoop_name as a metadata item in the header showed up between version 8 and 9. And explains the odd number of bytes. I updated version 9 to parse correctly if there is a hoop_name. And prior to 9 that item doesn't exist. So it unless there's a pair of other variable length data that should account for the variable length items, and work correctly.

Also, as a side note. In Wilcom e4, with the export embroidery file, there's an options button which brings up the save options which has save version type for PES.

[quangnt183]

:( oh my god, that's so true I am a newbie. Actually I am working on a web module that count the number of stitches and also determine the color in a PES file. Thank you!

[tatarize]

Well, if it's of any use to you, I did write a parser for some parts of PES stuff in Javascript for some reason that I can no longer recall.

https://jsfiddle.net/Tatarize/sL8z1vmh/

[quangnt183]

Thank a lot, I am really appreciated your help!