Encoding error on Pico-8 glyphs (or possibly: AttributeError: 'bytes' object has no attribute 'hex')

[dagondev]

Wrote this at bbs, but not sure if you are looking there anymore. (http://www.lexaloffle.com/bbs/?pid=34133#p34133)

Dear dddaaannn.

I am trying to use your picotool to minify my game. Currently I have: 65302/65536 char count and 8136/8192 token count. When I am trying to use luamin on my cart I get:

C:\Python34>python p8tool luamin saveyourself.p8
saveyourself.p8 -> saveyourself_fmt.p8
warning: token count 29022 exceeds the Pico-8 limit of 32768Traceback (most rece
nt call last):
  File "p8tool", line 8, in <module>
    sys.exit(tool.main(sys.argv[1:]))
  File "C:\Python34\pico8\tool.py", line 512, in main
    return args.func(args)
  File "C:\Python34\pico8\tool.py", line 349, in do_luamin
    return process_game_files(args.filename, luamin, args=args)
  File "C:\Python34\pico8\tool.py", line 210, in process_game_files
    procfunc(g, out_fname, args=args)
  File "C:\Python34\pico8\tool.py", line 242, in luamin
    lua_writer_cls=lua.LuaMinifyTokenWriter)
  File "C:\Python34\pico8\game\game.py", line 746, in to_file
    self.to_p8_file(outfh, *args, **kwargs)
  File "C:\Python34\pico8\game\game.py", line 659, in to_p8_file
    for l in self.gfx.to_lines():
  File "C:\Python34\pico8\gfx\gfx.py", line 95, in to_lines
    yield bytes(newdata).hex() + '\n'
AttributeError: 'bytes' object has no attribute 'hex'

I have ton of text packed into loooong strings if that helps. Please help me, as I can't even export my game to html at this point.

Thanks in advance!

[dagondev]

In Python 3.3 and 3.4 I get error as specified above and it is related to the fact, in P3.3 way of obtaining hex changed: http://stackoverflow.com/questions/12917278/how-to-use-the-hex-encoding-in-python-3-2-or-higher

In Python 3.2 I get: c:\Python32>python.exe p8tool luamin saveyourself.p8 saveyourself.p8 -> saveyourself_fmt.p8 Traceback (most recent call last): File "p8tool", line 8, in <module> sys.exit(tool.main(sys.argv[1:])) File "c:\Python32\pico8\tool.py", line 512, in main return args.func(args) File "c:\Python32\pico8\tool.py", line 349, in do_luamin return process_game_files(args.filename, luamin, args=args) File "c:\Python32\pico8\tool.py", line 210, in process_game_files procfunc(g, out_fname, args=args) File "c:\Python32\pico8\tool.py", line 242, in luamin lua_writer_cls=lua.LuaMinifyTokenWriter) File "c:\Python32\pico8\game\game.py", line 746, in to_file self.to_p8_file(outfh, *args, **kwargs) File "c:\Python32\pico8\game\game.py", line 634, in to_p8_file if transformed_lua.get_char_count() > PICO8_LUA_CHAR_LIMIT: File "c:\Python32\pico8\lua\lua.py", line 55, in get_char_count return sum(len(l) for l in self.to_lines()) File "c:\Python32\pico8\lua\lua.py", line 55, in <genexpr> return sum(len(l) for l in self.to_lines()) File "c:\Python32\pico8\lua\lua.py", line 153, in to_lines for line in writer.to_lines(): File "c:\Python32\pico8\lua\lua.py", line 287, in to_lines strs.clear() AttributeError: 'list' object has no attribute 'clear' Clear was added in P3.3...

In P3.1 there isn't argparse module, so it is completely broken.

[dansanderson]

Thanks for the report! I'm currently unable to reproduce the error with your cart because picotool doesn't yet support Pico-8 glyphs, which Pico-8 encodes as arbitrary high bytes. picotool tries to read the code region as utf-8 because I wanted the code APIs to use text strings and there were a few older carts created with external editors that had utf-8 high char sequences in comments (which Pico-8 ignores). Pico-8 doesn't actually use utf-8 in general, so this is incorrect. The correct solution is probably for picotool to deal entirely with bytestrings with regard to the Lua code region. This will be a significant upgrade.

For what it's worth, I'd expect picotool to work in Python 3.4+. Based on what other Python 3 projects appear to be doing, it looks like it'd be unrealistic to try and support older versions. I'm willing to take on more robust support for older versions if that's a common practice, though I'd imagine we'd need to run all tests in all supported versions of Python. b'...'.hex() is supported by Python 3.6, so I'm not sure yet if there's a fix needed there specifically or if the error is somewhere else.

[dansanderson]

(I also need to add support for the __label__ region, which is new in 0.10. :) )

[dagondev]

I have no problem with using newest Python, I am just assumed proper version is 3.3 as it was coresponding with time you released repo (as you didn't provide which subversion use).

Regardless my error: I forgot to state this, but while obviously my cart isn't parsable if you take it straight from pico8 as your tool tells about wrong header - converting it to the UTF8 (without BOM if I remember correctly) makes it available, as I assumed it was necessary. Weird thing was that converting it to the UTF-8 with BOM worked on Notepad++, but on Sublime Text 3 I needed to convert without BOM o.o or was it other way around - I don't remember, sorry!

[dansanderson]

I have converted all of picotool to treat text in Pico-8 carts as bytestrings instead of text strings to account for Pico-8's custom "encoding" (which is just lower ASCII with custom high characters for glyphs). I probably broke it all to hell but the tests pass. :)

picotool also now has label support. In theory, it should now be able to process the cart in the original bug report.