search

itchio / butler

🎩 Command-line itch.io helper
MIT License
744 stars 53 forks source link

Server chokes on UTF-16 --userversion-file #95

Open KatalystGames opened 7 years ago

KatalystGames commented 7 years ago

My project has randomly started getting the below error when attempting to push to itch.

*json.SyntaxError invalid character '<' looking for beginning of value C:/Multi-Runner/builds/58469115/0/itchio/butler/src/github.com/itchio/butler/push.go:29 (0x414c97) C:/Multi-Runner/builds/58469115/0/itchio/butler/src/github.com/itchio/butler/main.go:475 (0x41c2fd) C:/Go/src/runtime/proc.go:183 (0x450a0e) C:/Go/src/runtime/asm_386.s:1612 (0x47a831)

fasterthanlime commented 7 years ago

Is this still happening? I'm not able to reproduce, might have been a fluke of the API server

KatalystGames commented 7 years ago

Yes, it is still happening. I just tried to push a different project I have on Itch and it is also getting that same error now.

C:\Users\Michael>butler push D:\UnityWork\Builds\SciFiTowerDefense\Windows katalystkreations/xenostorm:windows-alpha --userversion-file D:\UnityWork\Builds\SciFiTowerDefense\version.txt *json.SyntaxError invalid character '<' looking for beginning of value C:/Multi-Runner/builds/58469115/0/itchio/butler/src/github.com/itchio/butler/push.go:29 (0x414c97) C:/Multi-Runner/builds/58469115/0/itchio/butler/src/github.com/itchio/butler/main.go:475 (0x41c2fd) C:/Go/src/runtime/proc.go:183 (0x450a0e) C:/Go/src/runtime/asm_386.s:1612 (0x47a831)

fasterthanlime commented 7 years ago

Are you using a proxy? Which country are you pushing from?

KatalystGames commented 7 years ago

No proxy and I'm from the US.

leafo commented 7 years ago

I just pushed a copy of your current windows build to your project page to test, and did not encounter any issues. Do other commands work as expected, can you run the command butler status katalystkreations/xenostorm:windows-alpha?

KatalystGames commented 7 years ago

Yup, the rest of the butler commands work fine. Just when I push.

C:\Users\Michael>butler status katalystkreations/xenostorm:windows-alpha +---------------+---------+------------------------+--------------------------------+ | CHANNEL | UPLOAD | BUILD | STATE | +---------------+---------+------------------------+--------------------------------+ | windows-alpha | #334802 | √ #12408 (from #12073) | √ 477 KiB signature, √ 2.8 KiB | | | | | patch, √ 423 MiB archive | +---------------+---------+------------------------+--------------------------------+

I've been using butler for a few weeks now without issue and this happened when I was writing an Editor script for Unity that would be a one button push to basically just open a CMD and fill in the push command with the relevant information. Now its seems all broken even when I do it manually.

fasterthanlime commented 7 years ago

@KatalystGames can you post the output of the same push command but with the -v flag added? No worries, we'll get to the bottom of this :) I'll make a build with more detailed output soon so we can see what's really going on.

KatalystGames commented 7 years ago

C:\Users\Michael>butler -v push D:\UnityWork\Builds\SciFiTowerDefense katalystkreations/xenostorm:windows-alpha --userversion-file D:\UnityWork\Builds\SciFiTowerDefense\version.txt *json.SyntaxError invalid character '<' looking for beginning of value C:/Multi-Runner/builds/58469115/0/itchio/butler/src/github.com/itchio/butler/push.go:29 (0x415c1f) C:/Multi-Runner/builds/58469115/0/itchio/butler/src/github.com/itchio/butler/main.go:475 (0x41dade) C:/Go/src/runtime/proc.go:183 (0x453fb0) C:/Go/src/runtime/asm_amd64.s:2086 (0x482fd1)

fasterthanlime commented 7 years ago

Can you update butler to the latest version (butler upgrade --head) and try again? The message should be more detailed this time.

KatalystGames commented 7 years ago

C:\Users\Michael>butler -v push D:\UnityWork\Builds\SciFiTowerDefense katalystkreations/xenostorm:windows-alpha --userversion-file D:\UnityWork\Builds\SciFiTowerDefense\version.txt *json.SyntaxError invalid character '<' looking for beginning of value C:/Multi-Runner/builds/58469115/0/itchio/butler/src/github.com/itchio/butler/vendor/github.com/itchio/go-itchio/itchio.go:189 (0x59ad42) C:/Multi-Runner/builds/58469115/0/itchio/butler/src/github.com/itchio/butler/push.go:53 (0x415edd) C:/Multi-Runner/builds/58469115/0/itchio/butler/src/github.com/itchio/butler/push.go:29 (0x415c1f) C:/Multi-Runner/builds/58469115/0/itchio/butler/src/github.com/itchio/butler/main.go:475 (0x41dade) C:/Go/src/runtime/proc.go:183 (0x453fb0) C:/Go/src/runtime/asm_amd64.s:2086 (0x482fd1)

fasterthanlime commented 7 years ago

I've made it print the value it can't parse - can you upgrade and try again? same as before.

KatalystGames commented 7 years ago

Due to the div tags it was causing havoc to post so here is a screenshot! lol https://gyazo.com/e8bdccd448ea35dfd5292b4973f5b1ef

fasterthanlime commented 7 years ago

Thanks! Can you upload the contents of D:\UnityWork\Builds\SciFiTowerDefense\version.txt as a github attachment?

KatalystGames commented 7 years ago

version.txt

I think you'll be disappointed. lol

fasterthanlime commented 7 years ago

Actually, I'm not! That's the problem :) it begins with a UTF-8 BOM (FF, FE) - I'll need to strip that in butler :)

KatalystGames commented 7 years ago

Ahh, must be the way my Unity Editor script was generating text files that differed from when I manually did it with Windows previously. :D

fasterthanlime commented 7 years ago

Oh, additionally it's actually a UTF-16 BOM (and the whole file is UTF-16). I'm afraid we don't support that yet, but I'll look into it

KatalystGames commented 7 years ago

Which formats are supported so I can make sure when it creates the file, its using one of those?

fasterthanlime commented 7 years ago

UTF-8 is the preferred standard. I hadn't seen UTF-16 in a while! (except when dealing with windows formats).

That said I'd rather modify butler so that it accepts UTF-16 as well, just so anyone runs into the same issue!

edit: linking this for further reference: http://stackoverflow.com/questions/15783830/how-to-read-utf16-text-file-to-string-in-golang

dw5 commented 1 year ago

I got this, due to having somewhere charset=us-ascii

• Pushing 702.37 MiB (7036 files, 264 dirs, 0 symlinks)
Building diff context
string field contains invalid UTF-8

Maybe charset converter should/could be built in to butler? Like if it detects, yes no pops up, and also be available as additional utility command?

github.com/itchio/wharf/wire.(*WriteContext).WriteMessage
        /home/gitlab-runner/go/pkg/mod/github.com/itchio/wharf@v0.0.0-20200618110241-8896e2c6e09b/wire/write_context.go:55
github.com/itchio/wharf/pwr.(*DiffContext).WritePatch
        /home/gitlab-runner/go/pkg/mod/github.com/itchio/wharf@v0.0.0-20200618110241-8896e2c6e09b/pwr/diff.go:64
github.com/itchio/butler/cmd/push.Do
        /home/gitlab-runner/builds/wYnefs8J/0/itchio/butler/cmd/push/push.go:362
github.com/itchio/butler/cmd/push.do
        /home/gitlab-runner/builds/wYnefs8J/0/itchio/butler/cmd/push/push.go:91
main.doMain
        /home/gitlab-runner/builds/wYnefs8J/0/itchio/butler/main.go:259
main.main
        /home/gitlab-runner/builds/wYnefs8J/0/itchio/butler/main.go:134
runtime.main
        /usr/local/go/src/runtime/proc.go:255
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1581
github.com/itchio/wharf/pwr.(*DiffContext).WritePatch
        /home/gitlab-runner/go/pkg/mod/github.com/itchio/wharf@v0.0.0-20200618110241-8896e2c6e09b/pwr/diff.go:66
github.com/itchio/butler/cmd/push.Do
        /home/gitlab-runner/builds/wYnefs8J/0/itchio/butler/cmd/push/push.go:362
github.com/itchio/butler/cmd/push.do
        /home/gitlab-runner/builds/wYnefs8J/0/itchio/butler/cmd/push/push.go:91
main.doMain
        /home/gitlab-runner/builds/wYnefs8J/0/itchio/butler/main.go:259
main.main
        /home/gitlab-runner/builds/wYnefs8J/0/itchio/butler/main.go:134
runtime.main
        /usr/local/go/src/runtime/proc.go:255
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1581
computing and writing patch
github.com/itchio/butler/cmd/push.Do
        /home/gitlab-runner/builds/wYnefs8J/0/itchio/butler/cmd/push/push.go:364
github.com/itchio/butler/cmd/push.do
        /home/gitlab-runner/builds/wYnefs8J/0/itchio/butler/cmd/push/push.go:91
main.doMain
        /home/gitlab-runner/builds/wYnefs8J/0/itchio/butler/main.go:259
main.main
        /home/gitlab-runner/builds/wYnefs8J/0/itchio/butler/main.go:134
runtime.main
        /usr/local/go/src/runtime/proc.go:255
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1581