Sync of binary files fails + solution

[abrowdi]

While playing around with the web server "microWebSrv.py" I noticed that sync of binary files (jpg, png, ico, etc.) is broken. The files are larger and thus corrupted when served through the web server.

It seems that the files are converted during the transfer with pymakr plugin for atom/vsc. The files are read as utf8 which causes the error.

After changing

var contents = fs.readFileSync(this.py_folder + filename,'utf8') in sync.js to var contents = fs.readFileSync(this.py_folder + filename,'binary') and commenting //contents = utf8.encode(contents) in monitor.js

the sync of binary files works for me. I also tried to sync a python file with utf8 characters in it: print('123abc©') and sync & execution of such a file worked as expected. Though I am not sure if it breaks something for others with maybe different system encodings.

You can reproduce the error by syncing the attached file (content: 256 bytes, 0x00-0xFF) and calling stat('00-FF.txt'), the file got larger. 00-FF.txt

[RalphHogenbirk]

Hey @abrowdi, that's a nice usecase. Pymakr wasn't really ment or tested for uploading of anything else than code and text. So the utf8 encoding has been there from the beginning. I'm working on a complete refactor of the upload code now (upload-refactor branch). It's based on mpfshell, which also assumes utf8 encoding of whatever you upload. Micropython boards usually don't have a lot of memory or are used for webservers or anything graphic. But our latest Pycom boards can definitely handle it, so maybe it's time to start supporting it for Pymakr.

I'll give your code suggestion a try in my new branch, and will see if that works consistently for utf8 encoded files as well. I think it should not break anything. I was momentarily questioning is how large of a file you want to upload using Pymakr, because it's all running through the REPL using micropython, which is fine for small files but large files will take a while (FTP is faster). But if you specifically want to do it because it's easier with Pymakr and accept waiting a bit longer for it to upload, it should be fine. Also the image files will probably be a one-time upload and not change again after that.

[abrowdi]

Hi @RalphHogenbirk, thanks for considering the changes! I know that transfering the webserver files via ftp is much quicker but the workflow needs an extra step. If one changes the *.py files and the html/js/css/jpg/png/... files and wants to sync, she needs to first sync via pymakr and then switch to the ftp client and sync the other files. It would be great to just do everything in pymakr. In my case all web server filey are actually binary files because to save space on the micro and during transmission, I gzip every static file and then serve it with the HTTP header "Content-encoding: gzip".

[RalphHogenbirk]

@abrowdi Another little update on this. I worked on it now and pushed a version to the develop branch that supports the upload of any file type (tested with png/mpy/css/js/html files). You still have to add these file types to the 'Sync file types' option, but then it works. You can download it from the develop branch and manually install that version (theres an instruction in the readme on how to do it).

Let me know if it's something you still need and want to test out.

In the end I removed the encoding param completely from the readFileSync() call. And instead of converting it to a string and then converting it to hex, I generate chunks of base64 strings directly from the file buffer. This way the encoding is never touched during the process, and nothing is corrupted. Next up is fixing the download feature in the same manner.