Problem copying an mpy file to the ESP8266

[peterhinch]

I can copy the attached file to a Pyboard without problems. Copying it to the Feather Huzzah reference board with an empty filesystem produces the following outcome:

[adminpete@axolotl]: /mnt/qnap2/data/Projects/MicroPython/projects/mqtt_as
$ rsusb
Connecting to /dev/ttyUSB0 ...
Welcome to rshell. Use Control-D to exit.
/mnt/qnap2/data/Projects/MicroPython/projects/mqtt_as> ls /pyboard/
boot.py
/mnt/qnap2/data/Projects/MicroPython/projects/mqtt_as> cp mqtt_as.mpy /pyboard/
False
>raw REPL; CTRL-B to exit
>raw REPL; CTRL-B to exit
>raw REPL; CTRL-B to exit
>
MicroPython v1.8.7-797-ge5e49be on 2017-06-21; ESP module with ESP8266
Type "help()" for more information.
>>> 
raw REPL; CTRL-B to exit

I have tried erasing flash. It copies the larger .py source file without issue so I don't believe there is a filesystem problem. Oddly I've copied .mpy files to the ESP8266 before without issue. Is it a particular binary value problem? My invocation is via

alias rsusb='rshell -p /dev/ttyUSB0 --editor nano --buffer-size=30'

[adminpete@axolotl]: /mnt/qnap2/data/Projects/MicroPython/projects/mqtt_as
$ rshell -V
0.0.8

test.zip

[dhylands]

Try editing this line: https://github.com/dhylands/rshell/blob/eae3173190fc58bcb60790af26797c8acfa3d238/rshell/main.py#L1261 and changing it to

    self.has_buffer = False

(or alternatively just comment out the line that calls remote_eval with test_buffer). This will cause it to send the data is ASCII Hex data instead of binary.

I have a sneaky suspicion that the ESP8266 doesn't do transparent binary properly.

[peterhinch]

That works. I'd guessed that the fault was with the ESP raw REPL. But how to proceed: should an issue be raised against the ESP or should rshell be modified to suit?

[dhylands]

It probably makes sense to add a command line option to force ASCII buffering in rshell.

I'd like to see if we can figure out which characters are causing the problem, I'd guess newline, CR, or Control-A thru D

[peterhinch]

This gets odder. I had 'fixed' the problem by commenting out line 1261. I restored line 1261 so the code was standard. I created a binary file of 256 bytes from 0-255 inclusive, with the aim of subsequently eliminating values to determine which were problematic. It copied across successfully. I then read it into a bytes object on the ESP8266 and verified that the object contained every possible byte.

I re-checked that it will not copy the .mpy file I supplied unless line 1261 is commented out. Any ideas? I'm stumped.

[dhylands]

That makes me think that it might be a particular sequence of bytes then or something.

Probably the way to debug this is to add some prints to the recv_file_from_host routine. Print out the received bytes in hex. To see this, you can run rshell with the -d option and it will print all of the commands being sent back and forth.

You could probably even go simpler and initally just print the number of bytes received. One of the receives will probably not match with the number of bytes sent.

[mfhepp]

I have the same problem here. Copying a relatively simple program to main.py on the ESP8266 board (Adafruit Feather with esp8266-20170612-v1.9.1.bin firmware) takes forever. If you interrupt it, the file system on the ESP will be corrupted thereafter and you have to erase and reinstall the MicroPython firmware.

Copying a simple text file works.

I thought that percent signs (%) where the problem, so I removed them temporarily. With Line 1261 commented out, this again results in an endless copy process.

With the original line 1261 in place, it aborts with

"serial port /dev/cu.SLAB_USBtoUART closed"

Strange and a bit annoying ;-) I want to avoid the WebREPL loop. Are there any other options for copying files to an ESP8266 board?

[mfhepp]

BTW: Ampy from from Adafruit copies the very same file without problems.

ampy --port /dev/tty.SLAB_USBtoUART put main.py

[peterhinch]

I've never seen this with a .py file, only with that one specific .mpy file. Further with L1261 commented out, I've had no problems at all. I wonder if this may be a different issue.

[mfhepp]

I can reproduce the problem pretty well now:

Configuration:

• Adafruit Feather HUZZAH ESP8266
• Micropython binary esp8266-20170612-v1.9.1.bin
• OSX El Capitan 10.11.6

With a fresh install, I try to copy the attached main.py (renamed to .txt to be accepted by this forum) onto the uPython board.

With ampy, copying back and forth works like a charm. The retrieved file is 4 bytes longer but otherwise fine. $ ampy --port /dev/tty.SLAB_USBtoUART put main.py $ ampy --port /dev/tty.SLAB_USBtoUART get main.py main-retrieved.py

With rshell, copying a small test file with just two lines of text works.

With the attached main.py, the transfer hangs, and upon reset and inspection with ampy, it seems that zero bytes have been transferred. $ rshell --port /dev/cu.SLAB_USBtoUART cp main.py /pyboard/

When I turn off debugging, as suggested by Adafruit, the transfer ends with the message serial port /dev/cu.SLAB_USBtoUART closed and also zero bytes are transferred.

I would very much like to see this bug fixed, because ampy is pretty limited in comparison to rshell.

Martin

Attachment: main.py rename to .py

[mfhepp]

Small addition: I tried a lorem ipsum text file of the same length, and the problem stays the same. So I guess this problem is likely related to buffering or timing of the transfer on ESP8266 boards rather than character sequences. The lorem ipsum does not contain symbols except for the full stop, the rest is a-z.

[peterhinch]

I can't replicate this using rshell with L1261 commented out. Using Adafruit Feather HUZZAH ESP8266:

[adminpete@axolotl]: ~/Downloads
$ mv main.txt main.py
[adminpete@axolotl]: ~/Downloads
$ rsusb
Connecting to /dev/ttyUSB0 ...
Welcome to rshell. Use Control-D to exit. Line 1261 commented out.
/home/adminpete/Downloads> cp main.py /pyboard
/home/adminpete/Downloads> cp /pyboard/main.py main_x.py
/home/adminpete/Downloads> 
[adminpete@axolotl]: ~/Downloads
$ diff main.py main_x.py
[adminpete@axolotl]: ~/Downloads
$ 

Note my alias for invoking rshell. Are you using --buffer-size=30?

alias rsusb='rshell -p /dev/ttyUSB0 --editor nano --buffer-size=30'

[mfhepp]

Dear Peter: Thanks - the buffer size option does the trick!

alias rsusb='rshell -p /dev/ttyUSB0 --editor nano --buffer-size=30'

rShell now works like a charm on the Adafruit Feather HUZZAH.

[mfhepp]

Addition: It works now even with line 1261 active, i.e. with the standard version:

self.has_buffer = self.remote_eval(test_buffer)

[kevinkk525]

I'm on rshell 0.0.16 from pip3 and it does not work for me. I get a weird output in repl after trying to copy a *.mpy file which always fails.

[dhylands]

Make sure you're using smallish buffer size (as mentioned above) and -a when using rshell with an ESP8266 or ESP32

[dhylands]

It looks like the UART buffer may only be 15 characters, so you might need to use --buffer-size=14 https://github.com/micropython/micropython/issues/4402#issuecomment-452675333

[kevinkk525]

Thanks for your quick answers. Sadly that does not help either. It creates a 0 Bytes file and the error message "timed out error in transfer to remote". Entering the repl after copying shows this:

repl_serial_to_stdout dev = <rshell.main.DeviceSerial object at 0x7f8896160320>
alse
>
MicroPython v1.9.4-771-gb33f108cd-dirty on 2019-01-07; ESP module with ESP8266
Type "help()" for more information.
>>> :6
Traceback (most recent call last):
  File "<stdin>", line 1
SyntaxError: invalid syntax
>>>

(I can however copy .mpy files with mpfshell, if that information helps in any way.) Hardware: esp8266 nodemcu

[peterhinch]

Is this occurring with my client.mpy? If so, I've copied numerous versions using

alias rsusb='rshell -p /dev/ttyUSB0 --editor nano --buffer-size=30 -a'

[kevinkk525]

Interestingly it occurs with some extensions to your client.mpy file. Your answer however gave me the clue. I thought using this was the right way: cp -a client.mpy /pyboard/ But apparently I have to start rshell with the "-a" option instead, which I have not seen before and misinterpreted the command of Dave to using "-a" only in the cp command. Now it works correctly. Thank you both for the help!

[dhylands]

What the -a option to rshell does is cause all file transfers to occur using pure ascii (so the binary file data gets converted into ascii hex). It turns out that some serial drivers on some ports will do translations on some binary characters which messes up the file transfer.

[peterhinch]

Yes. We never did manage to figure out what the ESP8266 was doing but -a does fix it.

[dhylands]

One potential improvement would be to use base64 encoding rather than hex. This would improve the transfer speed.

[peterhinch]

Yes, others have commented on speed but I have no complaints.

[sistemicorp]

I see this issue on STM32F405 PyBoard, rshell version 0.0.26. MicroPython version v1.11-182-g7c15e50eb on 2019-07-30; PYBv1.1 with STM32F405RG.

Based on the discussion above I tried several things, 1) modifying the file, trimming off lines trying to see if there was a sequence of chars in the file that were causing the problem, 2) copying the files in different order, 3) resetting the filesystem and trying again, 4) copying files from largest to smallest size, ...

Nothing I tried resolved the issue. I have had this issue come and go int he past, but sometimes it just never goes away.

I was able to copy the files with ampy.

[dhylands]

Would you be able to send me the files? zip them up and send them to dhylands@gmail.com

I've never had any issues copying files to the pyboard, so I'm really interested in figuring out what the problem might be.

[gavinB-orange]

Seeing this problem when copying files to a Raspberry Pi Pico. Similarly sized binary files are OK, but some python code file hang. The workaround appears to be to increase the buffer size.

rshell --buffer-size=2048 -p /dev/ttyACM0

works fine, but change that 2048 to the default 512 and it hangs.

[davehylands]

There appears to be some USB issues with the pico. I have a pico branch in rshell that has been working for me so far.

See here: https://forum.micropython.org/viewtopic.php?f=15&t=9660&p=54175#p54175 for some more details.