uPyLoader doesn't work....

[profra]

I did git pull now and I am without MY uPyLoader, which I am used to use every day! ... see picture... what to do?

[BetaRavener]

You can revert to older version or wait for a day until all is tested and fixed (actually more because I don't have ESP with me, so I can't test things that require connection). The issue is due to changes in project structure so that it's easier to maintain, but unfortunately few things were not refactored correctly.

[profra]

How can I revert to older version... I don't have any backup 👎

[BetaRavener]

Git itself is tracking histories ;) Something like this should work git checkout 43d0e05 ., although I'm not sure if I got the ID right. You need to get on this commit 43d0e050.

[BetaRavener]

Alternatively you could use built release (or source codes from release). Only missing feature would be updating timestamps, that were added later. Next time I will make dev branch so that main stays intact, sorry for inconvenience.

[profra]

Success, many thanks.... works OK.

[BetaRavener]

Could you try new commit if everything is alright? I have checked all functions that can be used without connection, but there might be some leftover issue. In case it crashes somewhere, just send traceback screenshot like you did last time. Thanks!

[profra]

I'm at work now and don't have any 8266 board with... as soon as I am at home I will give you feedback.

[profra]

The first message... git clone... OK, python3 main.py... OK... programme started, connect... OK, list of files on Remote... OK, File-Navigate... OK, file transfer to Remote... OK, file transfer to Local... OK, Compile does nothing... that is all for now... I do more tests later, I will give you feedback... I will be back in two hours. Please, explain to me the philosophy of (right mouse-click) commands 'Transfer' (does nothing) and 'Set as destination' (does nothing)...

[profra]

... compile works... after correct Settings in Options...

[profra]

It seems that all functions (which I am able to check) are working OK... BUT I have some ideas which I want to share with you... a/ I have been using esptool v2.1 (for NIX and WIN environment) for a long time and that's why the function 'File-Flash firmware' doesn't have sense anymore for me... and I guess also for others... IMHO would be better to call some batch files (X.sh for NIX or Y.bat for WIN) which the user can 'Pick' instead of 'Python2 path'... and these batch files could have only one variable 'Firmware file' ... do it the same way as in 'Option-Settings-Editor' b/ It would be nice if uPyLoader remembered position and size of all windows from the last session... it is really annoying to manipulate the windows to my working position and size after every new start c/ ... also to open the same 'Local directory' after the new start would be awesome :-) d/ I open the files for editing with my favorite external editor... nice feature... would it be possible also for files from 'Remote'? .... and after editing to save back to 'Remote'? e/ what about the function 'Synchronize'? ... the newest files would be copied between 'Local' and 'Remote' f/ The current GUI is a bit confusing... e.g. 'MCU name' has too long field and 'PC path' too short... I allowed myself to make a small picture with IMHO a little clearer GUI

Please take my comments only as the inspiration and in no way as the criticism. uPyLoader is very handy tool with a lot of assets, it is 'my right hand' when working with MP or CP.

[BetaRavener]

Hi, thanks for testing it for me. Well, the right-click context menu is currently just placeholder, the development is happening in dev-transfer branch so it won't break things again. They will be doing something eventually :)

Regarding the ideas: a) I'm missing how is esptool 2.1 related to compilation being obsolete. Oh I completely misread there, my mind was in other places :) Still, I'm little confused - I'm also using esptool to flash firmware, the dialog is just a nice GUI interface to it. The problem is that esptool was only for python2 at the time of implementation. I checked their pages now and it seems python 3 is now supported too, so this may be obsolete. Is this what you are trying to tell me? b) It does this. It's supposed to do this. First problem I noticed is that it's no longer able to create new config.json if not present. Probably something to do with patch for hidden config files. Second issue which you may be referring to is the ratio between size of Local and MCU column? There's problem that Qt doesn't save that info, so I will have to make a workaround. But with existing config file, position and size of window is saved and restored (just tested). c) Also related to missing config file, seems like I'll have to fix it right away. d) That's hard one, as IDE has no idea of remote filesystem. Idea: uPyLoader would have to make file in temp folder, open that in IDE and check for changes on that file every so often. Not very resource friendly (when to stop tracking file?) and the filename in IDE would be garbage (e.g. tmpxp8sd0x9.py) as to prevent conflicts. But maybe if it was just one file at a time, it might be doable. e) Not implemented yet but considered in new design. @jdeltoft proposed to compute hashes and check if differs, which seems to be the best way for now. You still have to click button for transfer - not because I want to annoy you, but it would wear down ESP's flash too fast if every (even intermediary) change would cause transfer. f) New design completely removes those 2 text fields and will probably move some of the buttons to context menu (right-click), e.g. Remove, Rename,.. The interface should then be a lot cleaner.

[BetaRavener]

@profra b) partially and c) fixed by dd778d8. Please pull new version.

[profra]

Hi, it was a long day for me... but now I am back to the PC... ad a/ yes, exactly, I wanted to say that esptool v2.1 is the newest version and AFAIK it is possible to install it in both Pythons (2.7 or 3.x)... works also for both esp8266 and esp32... a lot of parameters... this whole thema of flashing with optional batch files would solve the problem forever... the user will only pick the right .sh/bat for erasing, flashing and binary file (only one for esp8266)... it is more complicated for esp32 (more memory segments) but it is also simply solvable... field for binary file stays empty and the user will be responsible for including the right parameters into batch file.... there are a lot of others ways how to solve it.... but leave the output window to see the progress of activity erasing/flashing ad b,c/ I'm glad you agree with my opinion... opening the new session should start there where the last session ended... ad d/ I have been using also rshell which is really a very handy tool but only command line... look at that https://github.com/dhylands/rshell ... good inspiration and may help with your implementation ad e/ rsync is also included to rshell... nice feature ad f/ to tune the GUI always makes sense, it gets a user community... the easier the better... but it's work on all the time and we will have still time to discuss it...

I am going to pull new version now and I will give you feedback

[profra]

I did some tests and results are: ad b,c/ the last directory for Local is remembered... good, the uPyLoader's main window is at the same position and size as during the last session... good, the columns Local and Remote don't start with the same width as were adjusted during the last session... bad, I can modify the width of columns Size, Type, Date_Modified but not the column Name... bad, the window of Terminal (View-Terminal) holds the position and size from the last session... good

New info: I did some tests with MP on esp32... from here https://github.com/loboris/MicroPython_ESP32_psRAM_LoBo... board is connected, the terminal is working, the process of 'Init transfer files' failed... see picture It would be nice if uPyLoader worked also with MP for esp32... Do you have any plans in this direction?

[BetaRavener]

Regarding the ESP32, someone made an unfortunate decision to throw error if something tries to communicate on the same UART as REPL. This makes any custom communication over the same channel impossible (you would have to connect another UART to USB device to ESP32 == new COM port). There is not really anything I can do - simple REPL is unreliable for data transfer and I can't use anything else. Shame on the developers. Your only option is to try WiFi communication which doesn't require scripts (but do require Websockets implementation, which when I last checked was still in progress)

[profra]

But rshell works with MP for esp32... it is using also serial channel... where is the difference in communication between rshell and uPyLoader?

[profra]

FYI... I asked Damien on the forum if can help us somehow.

[BetaRavener]

Wasn't necessary. From rshell page: "This is a simple shell which runs on the host and uses MicroPython's raw-REPL to send python snippet...". So yes, it's working because it's only using REPL. The purpose of transfer scripts is solely to bypass REPL to provide faster and more reliable communication. I can't remember right now if it's even possible to have duplex communication over REPL (is it possible to read from REPL inside script?).

[profra]

Please look at the answer from @Roberthh on the forum... it is clear he knows... and describes some solutions which could be helpful. https://forum.micropython.org/viewtopic.php?f=18&t=4011&sid=ed5b479740d20373e1c36c3a57f3b148

[BetaRavener]

Well, I rushed into an answer there, the stuff about system buffers is interesting. The blocking is behavior is unfortunately deal-breaker, because any lost character would cause script to block forever... But maybe I can come up with some solution.

Also, I didn't know that you can disable interrupts, so the transfer protocol uses encoding for all chars with value less than 10 I think.. It requires more processing but on the other side leaves interrupt enabled in case there was an error.

[profra]

FYI... https://forum.micropython.org/viewtopic.php?f=18&t=4011&p=23039#p23039

[BetaRavener]

Interesting find. Anyway, I'm going to try and write new scripts that will use just sys buffers, as most users isn't probably gonna try flash some other firmware.

[BetaRavener]

@profra Please check, if the new script implementation is working with ESP32. Use git checkout dev-scripts and pull the new files. I still don't have ESP device around so this is just best effort implementation. I will test it more on weekend (but only with ESP8266).

[profra]

I get this... fp@fp-VC62B uPyLoader $ git checkout dev-scripts error: pathspec 'dev-scripts' did not match any file(s) known to git. fp@fp-VC62B uPyLoader $

[profra]

I did usuall git pull... is it OK? fp@fp-VC62B uPyLoader $ git pull remote: Counting objects: 16, done. remote: Compressing objects: 100% (5/5), done. remote: Total 16 (delta 11), reused 16 (delta 11), pack-reused 0 Unpacking objects: 100% (16/16), done. From https://github.com/BetaRavener/uPyLoader

• [new branch] dev-scripts -> origin/dev-scripts f49f4c6..3f4d21d dev-transfer -> origin/dev-transfer Already up-to-date.

[profra]

... working with WiPy 2 binary... but it says nothing... was OK also before... I will try it with the binary from Damien... failed during "Init transfer files" ... I have to go now... will be back after 8 PM

[BetaRavener]

Try the branch command now after pulling, or use this SO post to get on the right branch. Sorry for the confusion. You should see star (or some indicator) next to dev-scripts branch after using git branch command, telling you that it's active one. If successful, the MCU scripts should have #V2 comment on top.

If you already are on the right branch and Init transfer files failed, then I made mistake somewhere and will have to debug :)

[profra]

... a bit later, sorry.... you have here the hardcopy of my screen... you can analyze it... a/ Lolin32 with Wipy 2 binaries... File transfer has stopped, see the error... uPL is not responding -> Force Quit b/ Lolin32 with MP from Damien... the same result

fp@fp-VC62B uPyLoader $ git branch
* master
fp@fp-VC62B uPyLoader $ git checkout dev-scripts
Branch dev-scripts set up to track remote branch dev-scripts from origin.
Switched to a new branch 'dev-scripts'
fp@fp-VC62B uPyLoader $ git pull
Already up-to-date.
fp@fp-VC62B uPyLoader $ git checkout dev-scripts
Already on 'dev-scripts'
Your branch is up-to-date with 'origin/dev-scripts'.
fp@fp-VC62B uPyLoader $ python3 main.py
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/home/fp/opt/github/uPyLoader/src/connection/serial_connection.py", line 223, in _upload_transfer_files_job
    res = self.send_file(data.encode('utf-8'), transfer)
  File "/home/fp/opt/github/uPyLoader/src/connection/serial_connection.py", line 262, in send_file
    ack = self.read_timeout(2)
  File "/home/fp/opt/github/uPyLoader/src/connection/serial_connection.py", line 94, in read_timeout
    data.append(rec)
TypeError: an integer is required

[profra]

... the same results with 8266...

[BetaRavener]

Doh fixing the code that wasn't broken is always bad idea :) I have just pushed the correct code, so you can try again if you feel like testing stupid code tonight.

[profra]

started but stopped forever ... no error, see picture

[profra]

with 8266 other behaviours... file transfer started but than many File Transfer failed... see picture

[BetaRavener]

Actually, that's pretty cool 😮

No seriously, this will require some proper testing and debugging first. You could try that yourself (the related code is all in serial_connection.py) or just watch a movie tonight and wait till Friday evening.

[profra]

if you have time and taste for work I am available until midnight :-)

[profra]

isn't it better to test it only with 8266 at the beginning?

[profra]

serial_communication.py OR serial_connection.py?

[BetaRavener]

The second (edited original post)

It is better but I'm in different city and have no 8266 with me. Also, I have no idea why the thread would spawn so many errors, so I have no idea what to fix. You could try delete both scripts on the MCU first, then open terminal and see if there's any interesting output. I would just set a breakpoint in the responsible function and step through the code to see what's happening, but I don't know if you have IDE for this.

[profra]

It was an interesting idea... deleteupload.py nad download.py... transfer started, went until end of the blue field and stopped with many windows with Error...folded on itself... if I click 14x on OK the whole process is ended successfully and files upload.py and download.py are uploaded!!! What does it mean?

[profra]

14x ... tested more times I have no IDE

[profra]

I send you output from terminal... begins after start of 'Init transfer files' and ends after 14th click on OK...

>>> 
>>> 
paste mode; Ctrl-C to cancel, Ctrl-D to finish
=== # V2
=== import sys
=== import time
=== 
=== 
=== def _read_timeout(cnt, timeout_ms=1000):
===     s_time = time.ticks_ms()
===     data = sys.stdin.buffer.read(cnt)
===     if time.ticks_diff(time.ticks_ms(), s_time) > timeout_ms or len(data) != cnt:
===         return None
===     return data
=== 
=== 
=== def _upload():
===     suc = False
===     with open("__upload.py", "wb") as f:
===         while True:
===             d = _read_timeout(2)
===             if not d or d[0] != ord("#"):
===                 x = sys.stdout.buffer.write(b"#2")
===                 break
===             cnt = d[1] & 0x7F
===             if cnt == 0:
===                 suc = True
===                 break
===             d = _read_timeout(cnt)
===             if d:
===                 esc = False
===                 for c in d:
===                     if c == 0:
===                         esc = True
===                         continue
===                     x = f.write(bytes([c & 0x0F if esc else c]))
===                     esc = False
===                 x = sys.stdout.buffer.write(b"#1")
===             else:
===                 x = sys.stdout.buffer.write(b"#3")
===                 break
===     x = sys.stdout.buffer.write(b"#0" if suc else b"#4")
=== 
=== 
=== _upload()
=== 
=== 
>> 
PYB: soft reboot
#22 ets_task(40100130, 3, 3fff837c, 4)
connecting to last APTrue
WebREPL is not configured, run 'import webrepl_setup'
mem_free: 27488 mem_alloc: 8480
MicroPython v1.9.3-6-g1742ab2 on 2017-11-03; ESP module with ESP8266
Type "help()" for more information.
>>> 
paste mode; Ctrl-C to cancel, Ctrl-D to finish
=== file_name="__download.py"
=== with open("__upload.py") as f:
===     exec(f.read(), globals())
=== 
>> 
PYB: soft reboot
#23 ets_task(40100130, 3, 3fff837c, 4)
connecting to last APTrue
WebREPL is not configured, run 'import webrepl_setup'
mem_free: 27376 mem_alloc: 8592
MicroPython v1.9.3-6-g1742ab2 on 2017-11-03; ESP module with ESP8266
Type "help()" for more information.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 

[BetaRavener]

Thanks. Sorry I've not been around. The fact that removing scripts helps is not good, because that would indicate some conflict which shouldn't be there. Another issue - I don't like PYB: soft reboot line after the upload file gets pasted. It seems like I'm crashing micropython somehow. Actually, the communication probably failed and I'm sending '\4' character which I thought would break the read (it ends paste mode in micropython so I thought it may be equivalent of Ctrl-Z on Windows). Apparently it causes soft reboot. Thanks for the effort but I really need to debug this when I have ESP with me.

[profra]

@BetaRavener Hi, I wrote an email to you on hotmail, look at it...

[BetaRavener]

@profra Hi, I've just fixed the error in dev-scripts branch. The problem was actually just one character - changing 3 to 2. I changed protocol but forgot to change how many characters are read and so it always read some additional garbage that never compared as successful transfer. The rest of the changes in commit just prevents those 14 error windows appearing if there was any other error :)

Please, try it with ESP32 as it was the main goal.

[profra]

@BetaRavener Hi, I was a whole weekend at our cottage, I had a lot of work and I was tired... I didn't turn on the PC at all. I have started tests now... first with esp8266 and I have found some bug?... if you tag file(s) and start the 'Transfer' (upload) the file(s) is (are) transferred correctly but info-window File Transfer (Saving File) is not closed... you have to do it manually. Otherwise, it seems it is working. I have started also tests with esp32 but there I have some problems with FW for Wipy_2, the latest is not correctly working... there is some bug, info from the forum... I have to downgrade the FW... more info later.

[BetaRavener]

I have just fixed the dialog issue in new commit.

[profra]

OK, I do git pull ... BUT good message... I have installed MP from Damien... here http://micropython.org/download#esp32 ... it is WORKING!!! I do still some tests with the new version and let you know....

[profra]

Git pull done...I can not find any mistake ... CONGRATULATIONS!!! I'll try everything also with Wipy 2 FW, but there I'll have some work with the downgrade of FW.

[profra]

The downgrade was done... uPL doesn't work for Wipy 2 FW now... the opposite situation than at the beginning :-) ... I'm sending you a screen scan and also the whole output from the terminal.... I think the cause is in the modified python language from PYCOM... the module '__upload.py' was uploaded... the analysis is up to you...

ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
flash read err, 1000
Falling back to built-in command interpreter.
OK
>ets Jun  8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff9010,len:12
ho 0 tail 12 room 4
load:0x3fff9020,len:388
load:0x40078000,len:11584
load:0x4009fb00,len:848
entry 0x4009fc9c
MicroPython v1.8.6-796-g489fafa0 on 2017-10-15; LoPy with ESP32
Type "help()" for more information.
>>> 
paste mode; Ctrl-C to cancel, Ctrl-D to finish
=== #V2
=== import sys
=== import time
=== 
=== 
=== def _read_timeout(cnt, timeout_ms=2000):
===     s_time = time.ticks_ms()
===     data = sys.stdin.buffer.read(cnt)
===     if time.ticks_diff(time.ticks_ms(), s_time) > timeout_ms or len(data) != cnt:
===         return None
===     return data
=== 
=== 
=== def _upload():
===     suc = False
===     with open("__upload.py", "wb") as f:
===         while True:
===             d = _read_timeout(2)
===             if not d or d[0] != ord("#"):
===                 x = sys.stdout.buffer.write(b"#2")
===                 break
===             cnt = d[1] & 0x7F
===             if cnt == 0:
===                 suc = True
===                 break
===             d = _read_timeout(cnt)
===             if d:
===                 esc = False
===                 for c in d:
===                     if c == 0:
===                         esc = True
===                         continue
===                     x = f.write(bytes([c & 0x0F if esc else c]))
===                     esc = False
===                 x = sys.stdout.buffer.write(b"#1")
===             else:
===                 x = sys.stdout.buffer.write(b"#3")
===                 break
===     x = sys.stdout.buffer.write(b"#0" if suc else b"#4")
=== 
=== 
=== _upload()
=== 
=== 
aceback (most recent call last):
  File "<stdin>", line 42, in <module>
  File "<stdin>", line 38, in _upload
  File "<stdin>", line 18, in _upload
  File "<stdin>", line 8, in _read_timeout
AttributeError: 'FileIO' object has no attribute 'buffer'
>>> ##V2import sysimport timedef _read_timeout(cnt, timeout_ms=2
>>> 
>>> 
>>> 

[BetaRavener]

You're right, now the Wipy has problem that there is no buffer on stdin and stdout. However, I now wonder why we need to use that buffer, tests show me that sys.stdin object itself has read method, so we could use that, and it's likely that Wipy will also support that. Hopefully there won't be any difference in behavior. I'll make changes tomorrow and let you test it :)

[profra]

Well, I look forward to it ... it would be great if it worked with all MP versions ... I'm a bit of a dreamer 👍