New Release - OSError : ENOENT - trying to debug - not sure why yet

tongclement commented 3 years ago

Error in full (Using the new version:) (Using the install update if available method)

Traceback (most recent call last):
  File "boot.py", line 41, in <module>
  File "boot.py", line 33, in ota_update_checker
  File "ota_updater.py", line 78, in install_update_if_available
  File "ota_updater.py", line 103, in _check_for_new_version
  File "ota_updater.py", line 118, in get_version
OSError: [Errno 2] ENOENT

Will add more when I am able to solve it.

rdehuyss commented 3 years ago

I found out the hard way that different micropython versions handle files totally different.

What Micropython version do you use?

suysh commented 3 years ago

My version (downloaded recently): MicroPython v1.13 on 2020-09-11; ESP module with ESP8266

rdehuyss commented 3 years ago

I'll see somewhere next week if I can test in that version but due to my private situation, it's difficult to put in a lot of time for the moment.

suysh commented 3 years ago

Do you know which version you use? I can also try with older MicroPython versions.

tongclement commented 3 years ago

Do you know which version you use? I can also try with older MicroPython versions.

+1 thanks! - I am happy to downgrade for now if required. Thank you for helping to maintain this.

tongclement commented 3 years ago

My version (downloaded recently): MicroPython v1.13 on 2020-09-11; ESP module with ESP8266

Btw, are you experiencing the same issue? Thanks!

tongclement commented 3 years ago

My version is the latest one: v1.13 - the latest stable version on esp32 (Normal version without ram)

Specifically esp32-idf3-20200902-v1.13.bin

Thanks very much!

rdehuyss commented 3 years ago

Do you know which version you use? I can also try with older MicroPython versions.

I'm using a special version for the M5Stack: https://github.com/m5stack/M5Stack_MicroPython

yeyeto2788 commented 3 years ago

Seems like errors are defined here and are coming directly from here.

Could be a bit difficult to understand some of them.

But looking at the documentation here it says the following:

CPython raises a socket.gaierror exception (OSError subclass) in case of error in this function. MicroPython doesn’t have socket.gaierror and raises OSError directly. Note that error numbers of getaddrinfo() form a separate namespace and may not match error numbers from the uerrno module. To distinguish getaddrinfo() errors, they are represented by negative numbers, whereas standard system errors are positive numbers (error numbers are accessible using e.args[0] property from an exception object). The use of negative values is a provisional detail which may change in the future.

Hope it helps!

tongclement commented 3 years ago

Seems like errors are defined here and are coming directly from here.

Could be a bit difficult to understand some of them.

But looking at the documentation here it says the following:

CPython raises a socket.gaierror exception (OSError subclass) in case of error in this function. MicroPython doesn’t have socket.gaierror and raises OSError directly. Note that error numbers of getaddrinfo() form a separate namespace and may not match error numbers from the uerrno module. To distinguish getaddrinfo() errors, they are represented by negative numbers, whereas standard system errors are positive numbers (error numbers are accessible using e.args[0] property from an exception object). The use of negative values is a provisional detail which may change in the future.

Hope it helps!

@yeyeto2788 Thanks for the info! So I am getting #define MP_ENOENT (2) // No such file or directory right now from line 118: if version_file_name in os.listdir(directory):

So does this mean that the new version hasn't been downloaded? So its back to the drawing board regarding how it detects new versions (?) I listed the directory using os.listdir and in the list, I can see the folder 'main' but not 'next'. Is there anything that I should be aware of? I am happy to help with the debugging and I can provide any info required. Thanks for anyone's help in advance.

Note: I live in a UTC+8 timezone so I might not be available during Belgium daytime. Thanks for anyone's help in advance.

tongclement commented 3 years ago

Btw, I was wandering whether these tags: github_src_dir='src', main_dir='app' matter when calling the updater. What do they mean? Thanks very much!

yeyeto2788 commented 3 years ago

@FalconBFR If you're using upstream code the line 118 it's actually when it tries to open up a file. Add a line with: print(directory + '/' + version_file_name) just before line 118 and compare it with the files listed on the filesystem you have on the board.

rdehuyss commented 3 years ago

Ok, I successfully tested this now also against esp32-idf4-20200902-v1.13.bin. See the logs below and please also compare the different releases on GitHub (https://github.com/rdehuyss/micropython-ota-updater/compare/0.5.2...0.5.3). I had to do some small changes compared to the MicroPython version running on M5Stack (https://github.com/m5stack/M5Stack_MicroPython) but I'm 100% sure that it works (at least with the esp32-idf4-20200902-v1.13.bin version).

If you have troubles, I'm afraid I cannot help anymore due to family constraints.

Logs (without startup and WIFI logging):

Uploading project (main folder)...
Not safe booting, disabled in settings

Uploading to /...
Reading file status
Failed to read project status, uploading all files
Creating dir app
[1/6] Writing file __init__.py (0kb)
[2/6] Writing file app/__init__.py (0kb)
[3/6] Writing file app/ota_updater.py (15kb)
[4/6] Writing file app/secrets.py (0kb)
[5/6] Writing file app/start.py (0kb)
[6/6] Writing file main.py (1kb)
Upload done, resetting board...
OKets Jun  8 2016 00:22:57

connecting to network...
network config: ('192.168.1.67', '255.255.255.0', '192.168.1.1', '192.168.1.1')
Checking version... 
        Current version:  0.0
        Latest version:  0.5.2
Updating to version 0.5.2...
Downloading version 0.5.2
        Downloading:  /rdehuyss/micropython-ota-updater/0.5.2/app/__init__.py to next/__init__.py
        Downloading:  /rdehuyss/micropython-ota-updater/0.5.2/app/ota_updater.py to next/ota_updater.py
        Downloading:  /rdehuyss/micropython-ota-updater/0.5.2/app/start.py to next/start.py
Version 0.5.2 downloaded to next
Copying secrets file from app/secrets.py to next/secrets.py
Copied secrets file from app/secrets.py to next/secrets.py
Deleting old version at app ...
Deleted old version at app ...
Installing new version at app ...
Update installed, please reboot now

connecting to network...
network config: ('192.168.1.67', '255.255.255.0', '192.168.1.1', '192.168.1.1')
Checking version... 
        Current version:  0.5.2
        Latest version:  0.5.2
Version 1 installed using USB
MicroPython v1.13 on 2020-09-02; ESP32 module with ESP32
Type "help()" for more information.
>>> import machine
>>> machine.reset()

connecting to network...
network config: ('192.168.1.67', '255.255.255.0', '192.168.1.1', '192.168.1.1')
Checking version... 
        Current version:  0.5.2
        Latest version:  0.5.3
Updating to version 0.5.3...
Downloading version 0.5.3
        Downloading:  /rdehuyss/micropython-ota-updater/0.5.3/app/__init__.py to next/__init__.py
        Downloading:  /rdehuyss/micropython-ota-updater/0.5.3/app/ota_updater.py to next/ota_updater.py
        Downloading:  /rdehuyss/micropython-ota-updater/0.5.3/app/start.py to next/start.py
Version 0.5.3 downloaded to next
Copying secrets file from app/secrets.py to next/secrets.py
Copied secrets file from app/secrets.py to next/secrets.py
Deleting old version at app ...
Deleted old version at app ...
Installing new version at app ...
Update installed, please reboot now

connecting to network...
network config: ('192.168.1.67', '255.255.255.0', '192.168.1.1', '192.168.1.1')
Checking version... 
        Current version:  0.5.3
        Latest version:  0.5.3
Version 2 installed using OTA Update
MicroPython v1.13 on 2020-09-02; ESP32 module with ESP32
Type "help()" for more information.

rdehuyss commented 3 years ago

I've also enabled GitHub discussions (https://github.com/rdehuyss/micropython-ota-updater/discussions) - please share knowledge, tips en trics there.

rdehuyss commented 3 years ago

Do not forget to also erase flash when deploying the latest version. There may be some leftovers which can cause problems.

This is off-course only needed the first time.

tongclement commented 3 years ago

Do not forget to also erase flash when deploying the latest version. There may be some leftovers which can cause problems.

This is off-course only needed the first time.

Thanks for the reminder. We wish you and your family a good Christmas!

tongclement commented 3 years ago

Currently trying the new version - Not related to this project but I am currently experiencing OSError28 - Device out of space when using a Replace.py script despite having just flashed the device. I will clean up my code base and try again. Might take some time.

tongclement commented 3 years ago

print(directory + '/' + version_file_name)

Thanks for the suggestion! @yeyeto2788 After adding your line to (its now line 115), the output is app/.version

What does this mean? Does it mean that it is functioning fine? Also, I was wandering whether it had anything to do with the lines: github_src_dir='src', main_dir='app' when calling the updater as recommended by the readme: updater = OTAUpdater('GitHub link ', headers={'Authorization': 'token {}'.format(token)}, main_dir='app', secrets_file="wifi.dat")

Once again, thanks so much for your help and your expertise! Without your clear explanation, it would be extremely difficult to debug this!

rdehuyss commented 3 years ago

Hi @FalconBFR,

Regarding your error OSError: [Errno 2] ENOENT

Can you please copy the files exactly (do not touch them). Your current stack trace does not match the code (any version) so it's difficult to see what is going on. I tested on the exact same firmware and it works fine with me.

To answer your questions:

main_dir: the folder where are your code lives that you want to update using micropython-ota-updater.
github_src_dir : this can be used when you create unit tests for your micropython code. You off-course don't want the tests folder but only everything which is inside src. An example where this is used is https://github.com/rdehuyss/chicken-shed-mgr.

A comple example of the git layout (italic words are folders):

github repo
- src: contains all code related to your project and running on the ESP
  - app: all your code which should be deployed to your ESP using micropython-ota-updater.
  - code that will not be touched by micropython-ota-updater.
- tests: code that you don't want on the ESP

In the scenario above, you need to use: github_src_dir='src', main_dir='app'

tongclement commented 3 years ago

@rdehuyss Thanks for the help! I will try this new directory format and report whether it is successful!

rdehuyss commented 3 years ago

@FalconBFR any news on this?

tongclement commented 3 years ago

@FalconBFR any news on this?

I think I understand the issue now and I am trying to fix it. However, just to confirm, should all the files on the board be in the root dir? Thanks!

As a side note, what tools do you guys use to upload files to your boards? I am using a random script called (replace.py) and it is terribly slow. Any suggestions are welcome.

mastercba commented 3 years ago

Try Thonny IDE for EsP32 and ESP8266. Other option can be PyCharm for micropython. Best!

On Fri, Jan 22, 2021 at 8:57 PM Clement Tong notifications@github.com wrote:

@FalconBFR https://github.com/FalconBFR any news on this?

I think I understand the issue now and I am trying to fix it. However, just to confirm, should all the files on the board be in the root dir? Thanks!

As a side note, what tools do you guys use to upload files to your boards? I am using a random script called (replace.py) and it is terribly slow. Any suggestions are welcome.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/rdehuyss/micropython-ota-updater/issues/11#issuecomment-765815285, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPOPO5J3G36PP3TPWR7VBDS3IUHZANCNFSM4VGX53AQ .

rdehuyss commented 3 years ago

However, just to confirm, should all the files on the board be in the root dir?

No. Have a look at https://github.com/rdehuyss/chicken-shed-mgr. There you can see a possible directory layout.

What tools do you guys use to upload files to your boards?

I use VSCode with PyMakr plugin.

tongclement commented 3 years ago

However, just to confirm, should all the files on the board be in the root dir?

No. Have a look at https://github.com/rdehuyss/chicken-shed-mgr. There you can see a possible directory layout.

What tools do you guys use to upload files to your boards?

I use VSCode with PyMakr plugin.

@rdehuyss Thanks! I already checked out that repo. However, on the board, I am quite sure that the main.py and the boot.py file needs to be located in the main dir (Unless you create a main and boot.py that points into the one inside src?) Therefore, should all the files (even the ones that is updated via OTA) be located in the main directory of the board? I think I understand the layout of the GitHub repo now and I will follow the format of your linked repo. Thanks for linking it! Once again, thanks for creating this and for your time!

rdehuyss commented 3 years ago

Yes, main.py and boot.py should always be in the root of your ESP device. I prefer them to almost not have any business logic as the ota-updater indeed does not update them. From the main.py, I call a method on a class in a subfolder (in case of chicken-shed-mgr app.start.py). Everything in app will be updated.

tongclement commented 3 years ago

Yes, main.py and boot.py should always be in the root of your ESP device. I prefer them to almost not have any business logic as the ota-updater indeed does not update them. From the main.py, I call a method on a class in a subfolder (in case of chicken-shed-mgr app.start.py). Everything in app will be updated.

So on the board, does Ota updater throw the new files into the main dir or a folder? Thanks

rdehuyss / micropython-ota-updater