Dashboard Update - Githubissues

msapogov commented 2 years ago

Experimentally, I found out that the version of the ESPHome dashboard can only be updated by deleting and reinstalling the adapter... iobroker del esphhome iobroker add esphome neither "download files" nor "restore" from the expert capabilities of the adapter work. Now, if you remove it and install it again, then the latest version is pulled up (at the moment v2022.1.1) but some objects have parameters set and they don't want to be lost along with the removal of the driver. How can I update the dashboard version without deleting the objects and their properties?

medjaiiii commented 2 years ago

I also noticed this feature, I wonder what to do?

msapogov commented 2 years ago

if the settings for the states of the objects are not important, then you can remove the adapter and install it again, then the dashboard will be updated. Well, or wait for the developer's response. Maybe there will be a separate button or command to update the dashboard.

Crash123Crash123 commented 2 years ago

I need to update too, to get ESP32CAM webserver working. Can I backup any folder, than delete espome and make a fresh install and than copy back the folder? It would be a big pain to make all new.

Edit: I deleted ESPHOME and added it new. All yaml are here. Of course I lost some states but mine are saved in influxdb so I lost only a few seconds. That´s ok for the moment.

msapogov commented 2 years ago

found a small solution how to update the dashboard in the IoB driver then new python modules are loaded, including the updated dashboard.

cd /opt/iobroker/node_modules/iobroker.esphome/ rm -R python_modules npm i --production

DutchmanNL commented 2 years ago

found a small solution how to update the dashboard in the IoB driver then new python modules are loaded, including the updated dashboard.

cd /opt/iobroker/node_modules/iobroker.esphome/ rm -R python_modules npm i --production

jup, trying to add this as an build in option so you do not need to update the adapter anymore for th dashboard just run an batch to install the new modulle

msapogov commented 2 years ago

Will you make a "refresh dashboard" button in the driver settings?

DutchmanNL commented 2 years ago

Will you make a "refresh dashboard" button in the driver settings?

that ws my attention yes, as I don't want to release a new adapteer version (without any code changes) every month when a new dashboard version is released.

So I would like to build something in that check the current an available. version, throw a log message if new version is available with option to update directly from ioBroker without new install or update of the adapter. itsel

medjaiiii commented 1 year ago

Thus, it stopped updating. The version is downloaded v2022.9.4. What should I do?

root@smart-home:/opt/iobroker/node_modules/iobroker.esphome# cd /opt/iobroker/node_modules/iobroker.esphome/ root@smart-home:/opt/iobroker/node_modules/iobroker.esphome# rm -R python_modules root@smart-home:/opt/iobroker/node_modules/iobroker.esphome# npm i --production npm WARN config production Use --omit=dev instead.

iobroker.esphome@0.2.4 install npip install

No python_modules directory; installing pip locally if needed. pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8) Collecting esphome>=2021.8 Using cached esphome-2022.9.4-py2.py3-none-any.whl (2.4 MB) Collecting tornado>=3.2 Using cached tornado-6.3.2-cp38-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (426 kB) Collecting PyYAML==6.0 Using cached PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (701 kB) Collecting esphome-dashboard==20221007.0 Using cached esphome_dashboard-20221007.0-py3-none-any.whl (1.6 MB) Processing /home/iobroker/.cache/pip/wheels/6a/48/01/c895c027e9b9367ec5470fbf371ee56e795a49ac6a19aa4c9f/paho_mqtt-1.6.1-py3-none-any.whl Collecting zeroconf==0.39.1 Using cached zeroconf-0.39.1-py3-none-any.whl (106 kB) Requirement already satisfied: pyserial==3.5 in /usr/local/lib/python3.8/dist-packages (from esphome>=2021.8) (3.5) Collecting tzlocal==4.2 Using cached tzlocal-4.2-py3-none-any.whl (19 kB) Collecting colorama==0.4.5 Using cached colorama-0.4.5-py2.py3-none-any.whl (16 kB) Collecting voluptuous==0.13.1 Using cached voluptuous-0.13.1-py3-none-any.whl (29 kB) Collecting tzdata>=2021.1 Using cached tzdata-2023.3-py2.py3-none-any.whl (341 kB) Processing /home/iobroker/.cache/pip/wheels/b6/fe/53/cbb405a8d5b53ae1ec8a58e7ffd596392194ee720f324aa4f6/esptool-3.3.1-py3-none-any.whl Collecting kconfiglib==13.7.1 Using cached kconfiglib-13.7.1-py2.py3-none-any.whl (145 kB) Requirement already satisfied: click==8.1.3 in /usr/local/lib/python3.8/dist-packages (from esphome>=2021.8) (8.1.3) Processing /home/iobroker/.cache/pip/wheels/5f/0d/e9/134bc067a7dd030d188de3976be93c90447a96c52688a1636d/platformio-6.0.2-py3-none-any.whl Collecting aioesphomeapi==10.13.0 Using cached aioesphomeapi-10.13.0-py2.py3-none-any.whl (52 kB) Collecting async-timeout>=4.0.1 Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB) Collecting ifaddr>=0.1.7 Using cached ifaddr-0.2.0-py3-none-any.whl (12 kB) Collecting backports.zoneinfo; python_version < "3.9" Using cached backports.zoneinfo-0.2.1-cp38-cp38-manylinux1_x86_64.whl (74 kB) Collecting pytz-deprecation-shim Using cached pytz_deprecation_shim-0.1.0.post0-py2.py3-none-any.whl (15 kB) Requirement already satisfied: cryptography>=2.1.4 in /usr/lib/python3/dist-packages (from esptool==3.3.1->esphome>=2021.8) (2.8) Processing /home/iobroker/.cache/pip/wheels/09/d5/d4/7485a909ef156de1ad5d336eecbd784dd9d54af373850ff4bc/reedsolo-1.5.4-py3-none-any.whl Collecting ecdsa>=0.16.0 Using cached ecdsa-0.18.0-py2.py3-none-any.whl (142 kB) Collecting bitstring>=3.1.6 Using cached bitstring-4.0.2-py3-none-any.whl (46 kB) Collecting aiofiles==0.8. Using cached aiofiles-0.8.0-py3-none-any.whl (13 kB) Requirement already satisfied: ajsonrpc==1. in /usr/local/lib/python3.8/dist-packages (from platformio==6.0.2->esphome>=2021.8) (1.2.0) Requirement already satisfied: marshmallow==3. in /usr/local/lib/python3.8/dist-packages (from platformio==6.0.2->esphome>=2021.8) (3.19.0) Requirement already satisfied: pyelftools<1,>=0.27 in /usr/local/lib/python3.8/dist-packages (from platformio==6.0.2->esphome>=2021.8) (0.29) Requirement already satisfied: semantic-version==2.10. in /usr/local/lib/python3.8/dist-packages (from platformio==6.0.2->esphome>=2021.8) (2.10.0) Collecting starlette==0.20. Using cached starlette-0.20.4-py3-none-any.whl (63 kB) Collecting wsproto==1.1. Using cached wsproto-1.1.0-py3-none-any.whl (24 kB) Collecting tabulate==0.8. Using cached tabulate-0.8.10-py3-none-any.whl (29 kB) Requirement already satisfied: bottle==0.12. in /usr/local/lib/python3.8/dist-packages (from platformio==6.0.2->esphome>=2021.8) (0.12.25) Collecting uvicorn==0.17. Using cached uvicorn-0.17.6-py3-none-any.whl (53 kB) Requirement already satisfied: requests==2. in /usr/lib/python3/dist-packages (from platformio==6.0.2->esphome>=2021.8) (2.22.0) Requirement already satisfied: protobuf<4.0,>=3.12.2 in /usr/lib/python3/dist-packages (from aioesphomeapi==10.13.0->esphome>=2021.8) (3.12.3) Collecting noiseprotocol<1.0,>=0.3.1 Using cached noiseprotocol-0.3.1-py3-none-any.whl (20 kB) Requirement already satisfied: six>=1.9.0 in /usr/lib/python3/dist-packages (from ecdsa>=0.16.0->esptool==3.3.1->esphome>=2021.8) (1.14.0) Requirement already satisfied: packaging>=17.0 in /usr/lib/python3/dist-packages (from marshmallow==3.->platformio==6.0.2->esphome>=2021.8) (20.3) Requirement already satisfied: typing-extensions>=3.10.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from starlette==0.20.->platformio==6.0.2->esphome>=2021.8) (4.5.0) Requirement already satisfied: anyio<5,>=3.4.0 in /usr/local/lib/python3.8/dist-packages (from starlette==0.20.->platformio==6.0.2->esphome>=2021.8) (3.6.2) Requirement already satisfied: h11<1,>=0.9.0 in /usr/local/lib/python3.8/dist-packages (from wsproto==1.1.->platformio==6.0.2->esphome>=2021.8) (0.14.0) Collecting asgiref>=3.4.0 Using cached asgiref-3.7.2-py3-none-any.whl (24 kB) Requirement already satisfied: idna>=2.8 in /usr/lib/python3/dist-packages (from anyio<5,>=3.4.0->starlette==0.20.->platformio==6.0.2->esphome>=2021.8) (2.8) Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.8/dist-packages (from anyio<5,>=3.4.0->starlette==0.20.->platformio==6.0.2->esphome>=2021.8) (1.3.0) ERROR: esphome 2022.9.4 has requirement tornado==6.1, but you'll have tornado 6.3.2 which is incompatible. Installing collected packages: PyYAML, esphome-dashboard, paho-mqtt, async-timeout, ifaddr, zeroconf, tornado, backports.zoneinfo, tzdata, pytz-deprecation-shim, tzlocal, colorama, voluptuous, reedsolo, ecdsa, bitstring, esptool, kconfiglib, aiofiles, starlette, wsproto, tabulate, asgiref, uvicorn, platformio, noiseprotocol, aioesphomeapi, esphome Successfully installed PyYAML-6.0 aioesphomeapi-10.13.0 aiofiles-0.8.0 asgiref-3.7.2 async-timeout-4.0.2 backports.zoneinfo-0.2.1 bitstring-4.0.2 colorama-0.4.5 ecdsa-0.18.0 esphome-2022.9.4 esphome-dashboard-20221007.0 esptool-3.3.1 ifaddr-0.2.0 kconfiglib-13.7.1 noiseprotocol-0.3.1 paho-mqtt-1.6.1 platformio-6.0.2 pytz-deprecation-shim-0.1.0.post0 reedsolo-1.5.4 starlette-0.20.4 tabulate-0.8.10 tornado-6.3.2 tzdata-2023.3 tzlocal-4.2 uvicorn-0.17.6 voluptuous-0.13.1 wsproto-1.1.0 zeroconf-0.39.1

up to date, audited 141 packages in 13s

15 packages are looking for funding run npm fund for details

found 0 vulnerabilities

DutchmanNL commented 11 months ago

please install latest version from git, we update the complete way how dashboard is integrated by other python solution which should solve this issue

SimonFischer04 commented 11 months ago

please install latest version from git, we update the complete way how dashboard is integrated by other python solution which should solve this issue

Yeah this issue seem to be fallen victim to the common case or unrelated issue misuse. @medjaiiii issue should in fact be solved by the new python logic.

But the original was about updating the integrated dashboard. And i'm not quite sure if this is solved already. Yes, i did not specifically define a version of the python dashbiard dependency to download - so installs the latest. But i dont think this automatically updates if it finds existing one.

SimonFischer04 commented 11 months ago

Will you make a "refresh dashboard" button in the driver settings?

that ws my attention yes, as I don't want to release a new adapteer version (without any code changes) every month when a new dashboard version is released.

So I would like to build something in that check the current an available. version, throw a log message if new version is available with option to update directly from ioBroker without new install or update of the adapter. itsel

This is also another thing i wanted to discuss. Im not quite sure if its the best idea to create another dependency management system inside the adapter. I think just using a refresh would ideally be not enough because the could be something breaking with newer dashboard versions requiring also the option to downgrade / install specific version. (F.e. legacy password option completely removed before new encryption system implemented)

I was kindof thinking the other direction leveraging existing versioning system. By (semi) automating new releases. By that I mean a dependabot ( like) system that automatically creates pull-requests for the dashboard python package as well.

What do you think?

DutchmanNL commented 11 months ago

But the original was about updating the integrated dashboard. And i'm not quite sure if this is solved already. Yes, i did not specifically define a version of the python dashbiard dependency to download - so installs the latest. But i dont think this automatically updates if it finds existing one.

problem indeed is it install the latest version, but after that not updating anymore and ESPHome Dashboard has monthly (sometimes even weekly) releases

DutchmanNL commented 11 months ago

I was kindof thinking the other direction leveraging existing versioning system. By (semi) automating new releases. By that I mean a dependabot ( like) system that automatically creates pull-requests for the dashboard python package as well.

I fully agree and support his approach, as this would also be visible for users that an update is available which will update the ESPHome dashboard too.

As I have seen, in the new integration we have the possibility to provide a version in our main.js, I tried that but its not working :(

I agree with approach of an GitHub action, if we are able to manage to specify the version in theory automation can be done to adopt a new ESPHome Dashboard automatically.

I already included automated releases in this repository, meaning the GitHub "deploy" action can already automatically make a release and publish it to NPM.

For that purpose only a comment must be made with the proper updates and version tags, than the workflow will kick in and create an NPM release.

This is handled in our GitHub release action: https://github.com/DrozmotiX/ioBroker.esphome/blob/main/.github/workflows/test-and-release.yml#L80-L145

SimonFischer04 commented 11 months ago

As I have seen, in the new integration we have the possibility to provide a version in our main.js, I tried that but its not working :(

I think i tried that in the beginning when integrating this lib. Will try again / look into this.

For that purpose only a comment must be made with the proper updates and version tags, than the workflow will kick in and create an NPM release.

What do you mean by comment. The changelog in the README? Anyways, first things first: Working on the dashboard dependency change.

DutchmanNL commented 11 months ago

What do you mean by comment. The changelog in the README? Anyways, first things first: Working on the dashboard dependency change.

no I mean in the git comment :)

see line 83 of our release workflow:

https://github.com/DrozmotiX/ioBroker.esphome/blob/main/.github/workflows/test-and-release.yml#L83

        # Trigger this step only when a commit on any branch is tagged with a version number
        if: |
            contains(github.event.head_commit.message, '[skip ci]') == false &&
            github.event_name == 'push' &&
            startsWith(github.ref, 'refs/tags/v')

so that means, as soon an commit is made with a release in it, like this one: https://github.com/DrozmotiX/ioBroker.esphome/commit/64183bdeb8bd4eacf5baaddb25529c26528965b6

the workflow will automatically deploy a new release to NPM But we have to make sure that:

package.json
io-package.json (including news)
package-lock.json
readme

are update accordingly

SimonFischer04 commented 11 months ago

package.json

io-package.json (including news)

package-lock.json

readme

are update accordingly

hm, i don't think any existing dependency management tool can handle these (iobroker) specifics?` I guess will gave to build something custom.

DutchmanNL commented 11 months ago

package.json

io-package.json (including news)

package-lock.json

readme

are update accordingly

hm, i don't think any existing dependency management tool can handle these (iobroker) specifics?` I guess will gave to build something custom.

I don't think so, expect that this must be build custom (specially the io-package part) to be supported. In generic, that's the logic I see a little:

if a new ESPDashboard version is release, the action should kick in (or check for new releases on a weekly base)
when a new release is available, that dependency must be added/changed to the code (question will be how/were but you were already looking into that)
version in mention package files must be update (I would like to just increase the patch, so 3rd digit in the release by +1
news must be written in io-package, schema is:

      "version": {
        "en": "ESPHome Dashboard update to $version

not sure if that's easily doable in an GitHub action, the could also consider to cover this by an NodeJS proces/executable which an run on one of my root servers

DutchmanNL commented 11 months ago

just thinking a little more around that, for now that's maybe even the most easiest solution as local the data can be updated and also release script used which handle all other things automatically (update the package files & translate news)

what do you think? Continue looking for solution as action or local NodeJS proces ?

SimonFischer04 commented 11 months ago

what do you think? Continue looking for solution as action or local NodeJS proces ?

Agree, existing Action base is probably not feasible. Thinking of probably just creating a js-script to run. That can be run locally (or in fact also just run in a simple action "base ubuntu, clone, install node, npm run dashboard-update" or something)

DutchmanNL commented 11 months ago

what do you think? Continue looking for solution as action or local NodeJS proces ?

Agree,

existing Action base is probably not feasible. Thinking of probably just creating a js-script to run. That can be run locally (or in fact also just run in a simple action "base ubuntu, clone, install node, npm run dashboard-update" or something)

Sounds like a plan and less complexity for now.

An GitHub action would be an ultimate nice solution but also quite complex I assume (not sure never created one 😅)

SimonFischer04 commented 11 months ago

(not sure never created one 😅)

same 😅

SimonFischer04 commented 11 months ago

But we have to make sure that:

package.json

io-package.json (including news)

package-lock.json

readme

are update accordingly

My current progress might be a "little" overcomplicated. 🤣 Wouldn't it be enough to "just" update the dashboard dependency, add info to readme and then run release-script patch.?

DutchmanNL commented 11 months ago

Wouldn't it be enough to "just" update the dashboard dependency, add info to readme and then run release-script

doing it locally yes, release script will take care of all other actions. But where do we define the ESPHome dashboard version to be used and are aware of new release to trigger the automation (or let the automation check for an new version)

that summary was an overview of all steps needed for a release, which are covered in the release script)

SimonFischer04 commented 11 months ago

Wouldn't it be enough to "just" update the dashboard dependency, add info to readme and then run release-script

doing it locally yes, release script will take care of all other actions. But where do we define the ESPHome dashboard version to be used and are aware of new release to trigger the automation (or let the automation check for an new version)

that summary was an overview of all steps needed for a release, which are covered in the release script)

my new plan would for now be:

a simple cron shedule triggering a js-script:
get currently version (simply using GET https://api.github.com/repos/esphome/esphome/releases)
compare local version with current release. if newer >
- change version in main.js / move it somewhere else
- add some auto-comment to readme (need to figure out how to parse / add something using script properly)
- run the release-script

DutchmanNL commented 11 months ago

Wouldn't it be enough to "just" update the dashboard dependency, add info to readme and then run release-script

doing it locally yes, release script will take care of all other actions. But where do we define the ESPHome dashboard version to be used and are aware of new release to trigger the automation (or let the automation check for an new version)

that summary was an overview of all steps needed for a release, which are covered in the release script)

my new plan would for now be:

a simple cron shedule triggering a js-script:

get currently version (simply using GET https://api.github.com/repos/esphome/esphome/releases)

compare local version with current release. if newer >

change version in main.js / move it somewhere else

add some auto-comment to readme (need to figure out how to parse / add something using script properly)

run the release-script

Jep makes sense

SimonFischer04 commented 11 months ago

Thinking a bit more... something else. Have to handle / deal with existing stuff in the "Worl in progress" section in the readme. Probably just temporarily remove it and add back in a dedicated commit after release.

DutchmanNL commented 11 months ago

Thinking a bit more... something else. Have to handle / deal with existing stuff in the "Worl in progress" section in the readme. Probably just temporarily remove it and add back in a dedicated commit after release.

When we do auto releasing, all other changes should be handled in branches to ensure main branch always contains code which is equal to npm.

By having that discipline there will never be an "work in progress section" when handling an update for dashboard

We also must avoid that "unready code" is deployed to latest, und I guess only way to properly do that is keeping main branch clean

As we will use the release script, any content of "work in progress" section will be part of release anyway so that's fine

Don't worry about that

SimonFischer04 commented 11 months ago

So the initial state of the README for the update-script would be:

## Changelog

<!--
    Placeholder for the next version (at the beginning of the line):
    ### __WORK IN PROGRESS__
    * (DutchmanNL) 
-->
### 0.4.1 (2023-11-05)
...

basically adding a markdown h3

### **WORK IN PROGRESS**
* (CI) Update integrated Dashboard from Version X to X and Python from Version X to X.

bellow the placeholder inside the changelog h2?

DutchmanNL commented 11 months ago

So the initial state of the README for the update-script would be:
## Changelog



### 0.4.1 (2023-11-05)

...
basically adding a markdown h3
### **WORK IN PROGRESS**

* (CI) Update integrated Dashboard from Version X to X and Python from Version X to X.
bellow the placeholder inside the changelog h2?

Yes and no 😅 After considering this, I realised that I cannot prevent code being in main stream not equal to npm due to dependency merges etc

So we have 2 scenarios

1) line under changelog starts with:

x.x.x (xxxx)

"### WORK IN PROGRESS" must be added + entry about Dashboard version update and release script executed

2) line under changelog starts with:

WORK IN PROGRESS

entry about Dashboard version update and release script executed

In both scenarios it's ok to release as it's beta only, stable must be handled by pr to ioBroker repo

DutchmanNL commented 11 months ago

Just to add, giving the proper arguments to release command will automatically handle version upgrade

I will add a script execution for that in my next commit as "dashboardRelease".

So if you run "npm run dashboardRelease" it will running release script, automatically choose the option "patch" and update readme with version + news to io-package

DutchmanNL commented 11 months ago

And, after a little research, this allows us to run it automatically by an GitHub action 😀

Build npm script to
[ ] Check current dashboard version vs version of last release
[ ] if new version available, execute the steps we discussed above
[ ] execute npm run dashboardRelease
[ ] build GitHub action to run the npm executable as mentioned here and for testing (as example) already implemented. like here

Npm process will do its job and a new release is created by GitHub action already available and triggers by release script.

That GitHub action can be triggered manually or by scheduled workflow (like dependency updater)

https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-nodejs

SimonFischer04 commented 10 months ago

So the initial state of the README for the update-script would be:
## Changelog



### 0.4.1 (2023-11-05)

...
basically adding a markdown h3
### **WORK IN PROGRESS**

* (CI) Update integrated Dashboard from Version X to X and Python from Version X to X.
bellow the placeholder inside the changelog h2?
Yes and no 😅 After considering this, I realised that I cannot prevent code being in main stream not equal to npm due to dependency merges etc

So we have 2 scenarios

line under changelog starts with:

x.x.x (xxxx)

"### WORK IN PROGRESS" must be added + entry about Dashboard version update and release script executed

line under changelog starts with:

WORK IN PROGRESS

entry about Dashboard version update and release script executed

In both scenarios it's ok to release as it's beta only, stable must be handled by pr to ioBroker repo

FYI: I currently try to get the readme-related stuff "solved" upstream: https://github.com/AlCalzone/release-script/pull/154

DutchmanNL commented 10 months ago

FYI: I currently try to get the readme-related stuff "solved" upstream: AlCalzone/release-script#154

Cool thank you!

DutchmanNL commented 10 months ago

implemented in 0.5.0-beta.10, will be part of 0.5.x release

Functionality:

adapter checks available versions during start
list is provided in admin interface to select specific versions
also possible to always use latest available (default setting)

DrozmotiX / ioBroker.esphome

Dashboard Update #118

x.x.x (xxxx)

WORK IN PROGRESS

x.x.x (xxxx)

WORK IN PROGRESS