Centralized docker file for all use cases

[GeoDerp]

Related to https://github.com/davidusb-geek/emhass/pull/200

[GeoDerp]

This is not tested yet. Still need to work out how to best test this thoroughly. If you test this yourself feel free to re-add the finish & run files. Sorry, I wasn't shore if they would be needed or not. (run shouldn't be)

[GeoDerp]

@davidusb-geek , Do we need (/would you like) to continue to publish the Docker images to DockerHub for HA addon? As the Add-on repository is github repository anyways? ie. on git commit (and config.yml version change ) users will pull the docker from git repo and build. I commented out the image: "davidusb/image-{arch}-emhass" Line so the addon builds using its local Dockerfile. But im happy to change it back.

Edit: I set https://github.com/GeoDerp/emhass-add-on.git as a repository in my HA environment and it run fine using the Local Dockerfile.

I also changed the version: to 0.6.7 on config.yaml in https://github.com/GeoDerp/emhass-add-on.git and it did tell my HA environment an update was available.

[GeoDerp]

I created a Test.md And shortend the Dockerfile to only the necessary options. Its still using the same base as PR 200, just less for the addon to have issues with.

[GeoDerp]

More then happy to uncomment the image line (just let me know). If so, I think it would be good to test the github workspaces to see if they require the build argument passed. (or they copy the build arguments from build.yaml)

[GeoDerp]

Just a note if you test this using the regular addon mode it will fail, as emhass pip its trying to find the config in /share directory. Testing using addon-git should work using repo and branch from pr 200: https://github.com/GeoDerp/emhass/tree/centralized-Docker And pip should work when built with PR 200.

[davidusb-geek]

Ok I will test and see if I got all this right. Just one quick question, I thought that the add-on image was always pulled from Dockerhub and then loaded. But you are saying that it is always build locally by the user HA instance?

[GeoDerp]

Ok I will test and see if I got all this right. Just one quick question, I thought that the add-on image was always pulled from Dockerhub and then loaded. But you are saying that it is always build locally by the user HA instance?

Good question. It depends if you have the image: "davidusb/image-{arch}-emhass" commented out or not in he config.yaml. That line will pull from docker hub. otherwise it will pull from git and build locally.

I have commented it out for local testing but we can add it back in.

[davidusb-geek]

Ok yes that's actually what I do when locally testing. Just comment that line and put the add-on files on the add-on folder in my HA instance, then a local add-on shows for testing. I think that we can leave the image for downloading from docker-hub, it can be useful for users with some limited resources (some Rpi's).

[GeoDerp]

Ok yes that's actually what I do when locally testing. Just comment that line and put the add-on files on the add-on folder in my HA instance, then a local add-on shows for testing. I think that we can leave the image for downloading from docker-hub, it can be useful for users with some limited resources (some Rpi's).

Sounds good. I'll uncomment the line and also tweak the test markdown now 👍

[GeoDerp]

If the publish worflows don't work maybe adding the line --generic latest. I think this may help building based on build.yaml (which store the build args), but i'm not sure.

If that doesnt work, try adding back: (with --generic latest)

build_from:
  amd64: ghcr.io/home-assistant/amd64-base-debian:bookworm
  armhf: ghcr.io/home-assistant/armhf-base-debian:bookworm
  armv7: ghcr.io/home-assistant/armv7-base-debian:bookworm
  aarch64: ghcr.io/home-assistant/aarch64-base-debian:bookworm

and?

ARG BUILD_FROM
FROM ${BUILD_FROM}

[davidusb-geek]

If the publish worflows don't work maybe adding the line --generic latest. I think this may help building based on build.yaml (which store the build args), but i'm not sure.

Where should we add this line?

[GeoDerp]

example:

  publish_armv7:
    name: Publish-armv7
    runs-on: ubuntu-latest
    steps:
      - name: Checkout the repository
        uses: actions/checkout@v2
      - name: Login to DockerHub
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - name: Publish
        uses: home-assistant/builder@master
        with:
          args: |
            --armv7 \
            --target emhass \
            --docker-hub davidusb \
            --generic latest

[GeoDerp]

Im seeing now if we could integrate https://github.com/home-assistant/builder/blob/master/.github/workflows/test.yml into this repo to test

[GeoDerp]

https://github.com/GeoDerp/emhass-add-on/blob/live-test/.github/workflows/test.yaml

[GeoDerp]

https://github.com/GeoDerp/emhass-add-on/actions/runs/7989275935/job/21815534964

Looks like armhf failed with #9 246.9 ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

Edit: I'll have a look at this tomorrow. Likely package troubleshooting .

[GeoDerp]

The test.yaml by the way iterates the architectures and builds with test tag. It is triggered manually or on pull request .

[davidusb-geek]

That is very nice. The armhf builds are failing since some months ago. I've tried several times to solve it without success

[GeoDerp]

@davidusb-geek

Got it working with this Test result: https://github.com/GeoDerp/emhass-add-on/actions/runs/8005438416

Requirements:

pvlib>=0.10.2
protobuf>=3.0.0
pytz>=2021.1
requests>=2.25.1
beautifulsoup4>=4.9.3
h5py==3.10.0
pulp>=2.4
pyyaml>=5.4.1
tables<=3.9.1
skforecast==0.11.0
flask>=2.0.3
waitress>=2.1.1
plotly>=5.6.0
emhass==0.7.7
#git+https://github.com/davidusb-geek/emhass

Docker

...
RUN apt-get update \
    && apt-get install -y --no-install-recommends \
    libffi-dev \
    python3 \
    python3-pip \
    python3-dev \
    python3-numpy \
    python3-wheel \
    python3-setuptools \
    python3-setuptools-whl \
    python3-scipy \
    python3-pandas \
    git \
    build-essential \
    gcc \
    coinor-cbc \
    coinor-libcbc-dev \
    libglpk-dev \
    glpk-utils \
    libhdf5-dev \
    libhdf5-serial-dev \
    netcdf-bin \
    libnetcdf-dev \
    pkg-config \
    gfortran \
    libatlas-base-dev \
    && ln -s /usr/include/hdf5/serial /usr/include/hdf5/include \
    && export HDF5_DIR=/usr/include/hdf5 \
    && pip3 install --extra-index-url=https://www.piwheels.org/simple --no-cache-dir --break-system-packages -r requirements.txt \
    && apt-get purge -y --auto-remove \
    gcc \
    build-essential \
    libhdf5-dev \
    libhdf5-serial-dev \
    pkg-config \
    gfortran \
    && rm -rf /var/lib/apt/lists/*
...

More testing need however to see if this actually works, Or just the build tests are claiming to. My guess is that all this is doing is running in python3.11.

Edit; Yeah no, this skrews up pandas

  File "pandas/_libs/tslibs/strptime.pyx", line 150, in pandas._libs.tslibs.strptime.array_strptime
ValueError: time data '2024-02-19T14:00:00+00:00' does not match format 'ISO8601' (match)

[GeoDerp]

@davidusb-geek , Tomorrow (Friday) I'll modify the build arguments for git and then ill be pritty happy for both related PR's. I may also make the armhf test seperate.

[davidusb-geek]

Hi @GeoDerp, so were you able to solve the build issues for armhf archs? I see that they are not part of the automated builds now:

architecture: [amd64,aarch64,armv7]

[GeoDerp]

Yeah I have just set armhf to its own step so we can see the other three work. currently the workflow is broken tho. Fixing it now. And unfortunately, I haven't worked out a fix for armhf (besides bumping to python 3.11 which brakes its own issues ). Not saying there isn't a fix, just haven't had the time.

[GeoDerp]

set fail-fast: false to false. This continues the matrix even when one fails: https://github.com/orgs/community/discussions/27192#discussioncomment-3254966 https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategyfail-fast

[GeoDerp]

Relativity happy with it (its not perfect). Found a error when using the latest EMHASS (0.7.7) pip with the new Docker. Will work out if that a reliant issue tomorrow (as new git EMHASS works). Will also proof read my work and then done. Hopefully.

The error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 4 (char 3)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 207, in <module>
    config_hass = response.json()
                  ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Extra data: line 1 column 4 (char 3

Edit: I'll also try pip 0.7.8

Edit: I was being dumb, this was also because of a change that I implemented in EMHASS (setting key default if url ==http://supervisor/core/api ). IE wont be an issue on new verions of EMHASS.

[GeoDerp]

This last commit https://github.com/davidusb-geek/emhass-add-on/pull/77/commits/e481d41c7f226c11bd11fe91790ba538bd5e3279, this helps users to backport pip versions if necessary. This commit can be reverted however to support only pip versions starting from PR 200. (result is slightly cleaner)

[GeoDerp]

Also another note. Updating pandas support to the latest version would fix the armhf issue by setting:

    python3-numpy \
    python3-wheel \
    python3-setuptools \
    python3-setuptools-whl \
    python3-scipy \
    python3-pandas \

in apt-get instead of python requirements.txt

[davidusb-geek]

Thanks for all this work John. You tell me when it's ready to merge. It seems that some tests are falling

[GeoDerp]

Hopefully I'll be able to finish it tonight. The test may just be a TCP timeout. Just would like to do one change. However I'm currently out .

[GeoDerp]

@davidusb-geek , Should be ready to go. Will just wait for build test confirmation.

[GeoDerp]

Just a note, the armv7 takes around a 1hr 30min to build test. My guess is that the armv7 servers are just less powerful.

[davidusb-geek]

Everything seems good. Yes that's normal for armv7, it has been like that for add-on releases. It was even worst before I found the trick of passing the pywheels url. Before that it took like 10 hours! Armhf is still failing but that won't block a new release. It seems like a problem installing Pandas along with Python 3.11 on that arch.

[GeoDerp]

Both PR's should be ready to be tested and pulled. No stress if you choose not to use it. 👍

Let me know if anything comes up. 😁