Centralized docker file for all use cases

GeoDerp commented 9 months ago

This is not tested yet. Still need to work out how to best test this thoroughly. If you test this yourself feel free to re-add the finish & run files. Sorry, I wasn't shore if they would be needed or not. (run shouldn't be)

GeoDerp commented 9 months ago

@davidusb-geek , Do we need (/would you like) to continue to publish the Docker images to DockerHub for HA addon? As the Add-on repository is github repository anyways? ie. on git commit (and config.yml version change ) users will pull the docker from git repo and build. I commented out the image: "davidusb/image-{arch}-emhass" Line so the addon builds using its local Dockerfile. But im happy to change it back.

Edit: I set https://github.com/GeoDerp/emhass-add-on.git as a repository in my HA environment and it run fine using the Local Dockerfile.

I also changed the version: to 0.6.7 on config.yaml in https://github.com/GeoDerp/emhass-add-on.git and it did tell my HA environment an update was available.

GeoDerp commented 9 months ago

I created a Test.md And shortend the Dockerfile to only the necessary options. Its still using the same base as PR 200, just less for the addon to have issues with.

GeoDerp commented 9 months ago

More then happy to uncomment the image line (just let me know). If so, I think it would be good to test the github workspaces to see if they require the build argument passed. (or they copy the build arguments from build.yaml)

GeoDerp commented 9 months ago

Just a note if you test this using the regular addon mode it will fail, as emhass pip its trying to find the config in /share directory. Testing using addon-git should work using repo and branch from pr 200: https://github.com/GeoDerp/emhass/tree/centralized-Docker And pip should work when built with PR 200.

davidusb-geek commented 9 months ago

Ok I will test and see if I got all this right. Just one quick question, I thought that the add-on image was always pulled from Dockerhub and then loaded. But you are saying that it is always build locally by the user HA instance?

GeoDerp commented 9 months ago

Ok I will test and see if I got all this right. Just one quick question, I thought that the add-on image was always pulled from Dockerhub and then loaded. But you are saying that it is always build locally by the user HA instance?

Good question. It depends if you have the image: "davidusb/image-{arch}-emhass" commented out or not in he config.yaml. That line will pull from docker hub. otherwise it will pull from git and build locally.

I have commented it out for local testing but we can add it back in.

davidusb-geek commented 9 months ago

Ok yes that's actually what I do when locally testing. Just comment that line and put the add-on files on the add-on folder in my HA instance, then a local add-on shows for testing. I think that we can leave the image for downloading from docker-hub, it can be useful for users with some limited resources (some Rpi's).

GeoDerp commented 9 months ago

Ok yes that's actually what I do when locally testing. Just comment that line and put the add-on files on the add-on folder in my HA instance, then a local add-on shows for testing. I think that we can leave the image for downloading from docker-hub, it can be useful for users with some limited resources (some Rpi's).

Sounds good. I'll uncomment the line and also tweak the test markdown now 👍

GeoDerp commented 9 months ago

If the publish worflows don't work maybe adding the line --generic latest. I think this may help building based on build.yaml (which store the build args), but i'm not sure.

If that doesnt work, try adding back: (with --generic latest)

build_from:
  amd64: ghcr.io/home-assistant/amd64-base-debian:bookworm
  armhf: ghcr.io/home-assistant/armhf-base-debian:bookworm
  armv7: ghcr.io/home-assistant/armv7-base-debian:bookworm
  aarch64: ghcr.io/home-assistant/aarch64-base-debian:bookworm

and?

ARG BUILD_FROM
FROM ${BUILD_FROM}

davidusb-geek commented 9 months ago

If the publish worflows don't work maybe adding the line --generic latest. I think this may help building based on build.yaml (which store the build args), but i'm not sure.

Where should we add this line?

GeoDerp commented 9 months ago

example:

  publish_armv7:
    name: Publish-armv7
    runs-on: ubuntu-latest
    steps:
      - name: Checkout the repository
        uses: actions/checkout@v2
      - name: Login to DockerHub
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - name: Publish
        uses: home-assistant/builder@master
        with:
          args: |
            --armv7 \
            --target emhass \
            --docker-hub davidusb \
            --generic latest

GeoDerp commented 9 months ago

Im seeing now if we could integrate https://github.com/home-assistant/builder/blob/master/.github/workflows/test.yml into this repo to test

GeoDerp commented 9 months ago

https://github.com/GeoDerp/emhass-add-on/blob/live-test/.github/workflows/test.yaml

GeoDerp commented 9 months ago

https://github.com/GeoDerp/emhass-add-on/actions/runs/7989275935/job/21815534964

Looks like armhf failed with #9 246.9 ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

Edit: I'll have a look at this tomorrow. Likely package troubleshooting .

GeoDerp commented 9 months ago

The test.yaml by the way iterates the architectures and builds with test tag. It is triggered manually or on pull request .

davidusb-geek commented 9 months ago

That is very nice. The armhf builds are failing since some months ago. I've tried several times to solve it without success

GeoDerp commented 9 months ago

@davidusb-geek

Got it working with this Test result: https://github.com/GeoDerp/emhass-add-on/actions/runs/8005438416

Requirements:

pvlib>=0.10.2
protobuf>=3.0.0
pytz>=2021.1
requests>=2.25.1
beautifulsoup4>=4.9.3
h5py==3.10.0
pulp>=2.4
pyyaml>=5.4.1
tables<=3.9.1
skforecast==0.11.0
flask>=2.0.3
waitress>=2.1.1
plotly>=5.6.0
emhass==0.7.7
#git+https://github.com/davidusb-geek/emhass

Docker

...
RUN apt-get update \
    && apt-get install -y --no-install-recommends \
    libffi-dev \
    python3 \
    python3-pip \
    python3-dev \
    python3-numpy \
    python3-wheel \
    python3-setuptools \
    python3-setuptools-whl \
    python3-scipy \
    python3-pandas \
    git \
    build-essential \
    gcc \
    coinor-cbc \
    coinor-libcbc-dev \
    libglpk-dev \
    glpk-utils \
    libhdf5-dev \
    libhdf5-serial-dev \
    netcdf-bin \
    libnetcdf-dev \
    pkg-config \
    gfortran \
    libatlas-base-dev \
    && ln -s /usr/include/hdf5/serial /usr/include/hdf5/include \
    && export HDF5_DIR=/usr/include/hdf5 \
    && pip3 install --extra-index-url=https://www.piwheels.org/simple --no-cache-dir --break-system-packages -r requirements.txt \
    && apt-get purge -y --auto-remove \
    gcc \
    build-essential \
    libhdf5-dev \
    libhdf5-serial-dev \
    pkg-config \
    gfortran \
    && rm -rf /var/lib/apt/lists/*
...

More testing need however to see if this actually works, Or just the build tests are claiming to. My guess is that all this is doing is running in python3.11.

Edit; Yeah no, this skrews up pandas

  File "pandas/_libs/tslibs/strptime.pyx", line 150, in pandas._libs.tslibs.strptime.array_strptime
ValueError: time data '2024-02-19T14:00:00+00:00' does not match format 'ISO8601' (match)

GeoDerp commented 9 months ago

@davidusb-geek , Tomorrow (Friday) I'll modify the build arguments for git and then ill be pritty happy for both related PR's. I may also make the armhf test seperate.

davidusb-geek commented 9 months ago

Hi @GeoDerp, so were you able to solve the build issues for armhf archs? I see that they are not part of the automated builds now:

architecture: [amd64,aarch64,armv7]

GeoDerp commented 9 months ago

Yeah I have just set armhf to its own step so we can see the other three work. currently the workflow is broken tho. Fixing it now. And unfortunately, I haven't worked out a fix for armhf (besides bumping to python 3.11 which brakes its own issues ). Not saying there isn't a fix, just haven't had the time.

GeoDerp commented 9 months ago

set fail-fast: false to false. This continues the matrix even when one fails: https://github.com/orgs/community/discussions/27192#discussioncomment-3254966 https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategyfail-fast

GeoDerp commented 9 months ago

Relativity happy with it (its not perfect). Found a error when using the latest EMHASS (0.7.7) pip with the new Docker. Will work out if that a reliant issue tomorrow (as new git EMHASS works). Will also proof read my work and then done. Hopefully.

The error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 4 (char 3)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 207, in <module>
    config_hass = response.json()
                  ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Extra data: line 1 column 4 (char 3

Edit: I'll also try pip 0.7.8

Edit: I was being dumb, this was also because of a change that I implemented in EMHASS (setting key default if url ==http://supervisor/core/api ). IE wont be an issue on new verions of EMHASS.

GeoDerp commented 9 months ago

This last commit https://github.com/davidusb-geek/emhass-add-on/pull/77/commits/e481d41c7f226c11bd11fe91790ba538bd5e3279, this helps users to backport pip versions if necessary. This commit can be reverted however to support only pip versions starting from PR 200. (result is slightly cleaner)

GeoDerp commented 9 months ago

Also another note. Updating pandas support to the latest version would fix the armhf issue by setting:

    python3-numpy \
    python3-wheel \
    python3-setuptools \
    python3-setuptools-whl \
    python3-scipy \
    python3-pandas \

in apt-get instead of python requirements.txt

davidusb-geek commented 9 months ago

Thanks for all this work John. You tell me when it's ready to merge. It seems that some tests are falling

GeoDerp commented 9 months ago

Hopefully I'll be able to finish it tonight. The test may just be a TCP timeout. Just would like to do one change. However I'm currently out .

GeoDerp commented 9 months ago

@davidusb-geek , Should be ready to go. Will just wait for build test confirmation.

GeoDerp commented 9 months ago

Just a note, the armv7 takes around a 1hr 30min to build test. My guess is that the armv7 servers are just less powerful.

davidusb-geek commented 9 months ago

Everything seems good. Yes that's normal for armv7, it has been like that for add-on releases. It was even worst before I found the trick of passing the pywheels url. Before that it took like 10 hours! Armhf is still failing but that won't block a new release. It seems like a problem installing Pandas along with Python 3.11 on that arch.

GeoDerp commented 9 months ago

Both PR's should be ready to be tested and pulled. No stress if you choose not to use it. 👍

Let me know if anything comes up. 😁

davidusb-geek / emhass-add-on

Centralized docker file for all use cases #77