Open gsaviane opened 1 year ago
dulfer
commented
1 year ago Hi gsaviane! Interesting you needed to add those. Although there are some deprecation warnings, the image builds just fine without the dev-packages.
This line should install all the needed libs
RUN apk add --no-cache python3 py3-pip && ln -sf python3 /usr/bin/python
What OS/release are you using on your build host?
gsaviane
commented
1 year ago Hi dulfer, I'm using a raspberrypi with bullseye (Debian 11). I guess there might be some differences between x86 and arm64 alpine images. Below there is the entire log that I get building the image. I was also suggesting some changes in dsmr2mqtt.py because the DataPersist does not work actually. Would you allow PRs in your project?
I'm kinda interested to extend it and producing to MQTT the same data that smartgateway generates
BR
Giorgio
============================================================================= docker build . --tag failing Sending build context to Docker daemon 115.2kB Step 1/13 : FROM alpine:latest ---> 04eeaa5f8c35 Step 2/13 : RUN apk update && apk upgrade -q --prune && apk add --no-cache tzdata ---> Using cache ---> b3d55e35d633 Step 3/13 : ENV PYTHONUNBUFFERED=1 ---> Using cache ---> aead555fe457 Step 4/13 : RUN apk add --no-cache python3 py3-pip && ln -sf python3 /usr/bin/python ---> Running in a7601a181d96 fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/aarch64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/aarch64/APKINDEX.tar.gz (1/19) Installing libbz2 (1.0.8-r4) (2/19) Installing libexpat (2.5.0-r0) (3/19) Installing libffi (3.4.4-r0) (4/19) Installing gdbm (1.23-r0) (5/19) Installing xz-libs (5.2.9-r0) (6/19) Installing libgcc (12.2.1_git20220924-r4) (7/19) Installing libstdc++ (12.2.1_git20220924-r4) (8/19) Installing mpdecimal (2.5.1-r1) (9/19) Installing ncurses-terminfo-base (6.3_p20221119-r0) (10/19) Installing ncurses-libs (6.3_p20221119-r0) (11/19) Installing readline (8.2.0-r0) (12/19) Installing sqlite-libs (3.40.1-r0) (13/19) Installing python3 (3.10.9-r1) (14/19) Installing py3-six (1.16.0-r3) (15/19) Installing py3-retrying (1.3.3-r3) (16/19) Installing py3-parsing (3.0.9-r0) (17/19) Installing py3-packaging (21.3-r2) (18/19) Installing py3-setuptools (65.6.0-r0) (19/19) Installing py3-pip (22.3.1-r1) Executing busybox-1.35.0-r29.trigger OK: 88 MiB in 35 packages Removing intermediate container a7601a181d96 ---> 7503bacbc6e4 Step 5/13 : RUN python3 -m ensurepip ---> Running in bb4693f434cc Looking in links: /tmp/tmp1xl875ig Requirement already satisfied: setuptools in /usr/lib/python3.10/site-packages (65.6.0) Requirement already satisfied: pip in /usr/lib/python3.10/site-packages (22.3.1) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv Removing intermediate container bb4693f434cc ---> 6261667c573f Step 6/13 : RUN pip3 install --no-cache --upgrade pip setuptools ---> Running in ca9adbbef725 Requirement already satisfied: pip in /usr/lib/python3.10/site-packages (22.3.1) Requirement already satisfied: setuptools in /usr/lib/python3.10/site-packages (65.6.0) Collecting setuptools Downloading setuptools-66.1.1-py3-none-any.whl (1.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 7.4 MB/s eta 0:00:00 Installing collected packages: setuptools Attempting uninstall: setuptools Found existing installation: setuptools 65.6.0 Uninstalling setuptools-65.6.0: Successfully uninstalled setuptools-65.6.0 Successfully installed setuptools-66.1.1 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv Removing intermediate container ca9adbbef725 ---> a9f6cfc3c135 Step 7/13 : RUN mkdir -p /opt/dsmr2mqtt ---> Running in f502fe5b64d9 Removing intermediate container f502fe5b64d9 ---> 2e9bd8ea38e9 Step 8/13 : WORKDIR /opt/dsmr2mqtt ---> Running in 08124f4a6efc Removing intermediate container 08124f4a6efc ---> 789b3ba1532d Step 9/13 : COPY ./requirements.txt /opt/dsmr2mqtt/requirements.txt ---> 1881302c1b82 Step 10/13 : RUN pip3 install -r /opt/dsmr2mqtt/requirements.txt ---> Running in edb961c8e465 Collecting dsmr-parser Downloading dsmr_parser-1.0.0-py3-none-any.whl (33 kB) Collecting paho-mqtt Downloading paho-mqtt-1.6.1.tar.gz (99 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.4/99.4 kB 1.4 MB/s eta 0:00:00 Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting async-timeout Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB) Collecting pyserial<4,>=3 Downloading pyserial-3.5-py2.py3-none-any.whl (90 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 90.6/90.6 kB 1.3 MB/s eta 0:00:00 Collecting Tailer==0.4.1 Downloading tailer-0.4.1.tar.gz (7.5 kB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting dlms-cosem==21.3.2 Downloading dlms_cosem-21.3.2-py3-none-any.whl (128 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 128.8/128.8 kB 1.5 MB/s eta 0:00:00 Collecting pytz Downloading pytz-2022.7.1-py2.py3-none-any.whl (499 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 499.4/499.4 kB 271.9 kB/s eta 0:00:00 Collecting pyserial-asyncio<1 Downloading pyserial_asyncio-0.6-py3-none-any.whl (7.6 kB) Collecting attrs>=20.3.0 Downloading attrs-22.2.0-py3-none-any.whl (60 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.0/60.0 kB 840.6 kB/s eta 0:00:00 Collecting typing-extensions>=3.10 Downloading typing_extensions-4.4.0-py3-none-any.whl (26 kB) Collecting cryptography>=35.0.0 Downloading cryptography-39.0.0-cp36-abi3-musllinux_1_1_aarch64.whl (4.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.1/4.1 MB 1.5 MB/s eta 0:00:00 Collecting asn1crypto>=1.4.0 Downloading asn1crypto-1.5.1-py2.py3-none-any.whl (105 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 105.0/105.0 kB 45.6 kB/s eta 0:00:00 Collecting python-dateutil>=2.8.1 Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 2.2 MB/s eta 0:00:00 Collecting cffi>=1.12 Downloading cffi-1.15.1.tar.gz (508 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 508.5/508.5 kB 2.4 MB/s eta 0:00:00 Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Requirement already satisfied: six>=1.5 in /usr/lib/python3.10/site-packages (from python-dateutil>=2.8.1->dlms-cosem==21.3.2->dsmr-parser->-r /opt/dsmr2mqtt/requirements.txt (line 2)) (1.16.0) Collecting pycparser Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 kB 94.2 kB/s eta 0:00:00 Installing collected packages: Tailer, pytz, pyserial, paho-mqtt, asn1crypto, typing-extensions, python-dateutil, pyserial-asyncio, pycparser, attrs, async-timeout, cffi, cryptography, dlms-cosem, dsmr-parser DEPRECATION: Tailer is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559 Running setup.py install for Tailer: started Running setup.py install for Tailer: finished with status 'done' DEPRECATION: paho-mqtt is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559 Running setup.py install for paho-mqtt: started Running setup.py install for paho-mqtt: finished with status 'done' DEPRECATION: cffi is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559 Running setup.py install for cffi: started Running setup.py install for cffi: finished with status 'error' error: subprocess-exited-with-error
× Running setup.py install for cffi did not run successfully. │ exit code: 1 ╰─
Trying to continue anyway. If you are trying to install CFFI from
a build done in a different context, you can ignore this warning.
/usr/lib/python3.10/site-packages/setuptools/config/setupcfg.py:515:SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead. warnings.warn(msg, warning_class) running install /usr/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( running build running build_py creating build creating build/lib.linux-aarch64-cpython-310 creating build/lib.linux-aarch64-cpython-310/cffi copying cffi/pkgconfig.py -> build/lib.linux-aarch64-cpython-310/cffi copying cffi/cparser.py -> build/lib.linux-aarch64-cpython-310/cffi copying cffi/ffiplatform.py -> build/lib.linux-aarch64-cpython-310/cffi copying cffi/vengine_gen.py -> build/lib.linux-aarch64-cpython-310/cffi copying cffi/setuptools_ext.py -> build/lib.linux-aarch64-cpython-310/cffi copying cffi/vengine_cpy.py -> build/lib.linux-aarch64-cpython-310/cffi copying cffi/init.py -> build/lib.linux-aarch64-cpython-310/cffi copying cffi/commontypes.py -> build/lib.linux-aarch64-cpython-310/cffi copying cffi/recompiler.py -> build/lib.linux-aarch64-cpython-310/cffi copying cffi/verifier.py -> build/lib.linux-aarch64-cpython-310/cffi copying cffi/backend_ctypes.py -> build/lib.linux-aarch64-cpython-310/cffi copying cffi/model.py -> build/lib.linux-aarch64-cpython-310/cffi copying cffi/cffi_opcode.py -> build/lib.linux-aarch64-cpython-310/cffi copying cffi/error.py -> build/lib.linux-aarch64-cpython-310/cffi copying cffi/api.py -> build/lib.linux-aarch64-cpython-310/cffi copying cffi/lock.py -> build/lib.linux-aarch64-cpython-310/cffi copying cffi/_cffi_include.h -> build/lib.linux-aarch64-cpython-310/cffi copying cffi/parse_c_type.h -> build/lib.linux-aarch64-cpython-310/cffi copying cffi/_embedding.h -> build/lib.linux-aarch64-cpython-310/cffi copying cffi/_cffi_errors.h -> build/lib.linux-aarch64-cpython-310/cffi running build_ext building '_cffi_backend' extension creating build/temp.linux-aarch64-cpython-310 creating build/temp.linux-aarch64-cpython-310/c gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Os -g -O2 -Os -g -O2 -Os -g -O2 -DTHREAD_STACK_SIZE=0x100000 -fPIC -DFFI_BUILDING=1 -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python3.10 -c c/_cffi_backend.c -o build/temp.linux-aarch64-cpython-310/c/_cffi_backend.o error: command 'gcc' failed: No such file or directory [end of output]
note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure
× Encountered error while trying to install package. ╰─> cffi
note: This is an issue with the package mentioned above, not pip. hint: See above for output from the failure.
Il giorno mer 25 gen 2023 alle ore 14:48 dulfer @.***> ha scritto:
Hi gsaviane! Interesting you needed to add those. Although there are some deprecation warnings, the image builds just fine without the dev-packages.
This line should install all the needed libs RUN apk add --no-cache python3 py3-pip && ln -sf python3 /usr/bin/python
What OS/release are you using on your build host?
— Reply to this email directly, view it on GitHub https://github.com/dulfer/dsmr2mqtt/issues/1#issuecomment-1403653526, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC5KCCRNOH45O2323NXV2SLWUEVMLANCNFSM6AAAAAAUDHLJLA . You are receiving this because you authored the thread.Message ID: @.***>
dulfer
commented
1 year ago ARM Alpine image could indeed be the culprit. The x86 images build without problems.
I also noticed the persist function required some tlc, along with a few other bugs. This is my first Python project, learning as I go and did not expect someone other than me to actually use this tbh. So I bodged them and have them stashed in my local repo. Feel free to contribute though 👍.
My goal was to have this script running in a single light-weight container and don't require a hefty database to calculate the periodica consumption/production data. Once I started including metering logic, I quickly realised that it will become a huge mess of variables that will keep history for the various meters. Lots of copying of the same logic to read and write the latest reading, calculate the meter values, do rollovers and persisting the state in the json file in case the container got restarted.
As a result, I am overhauling the entire thing and structuring it in a class Meter, which has properties of custom type 'MeterPeriod' to keep track of the periodic consumption/production values.
class Meter:
def __init__(self, name):
self.name = name
self.hour = MeterPeriod('{name}_hourly', 'H')
self.day = MeterPeriod('{name}_daily', 'D')
self.week = MeterPeriod('{name}_weekly', 'W')
self.month = MeterPeriod('{name}_monthly', 'M')
self.year = MeterPeriod('{name}_yearly', 'Y')
def update_value(self, value):
self.hour.update_value(value)
self.day.update_value(value)
self.week.update_value(value)
self.month.update_value(value)
self.year.update_value(value)MeterPeriod will look something like this:
class MeterPeriod:
def __init__(self, name, period, start_value=0):
self.name = name
self.period = period
self.start_value = start_value
self.counter = 0
self.current_period = self.get_current_period(self)
def get_current_period(self):
date = datetime.now()
period = self.period
if period == 'H':
rollover = date.hour
elif period == 'D':
rollover = date.day
elif period == 'W':
rollover = date.isocalendar()[1]
elif period == 'M':
rollover = date.month
elif period == 'Y':
rollover = date.year
return rollover
def update_value(self, value):
period = self.get_current_period(self)
if self.current_period != period:
self.start_value = value
self.counter = value - self.start_value
def get_value(self):
return round(self.counter, 3)
def get_start_value(self):Once a new telegram comes in, it then only needs to be set to the corresponding meter and all the historical data is automatically updated. At least, that's the theory. Still untested and might not even work.
I probably should create a separate branch for it. Work and family leave very little time to work on this, hence this is still in development and not even ready for testing, let alone ready to push to the repo. Expect this to happen someday though.
gsaviane
commented
1 year ago dulfer, my 2 cents. There were other projects in GitHub for reading from a P1 and pushing into MQTT. The reason I chose yours is that the topics map closer to what a smartgateway device does. Other projects are even lighter, they just rely on systemd and python, no need of any container. But they had their own custom mapping and you cannot find many ready-to-use chart definitions for them (homeassistant or grafana). So far my system is pretty stable. Daily reads are not working because of DatePersisten bug, but for the other things I managed to make it work. One last note: are you sure feeding so many topics where your agent performs differential calculation is it worth the effort? Consider that grafana is quite good in doing derivatives and so on a time series query. For instance, there is no need to have the daily readings if you can query the time series over 1 day buckets and produce the derivative. In my case I just use telegraf to feed time series into influxDB and do the calc in grafana.
Kind regards
Giorgio
Il giorno mer 25 gen 2023 alle ore 17:29 dulfer @.***> ha scritto:
ARM Alpine image could indeed be the culprit. The x86 images build without problems.
I also noticed the persist function required some tlc, along with a few other bugs. This is my first Python project, learning as I go and did not expect someone other than me to actually use this tbh. So I bodged them and have them stashed in my local repo. Feel free to contribute though 👍.
My goal was to have this script running in a single light-weight container and don't require a hefty database to calculate the periodica consumption/production data. Once I started including metering logic, I quickly realised that it will become a huge mess of variables that will keep history for the various meters. Lots of copying of the same logic to read and write the latest reading, calculate the meter values, do rollovers and persisting the state in the json file in case the container got restarted.
As a result, I am overhauling the entire thing and structuring it in a class Meter, which has properties of custom type 'MeterPeriod' to keep track of the periodic consumption/production values.
class Meter:
def __init__(self, name): self.name = name self.hour = MeterPeriod('{name}_hourly', 'H') self.day = MeterPeriod('{name}_daily', 'D') self.week = MeterPeriod('{name}_weekly', 'W') self.month = MeterPeriod('{name}_monthly', 'M') self.year = MeterPeriod('{name}_yearly', 'Y') def update_value(self, value): self.hour.update_value(value) self.day.update_value(value) self.week.update_value(value) self.month.update_value(value) self.year.update_value(value)MeterPeriod will look something like this:
class MeterPeriod:
def __init__(self, name, period, start_value=0): self.name = name self.period = period self.start_value = start_value self.counter = 0 self.current_period = self.get_current_period(self) def get_current_period(self): date = datetime.now() period = self.period if period == 'H': rollover = date.hour elif period == 'D': rollover = date.day elif period == 'W': rollover = date.isocalendar()[1] elif period == 'M': rollover = date.month elif period == 'Y': rollover = date.year return rollover def update_value(self, value): period = self.get_current_period(self) if self.current_period != period: self.start_value = value self.counter = value - self.start_value def get_value(self): return round(self.counter, 3) def get_start_value(self):Once a new telegram comes in, it then only needs to be set to the corresponding meter and all the historical data is automatically updated. At least, that's the theory. Still untested and might not even work.
I probably should create a separate branch for it. Work and family leave very little time to work on this, hence this is still in development and not even ready for testing, let alone ready to push to the repo. Expect this to happen someday though.
— Reply to this email directly, view it on GitHub https://github.com/dulfer/dsmr2mqtt/issues/1#issuecomment-1403888191, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC5KCCXSGQJVCRPXDVHSH33WUFIF7ANCNFSM6AAAAAAUDHLJLA . You are receiving this because you authored the thread.Message ID: @.***>
dulfer
commented
1 year ago Good comment and you are 100% right that other tools, like Grafana can do the calculations using time-series databases. Heck, even the platform I am building this tool for (Home Assistant) is capable of keeping track of this data for me. And those tools are likely way better at it as well.
In this case, I am building something that works well with the Home Assistant DSMR Reader integration. Home Assistant visualises the data and also keeps track of historical values. I want to implement (most of) the sensors/topics defined in https://github.com/home-assistant/core/blob/dev/homeassistant/components/dsmr_reader/definitions.py.
My initial goal was to end with a scaled-down, single container, database less, variant on the DSMR Reader/Data Logger by sanderdw. I should keep is as simple as possible and just have it do the translation (and publishing) from DSMR to MQTT topics. But having it keep track of the basic periodic consumption data and publish it to MQTT will extend its usefulness. I definitely won't be keeping a full database, just the totals per time period and preserving the value at the start of a period to survive crashes and restarts.
🤔Perhaps it could use a switch or env var that signals to omit calculating, publishing and storing this period. That way the script will just convert and publish incoming telegrams to the mqtt broker.
I chose building it into a docker image as it fits the way I roll. For me, there are several benefits of running workloads in containers compared to manually installing and updating. With easy roll-back in case the update comes with unforeseen issues. The overhead of running docker is negligible and the advantages outweigh the drawbacks. In my case, I have this running on a 3-node k8s cluster. Running the Py script directly on the host works fine as well of course, especially if you have a dedicated SBC like a pi available for this.
dulfer
commented
1 year ago I pushed an update to consumption-tracking branch that uses the classes for keeping track of totals as described above.
Including a fix for the bug during data persistence to disk as well as some refactoring.
It also contains env vars that act as switches for publishing simple meter totals to mqtt. If time series data and periodic total calculations are done using another tool, this switch allows for less overhead in messages to the MQTT broker.
Is there anything specific I can add/remove for you?
gsaviane
commented
1 year ago Hello Dulfer, sorry for the late reply, I got some days off. Thank you for asking, however I changed direction and managed to start with a new agent more suited to my use case. As I said, I prefer the agent to not take any responsibility on persistence and run without a container. The agent that I wrote is 100 Python lines in total, that fits better in my resource constrained Raspberry pi. The agent takes only data from stdin as it is fed through socat. In such a way it's even easier to test the module without a serial connection. One thing that you might find useful is to start the mqtt event loop in a separate thread with client.loop_start(). Without it the reconnection mechanism does not work, and if the broker is off for a while, your agent cannot resume automatically once the broker shows up again.
Cheers!
Giorgio
Il giorno ven 3 feb 2023 alle ore 09:56 dulfer @.***> ha scritto:
I pushed an update to consumption-tracking branch that uses the classes for keeping track of totals as described above. Including a fix for the bug during data persistence to disk as well as some refactoring.
It also contains env vars that act as switches for publishing simple meter totals to mqtt. If time series data and periodic total calculations are done using another tool, this switch allows for less overhead in messages to the MQTT broker.
Is there anything specific I can add/remove for you?
— Reply to this email directly, view it on GitHub https://github.com/dulfer/dsmr2mqtt/issues/1#issuecomment-1415391590, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC5KCCSUSH7QKFDRHQ33JVLWVTB2FANCNFSM6AAAAAAUDHLJLA . You are receiving this because you authored the thread.Message ID: @.***>
Hello, I got some errors building the image due to missing packages. After adding the following apk packages
alpine-sdk python3-dev libffi-dev
in Dockerfile
The image built fine (with some error though)