search

micw / homedatabroker

A middleware that reads, converts and publishes metrics from various sources to various targets
14 stars 3 forks source link

Config not found ... does not start #3

Closed Clanchef closed 1 year ago

Clanchef commented 1 year ago

Hello mciw,

thanks for this great project and I very much appreciate your work!

I just tried it out and created the config as follows for my tibber-pulse:

config.yaml sources:

outputs:

docker-compose.yaml version: "3" services: homedatabroker: image: ghcr.io/micw/homedatabroker:master container_name: homedatabroker restart: always privileged: true network_mode: host volumes:

As I understood homedatabroker replaces tibber-pulse reader and mosquitto so I stopped both of them before starting homedatabroker.

After docker-compose up -d I got the following error, it seems the config file could not be found:

2023-09-02T07:48:32.090+02:00 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application run failed

de.wyraz.homedatabroker.config.ConfigurationException: No such file at de.wyraz.homedatabroker.config.ConfigFileParser.readYamlConfig(ConfigFileParser.java:70) ~[classes!/:1.0.0-SNAPSHOT] at de.wyraz.homedatabroker.config.ConfigFileParser.initialize(ConfigFileParser.java:158) ~[classes!/:1.0.0-SNAPSHOT] at de.wyraz.homedatabroker.config.ConfigFileParser.initialize(ConfigFileParser.java:44) ~[classes!/:1.0.0-SNAPSHOT] at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:606) ~[spring-boot-3.1.0.jar!/:3.1.0] at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:386) ~[spring-boot-3.1.0.jar!/:3.1.0] at org.springframework.boot.SpringApplication.run(SpringApplication.java:310) ~[spring-boot-3.1.0.jar!/:3.1.0] at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:150) ~[spring-boot-3.1.0.jar!/:3.1.0] at de.wyraz.homedatabroker.HomeDataBroker.main(HomeDataBroker.java:23) ~[classes!/:1.0.0-SNAPSHOT] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78) ~[na:na] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na] at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na] at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[homedatabroker.jar:1.0.0-SNAPSHOT] at org.springframework.boot.loader.Launcher.launch(Launcher.java:95) ~[homedatabroker.jar:1.0.0-SNAPSHOT] at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[homedatabroker.jar:1.0.0-SNAPSHOT] at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) ~[homedatabroker.jar:1.0.0-SNAPSHOT]

micw commented 1 year ago

Hello,

config needs to be mounted to /app/config.yaml on docker.

You also need to map the metricst from tibber to MQTT (using "metrics:" config option on mqtt) - but you should get an expressive error message if you use the config from above.

Clanchef commented 1 year ago

Thanks for your quick reply:

changed config to this:

config.yaml: sources:

outputs:

docker-compose.yaml version: "3" services: homedatabroker: image: ghcr.io/micw/homedatabroker:master container_name: homedatabroker restart: always privileged: true network_mode: host volumes:

Now it seems to start:

martin@NUC:~/homedatabroker$ sudo docker logs homedatabroker

. _ _ /\ / '_ () \ \ \ \ ( ( )\ | ' | '| | ' \/ ` | \ \ \ \ \/ _)| |)| | | | | || (| | ) ) ) ) ' |__| .|| ||| |\, | / / / / =========|_|==============|__/=//// :: Spring Boot :: (v3.1.0)

2023-09-02T08:12:09.579+02:00 INFO 1 --- [ main] de.wyraz.homedatabroker.HomeDataBroker : Starting HomeDataBroker v1.0.0-SNAPSHOT using Java 17-ea with PID 1 (/app/homedatabroker.jar started by root in /app) 2023-09-02T08:12:09.584+02:00 INFO 1 --- [ main] de.wyraz.homedatabroker.HomeDataBroker : No active profile set, falling back to 1 default profile: "default" 2023-09-02T08:12:11.255+02:00 INFO 1 --- [ main] d.w.h.output.MQTTOutput.mqtt : Connected to 192.168.178.2 2023-09-02T08:12:11.341+02:00 INFO 1 --- [ main] de.wyraz.homedatabroker.HomeDataBroker : Started HomeDataBroker in 2.416 seconds (process running for 2.974)

But I cannot see MQTT Messages in mqtt explorer?

micw commented 1 year ago

Hello,

the problem is there:

metrics:
  source: dummy.dummy1
  topic: dummy/dummy1

The syntax for "source" is SOURCEID.METRICID. To see the metrics provided by pulse, increase the log level to DEBUG.

Clanchef commented 1 year ago

Can you please give me an example on how to configure this? The topic should be in mosquitto 1ESY1760708474, Source? and how to configure log level to debug?

micw commented 1 year ago

Like that:

            metrics:
              - source: pulse.powerL1
                topic: 1ESY1760708474/L1/power

To set the log level, pass the environment LOG_LEVEL=debug to docker.

I know, documentation needs lot of work ^^

Clanchef commented 1 year ago

:-)

thanks !!!

This worked now for me ... but no data in MQTT Explorer. Even removed user name and password from config ... still got "disconnected". Port 1883?

2023-09-02T09:58:11.707+02:00 INFO 1 --- [ main] de.wyraz.homedatabroker.HomeDataBroker : Starting HomeDataBroker v1.0.0-SNAPSHOT using Java 17-ea with PID 1 (/app/homedatabroker.jar started by root in /app) 2023-09-02T09:58:11.710+02:00 DEBUG 1 --- [ main] de.wyraz.homedatabroker.HomeDataBroker : Running with Spring Boot v3.1.0, Spring v6.0.9 2023-09-02T09:58:11.711+02:00 INFO 1 --- [ main] de.wyraz.homedatabroker.HomeDataBroker : No active profile set, falling back to 1 default profile: "default" 2023-09-02T09:58:13.114+02:00 DEBUG 1 --- [ main] d.w.h.output.MQTTOutput.mqtt : Connecting to 192.168.178.2 2023-09-02T09:58:13.364+02:00 INFO 1 --- [ main] d.w.h.output.MQTTOutput.mqtt : Connected to 192.168.178.2 2023-09-02T09:58:13.459+02:00 INFO 1 --- [ main] de.wyraz.homedatabroker.HomeDataBroker : Started HomeDataBroker in 2.354 seconds (process running for 2.895) 2023-09-02T09:58:14.174+02:00 DEBUG 1 --- [ scheduling-1] d.w.h.metric.MetricRegistry : pulse.energyImportTotal = 15765928.7641 WATT_HOUR 2023-09-02T09:58:14.175+02:00 DEBUG 1 --- [ scheduling-1] d.w.h.metric.MetricRegistry : pulse.energyExportTotal = 34518505.3580 WATT_HOUR 2023-09-02T09:58:14.175+02:00 DEBUG 1 --- [ scheduling-1] d.w.h.metric.MetricRegistry : pulse.energyImportTariff1 = 588.3216 WATT_HOUR 2023-09-02T09:58:14.175+02:00 DEBUG 1 --- [ scheduling-1] d.w.h.metric.MetricRegistry : pulse.energyExportTariff1 = 592.7256 WATT_HOUR 2023-09-02T09:58:14.175+02:00 DEBUG 1 --- [ scheduling-1] d.w.h.metric.MetricRegistry : pulse.energyImportTariff2 = 15765340.4424 WATT_HOUR 2023-09-02T09:58:14.175+02:00 DEBUG 1 --- [ scheduling-1] d.w.h.metric.MetricRegistry : pulse.energyExportTariff2 = 34517912.6324 WATT_HOUR 2023-09-02T09:58:14.175+02:00 DEBUG 1 --- [ scheduling-1] d.w.h.metric.MetricRegistry : pulse.powerTotal = -3215.47 WATT 2023-09-02T09:58:14.175+02:00 DEBUG 1 --- [ scheduling-1] d.w.h.metric.MetricRegistry : pulse.powerL1 = -904.88 WATT 2023-09-02T09:58:14.189+02:00 DEBUG 1 --- [ scheduling-1] d.w.h.metric.MetricRegistry : pulse.powerL2 = -1130.25 WATT 2023-09-02T09:58:14.189+02:00 DEBUG 1 --- [ scheduling-1] d.w.h.metric.MetricRegistry : pulse.powerL3 = -1180.33 WATT

Clanchef commented 1 year ago

Your mqtt is on port 1882 and not 1883 :-) Got it :-)

micw commented 1 year ago

oops ^^

micw commented 1 year ago

Where exactly is 1882 instead of 1883?

Clanchef commented 1 year ago

MQTT explorer sees the topic on port 1882 ... mosquitto is on 1883 per default

micw commented 1 year ago

I don't know what you mean with "sees the topic on port 1882". Is it particular to your setup? I have checked the code, if no port is set, it uses the default port from the mqtt library (which should be 1883)

Clanchef commented 1 year ago

I use MQTT Explorer and connect to port 1883 for mosquitto because of my config and I see the topics. The homedatabroker connects to port 1882 even if mosquitto is not running ... don't know why. I just can see topics

Clanchef commented 1 year ago

Strange ... after I have reconfigured everything to homedatabroker I get no connection to mqtt anymore Logs says everything OK (I see data and power) but no connection possible with MQTT Explorer (disconnected from server) either on port 1882 or 1883

micw commented 1 year ago

MQTT Explorer connects to an MQTT server. So HomeDataBroker does. It is not an MQTT Server, it just sends data to it.

Clanchef commented 1 year ago

Sorry for answering late. This was a misunderstanding. After turning on mosquitto again this works perfect now. But on restart of Mosquitto you need to restart homedatabroker again to make it work. But this is another topic :-)

Thanks for your help, great work again!

This is my config for those who want to migrate:

CONFIG.YAML

sources:
  - id: pulse
    type: tibber-pulse-http
    pulseUrl: http://192.168.178.37/data.json?node_id=3
    pulseUsername: admin
    pulsePassword: ####ownpassword
    cron: '*/2 * * * * *'

outputs:        
  - id: mqtt
    type: mqtt
    mqttHost: 192.168.178.2
#    mqttUsername: mqttuser
#    mqttPassword: S3cr3tPass

    metrics:
      - source: pulse.powerL1
        topic: 1ESY1760708474/powerL1
      - source: pulse.powerL2
        topic: 1ESY1760708474/powerL2
      - source: pulse.powerL3
        topic: 1ESY1760708474/powerL3
      - source: pulse.powerTotal
        topic: 1ESY1760708474/powerTotal
      - source: pulse.energyImportTotal
        topic: 1ESY1760708474/energyImportTotal
      - source: pulse.energyExportTotal
        topic: 1ESY1760708474/energyExportTotal

`DOCKER-COMPOSE`

version: "3"
services:
  homedatabroker:
    image:  ghcr.io/micw/homedatabroker:master
    container_name: homedatabroker
    restart: always
    privileged: true
    network_mode: host
    volumes:
      - /home/martin/homedatabroker/config.yaml:/app/config.yaml
    environment:
      - LOG_LEVEL=debug
Clanchef commented 1 year ago

Closed

micw commented 1 year ago

But on restart of Mosquitto you need to restart homedatabroker again to make it work. But this is another topic :-)

This is work-in-progress, I opened ticket #4 for it. I started to implement an automated test for it using "testcontainers" but could not get it to work yet.

The current implementation uses HiveMQ's MQTT client which has an "auto-reconnect". But seems that it still not solves the issue.

mivola commented 1 week ago

Sorry for answering late. This was a misunderstanding. After turning on mosquitto again this works perfect now. But on restart of Mosquitto you need to restart homedatabroker again to make it work. But this is another topic :-)

Thanks for your help, great work again!

This is my config for those who want to migrate:

CONFIG.YAML

sources:
  - id: pulse
    type: tibber-pulse-http
    pulseUrl: http://192.168.178.37/data.json?node_id=3
    pulseUsername: admin
    pulsePassword: ####ownpassword
    cron: '*/2 * * * * *'

outputs:        
  - id: mqtt
    type: mqtt
    mqttHost: 192.168.178.2
#    mqttUsername: mqttuser
#    mqttPassword: S3cr3tPass

    metrics:
      - source: pulse.powerL1
        topic: 1ESY1760708474/powerL1
      - source: pulse.powerL2
        topic: 1ESY1760708474/powerL2
      - source: pulse.powerL3
        topic: 1ESY1760708474/powerL3
      - source: pulse.powerTotal
        topic: 1ESY1760708474/powerTotal
      - source: pulse.energyImportTotal
        topic: 1ESY1760708474/energyImportTotal
      - source: pulse.energyExportTotal
        topic: 1ESY1760708474/energyExportTotal

`DOCKER-COMPOSE`

version: "3"
services:
  homedatabroker:
    image:  ghcr.io/micw/homedatabroker:master
    container_name: homedatabroker
    restart: always
    privileged: true
    network_mode: host
    volumes:
      - /home/martin/homedatabroker/config.yaml:/app/config.yaml
    environment:
      - LOG_LEVEL=debug

I used this thread and especially the code in this post to setup my replacement for tibber-pulse-reader within 10mins!

Thanks a lot!