search

HestiaPi / hestia-touch-openhab

OpenHAB2 files for HestiaPi Touch model
GNU General Public License v3.0
60 stars 17 forks source link

Enable external sensor #12

Closed gulliverrr closed 7 months ago

gulliverrr commented 4 years ago

Allow external temperature and humidity sensor to override internal sensor readings

rkoshak commented 4 years ago

Is this something you want to be able to do from the UI? If not, the simplest way to do this would be to change the Generic MQTT Thing to subscribe to a topic where the external sensor readings are published. One could even add logic to fall back to the internal sensors if the external sensor stops reporting or loses it's connection to the broker or something.

gulliverrr commented 4 years ago

Is this something you want to be able to do from the UI?

Yes, with a switch. An additional topic for external sensor values to be posted should exist and if the switch is on, the thermostat logic should ignore the internal sensor values and use the external's instead. If sensor goes down (LWT), communication is lost ('expire' after x hours) or no values are present, yes, as you mentioned, it should be fallback to internal readings. I got a few Shelly H&T "golf balls" that seem to work fine so I should be able to add this simple feature soon.

rkoshak commented 4 years ago

It's simple for 1 external sensor. What about for N external sensors? Is that something you are after? N external sensors is actually something I was going to implement for myself as I have just the one thermostat but lots of temp sensors scattered around the house and I'd like to use the max/min of all these temp sensors to drive the HVAC. That's probably too niche to include in the base software though.

BTW, thanks for letting me know about the Shelly H&T. Somehow I missed they released these. I might replace my DIY EPS8266's with those if they are cheap enough. I have Shelly1s deployed and like their approach.

gulliverrr commented 4 years ago

That's probably too niche to include in the base software though.

I'm after exactly this but I don't think I will publish it in something louder than a post in the wiki here rather than properly on GitHub (people get confused with many features - and my inbox gets full).

I have Shelly1s deployed and like their approach.

One little thing I didn't like is that you cannot set the complete topic path but only the id. So H&T publishes temperature here: shellies/[device_ID]/sensor/temperature which OH can handle of course with no problem but messes my naming convention (and my OCD)

rkoshak commented 4 years ago

True. Their "announcement" approach is kind of annoying too. All the devices publish to the same topic and you have to parse the JSON to figure out which device it's from. I posted a tutorial on that on the OH forum to get the version and whether there is an update from them over MQTT.

rkoshak commented 4 years ago

How much responsibility do we want to take for adopting to the many different topics and standards that will be out there for reporting temperature over MQTT? Would it be acceptable to define a standard that HestiaPi accepts and it's an exercise left to the user to publish the external sensors in that way, or modify the HestiaPi configs themselves to subscribe to the right topics.

I'm thinking something along the lines of the following.

hestia/external/[device or room]/temperature and hestia/external/[device or room]/humidity

We can do a wild card subscription to hestia/external and trigger a Rule that triggers when anything is published to and of the topics below that we can get both the topic and the message. The Rule can then keep track of the different devices and deal with calculating the min and max as needed. If/when we move to the NGRE for Rules, we can even automatically create individual Items for these, but for now with Rules DSL I think we will need to keep the info internal to the Rule.

Timers can be used to timeout the reading in case a sensor stops reporting. I'm not sure how to handle this in as generic and dynamic way as possible and process LWT messages for the individual devices. So I think we just keep a given reading for a certain amount of time and forget it if that reading doesn't get replaced with an update.

This will give the users the most flexibility and least amount of work I think. They just need to get their readings published to the right topics and HestiaPi handles the rest. That makes handling multiple external sensors as easy for the user as handling just the one. And it doesn't require any configuration of OH itself to support. They don't even need to flip a switch to turn it on.

But for those with devices like the shellies you mention it may require extra work. This extra work can be done through OH (create a Thing to subscribe to the shellies topic, another to publish to the hestia topic and an Item using the follow profile to forward the message) or through a bridge configuration on mosquitto's config.

I'll put something together and post it to the forum and see if this would be acceptable or if we should only support the one. It might be a week or two though before I get something working.

gulliverrr commented 4 years ago

I posted a tutorial on that on the OH forum to get the version and whether there is an update from them over MQTT.

Is that it?

How much responsibility do we want to take for adopting to the many different topics and standards that will be out there for reporting temperature over MQTT? Would it be acceptable to define a standard that HestiaPi accepts and it's an exercise left to the user to publish the external sensors in that way, or modify the HestiaPi configs themselves to subscribe to the right topics.

I would believe so. I'm sure there would be more 'shellies' out there where they will conflict with 'a' suggested method but our topic would be 'easily' modified to match any external sensor.

We can do a wild card subscription to hestia/external and trigger a Rule that triggers when anything is published to and of the topics below that we can get both the topic and the message.

Great idea. Haven't touched NGRE yet to comment. Here is the forum post for people to comment.

rkoshak commented 4 years ago

Is that it?

I was actually thinking of this one. But that requires OH 2,5.

Great idea. Haven't touched NGRE yet to comment.

I've been pushing it a little hard here, but it really will solve a lot of our problems:

But to be effective, we will need to move to OH 2.5. It shouldn't be that big of a deal though. Based on what I've seen it should upgrade without any problem, but I need some time to dedicate to it in case something does go wrong before I try.

gulliverrr commented 4 years ago

I will give it a go on a Pi 3 first and let you know.

rkoshak commented 4 years ago

There is a very recent (as in the last week) change to the Exec binding that might trip you up. You need to add all the commands used by the exec binding to a whitelist . See https://community.openhab.org/t/openhab-2-5-x-patch-releases/90248/58 and a few posts down Yannick posted a script that will let you generate the white list automatically.

It seems the version of jq installed on Jessey doesn't work the same as what's in that post. Use:

sudo -u openhab curl http://localhost:8080/rest/things | jq -r '.[] | select(.UID | starts

with("exec")) | .configuration.command' | sort | uniq >> exec.whitelist

or just use this as exec.whitelist:

/home/pi/scripts/backup.sh
/home/pi/scripts/getbackuptimestamp.sh
/home/pi/scripts/getBMEhumi.sh
/home/pi/scripts/getBMEtemp.sh
/home/pi/scripts/getcpuload.sh
/home/pi/scripts/getcputemperature.sh
/home/pi/scripts/gethumiditymode.sh
/home/pi/scripts/getpublicip.sh
/home/pi/scripts/getssid.sh
/home/pi/scripts/getsystemtype.sh
/home/pi/scripts/gettempunit.sh
/home/pi/scripts/gettz.sh
/home/pi/scripts/getuptime.sh
/home/pi/scripts/getuseddiskspace.sh
/home/pi/scripts/getwifiinfo.sh
/home/pi/scripts/getwlan0ip.sh
/home/pi/scripts/getwlan0mac.sh
/home/pi/scripts/reboot.sh
/home/pi/scripts/restore.sh &
/home/pi/scripts/shutdown.sh

UPDATE: It seems that there is still a bug with the whitelist. Once OH is started, you need to touch the exec.whitelist file to get OH to read it. This has already been fixed I think but won't be in the default install until the next subpoint release.

Beyond that, the upgrade seems to be working without a hitch. It'll take some time to test everything but I'm not seeing any errors.

gulliverrr commented 4 years ago

For anyone else who want to test waters of openHAB 2.5, here are the steps I followed (needed today):

sudo /bin/systemctl stop openhab2.service
sudo apt-get update
sudo apt-get upgrade
sudo -u openhab mkdir /etc/openhab2/misc
sudo -u openhab touch /etc/openhab2/misc/exec.whitelist
sudo nano  /etc/openhab2/misc/exec.whitelist

Add these lines:

/home/pi/scripts/backup.sh
/home/pi/scripts/getbackuptimestamp.sh
/home/pi/scripts/getBMEhumi.sh
/home/pi/scripts/getBMEtemp.sh
/home/pi/scripts/getcpuload.sh
/home/pi/scripts/getcputemperature.sh
/home/pi/scripts/gethumiditymode.sh
/home/pi/scripts/getpublicip.sh
/home/pi/scripts/getssid.sh
/home/pi/scripts/getsystemtype.sh
/home/pi/scripts/gettempunit.sh
/home/pi/scripts/gettz.sh
/home/pi/scripts/getuptime.sh
/home/pi/scripts/getuseddiskspace.sh
/home/pi/scripts/getwifiinfo.sh
/home/pi/scripts/getwlan0ip.sh
/home/pi/scripts/getwlan0mac.sh
/home/pi/scripts/reboot.sh
/home/pi/scripts/restore.sh
/home/pi/scripts/shutdown.sh

Start OH again: sudo /bin/systemctl start openhab2.service

And after it started, touch the whitelist file (this is the actual bug that should be fixed in next subpoint release) sudo touch /etc/openhab2/misc/exec.whitelist

As the last command has to be run after every start/restart of OH, I added in the /home/pi/scripts/kiosk-xinit.sh the last line:

      # Run unclutter
      unclutter &
      sudo touch /etc/openhab2/misc/exec.whitelist

So it is run on start, but not on restart, so keep that in mind. If you restart OH, run it by hand after OH started: sudo touch /etc/openhab2/misc/exec.whitelist

UPDATE: I still get: 2020-02-26 07:37:53.999 [WARN ] [e.smarthome.model.script.mode-change] - System is still starting, ignoring change to CoolingMode It seems stuck in initializing (which is set to false right after calling the whitelisted scripts, hmmm...)

rkoshak commented 4 years ago

You could add the touch to the initialization Rule. Then it will work for restarts too.

I still get: 2020-02-26 07:37:53.999 [WARN ] [e.smarthome.model.script.mode-change] - System is still starting, ignoring change to CoolingMode It seems stuck in initializing (which is set to false right after calling the whitelisted scripts, hmmm...)

Hmm. I'm not seeing that. Add the following two lines to the end of /var/lib/openhab2/etc/org.ops4j.pax.logging.cfg

log4j2.logger.initialize.name = org.eclipse.smarthome.model.script.initialization
log4j2.logger.initialize.level = DEBUG

This will enable the debug logging for the initialize Rule. That should tell you which line it's getting stuck on or failing on. From there we should be able to figure out what the problem is. I've seen this before when I had a typo in an Item name. It just up and stopped running without error, never completing the initialization Rule.

rkoshak commented 4 years ago

UPDATE: I've been doing some rebooting tests and I've not yet experienced the Initializing Rule not ending, but sometimes it does seem to get stuck for an extended amount of time on "Getting system stats". And I think the cause is that it's still loading and parsing the .rules file twice. See these logs:

2020-02-26 13:35:36.878 [DEBUG] [marthome.model.script.initialization] - FanCtrl is ON
2020-02-26 13:35:37.130 [DEBUG] [marthome.model.script.initialization] - Getting system stats
2020-02-26 13:38:47.287 [WARN ] [smarthome.model.script.temp-setpoint] - System is still starting, ignoring change to PreviousTempReading
2020-02-26 13:38:56.119 [DEBUG] [marthome.model.script.initialization] - Updating PreceMode Items for those that are Boost
2020-02-26 13:39:05.153 [DEBUG] [marthome.model.script.initialization] - Initializing flag cancelled, the system is ready to operate
2020-02-26 13:39:05.235 [DEBUG] [marthome.model.script.initialization] - Cancelling Boost mode where needed
2020-02-26 13:39:05.787 [DEBUG] [marthome.model.script.initialization] - Triggering devices
2020-02-26 13:39:05.988 [INFO ] [marthome.model.script.initialization] - Done initializing settings
2020-02-26 13:39:21.163 [INFO ] [el.core.internal.ModelRepositoryImpl] - Refreshing model 'default.rules'
2020-02-26 13:42:02.252 [INFO ] [lipse.smarthome.io.net.exec.ExecUtil] - executed commandLine '[/usr/bin/touch, /etc/openhab2/misc/exec.whitelist]'
2020-02-26 13:42:02.379 [INFO ] [marthome.model.script.initialization] - Initializing settings
2020-02-26 13:42:03.337 [DEBUG] [marthome.model.script.initialization] - SystemType is US
2020-02-26 13:42:03.907 [DEBUG] [marthome.model.script.initialization] - TempUnit is F
2020-02-26 13:42:04.444 [DEBUG] [marthome.model.script.initialization] - HumidityType is Humidify
2020-02-26 13:42:05.018 [DEBUG] [marthome.model.script.initialization] - MainSwitch is ON
2020-02-26 13:42:05.432 [DEBUG] [marthome.model.script.initialization] - HeatingMode is OFF
2020-02-26 13:42:05.777 [DEBUG] [marthome.model.script.initialization] - HotWaterMode is OFF

Notice it gets to "Getting system stats" and gets stuck for more than three minutes, then it zips through the rest of the Rule and then starts loading it again. Are you seeing this same behavior? Or is it truly stuck forever?

I'm considering outsourcing the initialization of the network and system stats to the Exec binding instead of calling them here anyway. I don't see any reason why initialization should be delayed for these informational pieces of data. We can link a Switch Item to the run channel of the Exec things and just sendCommand to those to kick off the scripts and let the result come in . The Rule needn't wait around for the result. Nothing outside of the LCD really cares about these values.

At a minimum setting the initializing flag to false needs to be moved above those line.

gulliverrr commented 4 years ago

Hmm. I'm not seeing that. Add the following two lines to the end of /var/lib/openhab2/etc/org.ops4j.pax.logging.cfg

It seems it was another timing issue with Pi 3. My Zero W was not doing it. Moving the initializing = false right above the logDebug(initLogName, "Getting system stats") seemed to 'hide' the error.

I'm considering outsourcing the initialization of the network and system stats to the Exec binding instead of calling them here anyway.

SGTM

sudofox commented 4 years ago

Where specifically are you adding the line with the unclutter? I see two calls to it in the script and neither are at the end of the file

gulliverrr commented 4 years ago

This is an experimental approach to find a workaround. The code mentioned is not committed yet.

rkoshak commented 4 years ago

OH 2.5.3 which was released late last week no longer requires the work around. All the Exec binding bugs concerning the exec.whitelist file have been fixed.

I'm going to open a new issue to discuss the way forward for checking in the JSONDB configs and other changes that are not currently configuration controlled.

bwright86 commented 3 years ago

So i'm digesting the new version that is preparing to be released, and I saw this issue discussing one integration i needed. Pulling in multiple readings from other sensors, and merging them together to decide the "house" temp/humi.

I am currently doing this with a Group Avg function, and am feeding the Avg into the TempProxy.

With the Dev-Release on the horizon, is this already integrated into the release, or still pending?

If so, could you provide some pointers to some rules that are doing this? (I am currently digging into the new JSON rules to divine this, but figured i would ask here for future folks.)

rkoshak commented 3 years ago

I've implemented the start of this but it's not yet part of v1.2 and actually expect it will be a v1.3 feature at this point.

At a high level this is what I've done:

From there I had to move on to other things and haven't come back to it yet. Where I was going was:

But all of that was planned out before we set up two setpoints, one for heating and another for cooling. That complicates things slightly as now you can have both heating and cooling enabled at the same time. And, given the temp deltas in my house between the basement and top floor, there could be cases where the min temperature would call for heat and the max temp would call for cooling at the same time.

You can see where I started and left off at https://community.hestiapi.com/t/wip-working-with-external-sensors/1537. I've not had a chance to look at it since I posted that, which was obviously posted before the move to JSONDB rules. It's on my list of things to come back to though.

But your use case is slightly different. And in many ways it's easier to implement. I would do it as follows:

That's it. To support your use case with AVG there are no need to change the Rules. All we need to change is how MyTemp is populated. However, be aware that this will greatly increase the rate at which the Process Sensor Changes rule runs. I imagine it would be possible to have too many external sensors causing a degraded performance in the over all system.

bwright86 commented 3 years ago

Thanks for the response. I will look at your WIP link and see if i can understand what you were doing. The whole two setpoint thing will be interesting, and am curious of the scenario where this applies (Maybe regions where low humidity and seasons with large temp swings)

For what I did, I liked the idea of having the HestiaPi sensor being considered separate, and just added to a group for consideration. This would make it easier to add other sensors into the group and the logic doesn't change (You are correct that the rule could get quite busy with enough sensors.)

For my setup, i am following a similar setup to what you laid out above. I have a main OH instance, and it builds an MQTT bridge to the HestiaPi to pull all the data back for persistence. I reconfigured it to also push the sensor topics back to the HestiaPI, so it could trigger on the changes. This seems to work quite well, and with the Group:Number:Avg Item, it becomes quite easy to add others to it.

I will dig into the new code, and start thinking of ways to handle 2 setpoints and the whole grouping of sensor data. I am still finding a way to explore the JSON data, as I use Powershell primarily for scripting.

gulliverrr commented 3 years ago

The whole two setpoint thing will be interesting, and am curious of the scenario where this applies (Maybe regions where low humidity and seasons with large temp swings)

That is the reason behind it, yes

rkoshak commented 3 years ago

I am still finding a way to explore the JSON data, as I use Powershell primarily for scripting.

For now your best approach will be through PaperUI. This was one of the compromises required to reduce the boot time by 2/3. It also makes it more supportable for those users who might not be that big on coding a way to get to and mess with rules without needing to drop to the terminal.

Your alternatives are to use the REST API to pull the JSON or look at /var/lib/openhab2/jsondb/automation.json. However the problem with these approaches is that the actual JavaScript is munged into one line with \n which makes it all but unreadable.

OH 3 will have a way to look at, export, and import rules to YAML (perhaps others) which will make this a little nicer. But OH 3 is still under development and I'd expect there to be a lag between it's release and HestiaPi's adoption of it.

bwright86 commented 3 years ago

Yes, it is good to have the GUI be able to edit everything, and I was excited to gain the speed boost for booting. (I hadn't realized the updates you've posted on the forum weren't available yet.)

I am able to pull the JSON objects down via Invoke-RestMethod in Powershell. This does a great job of converting the JavaScript back to readable text.

I am still learning how to display the objects and mentally digest them as I explore. Since they are nested objects, I have to figure out how to programatically pull the properties i'm interested in.

I am testing out the new features, and will attempt to inject my temp group Avg into the middle of the logic for Cooling/Heating. But so far, I am liking the improvements.

Soon i'll pull the SD Card from the spare Pi and swap it into my thermostat to start real-world testing.

Cheers!

rkoshak commented 3 years ago

Don't spin your wheels. If you get stuck I'll be happy to answer any questions. At a high level a JSON rule has three sections:

Everything else you see in the JSON is metadata about the rule: name, uid, etc.

hestiahacker commented 7 months ago

Using an external temperature and humidity sensor is possible using the latest image and instructions on how to do so are in the documentation.

As more hardware is obtained and tested, the section on using remote sensors will be expanded to include variations for different makes and models of sensors, however I believe the core thing this ticket was asking for has been achieved.

gulliverrr commented 7 months ago

I agree. Closing...