Autodiscover - HowTo

[pesor]

@elecoq129 wrote

one more question now on the autodiscovery configuration. When you say: T_he downloaded folder Autodiscovery, you copy to Home Assistant config folder (where you have your configuration.yaml)__

Do you mean copy the contents of the autodiscovery into config from HomeAssistant or you have to create also the directory structure and the files?

[pesor]

@elecoq129

Actually, the placement of the autodiscover folder, can be of your own choice, as it sadly can not run inside Home-Assistant .

You need to have the sensors folder, and the sensor.yaml in that folder.

In the sensor.yaml, you erase all but the first 4 lines, and a blank 5th line.

You run the TTGO-T-HIGrow-aut.py, as you normally would run a Python script.

I use the EDI - Eric6, that is a fantastic Python EDI.

[pesor]

@elecoq129 Wrote

I assume that You have to run the aut.py script in the win10 machines that runs HomeAssistant? If I manyuall chage the mac information in the sensor.yaml, that would not be enough for HomeAssistant to get the information from the sensor?

[pesor]

The TTGO-T-HIGrow-aut.py script do more than just update the sensors file, it actually initiate the modules into Home-Assistant.

If you change in the sensors.yaml file manually, it will not be registered by the Home-Assistant, and you will not get your information there.

As you started from an earlier release, you already there made the sensors in the configuration.yaml file, you can continue doing that, but it is a pain to maintain all those entities in the configuration.yaml, and you miss a lot of automations issue, which your get via the user interface, when doing the autodiscover.

If you do not have Python installed in your Windows10 you must do it, as with the VSCode and PlatformIO, YouTube have a lot of videos showing how to do it. You do not need programming skills, only the need to run a Python Script from time to time.

The Eric6 EDI, is just like a normal text editor, it just supports Python, and makes it possible for you to run the script within Windows10.

[elecoq129]

I have installed Eric Python, when I execute the script it comes up with the error with missing YAML package. Do you have any idea which one and how to install it? Can't find any tuto how to install it into Eric Python (Win version)?

[elecoq129]

The TTGO-T-HIGrow-aut.py script do more than just update the sensors file, it actually initiate the modules into Home-Assistant.

If you change in the sensors.yaml file manually, it will not be registered by the Home-Assistant, and you will not get your information there.

As you started from an earlier release, you already there made the sensors in the configuration.yaml file, you can continue doing that, but it is a pain to maintain all those entities in the configuration.yaml, and you miss a lot of automations issue, which your get via the user interface, when doing the autodiscover.

If you do not have Python installed in your Windows10 you must do it, as with the VSCode and PlatformIO, YouTube have a lot of videos showing how to do it. You do not need programming skills, only the need to run a Python Script from time to time.

The Eric6 EDI, is just like a normal text editor, it just supports Python, and makes it possible for you to run the script within Windows10.

I have Visual Studio Code and PlatformIO up&running much easier than Arduino...

[elecoq129]

The TTGO-T-HIGrow-aut.py script do more than just update the sensors file, it actually initiate the modules into Home-Assistant. If you change in the sensors.yaml file manually, it will not be registered by the Home-Assistant, and you will not get your information there. As you started from an earlier release, you already there made the sensors in the configuration.yaml file, you can continue doing that, but it is a pain to maintain all those entities in the configuration.yaml, and you miss a lot of automations issue, which your get via the user interface, when doing the autodiscover. If you do not have Python installed in your Windows10 you must do it, as with the VSCode and PlatformIO, YouTube have a lot of videos showing how to do it. You do not need programming skills, only the need to run a Python Script from time to time. The Eric6 EDI, is just like a normal text editor, it just supports Python, and makes it possible for you to run the script within Windows10.

I have Visual Studio Code and PlatformIO up&running much easier than Arduino...

I understand you're fan of Eric's stuff. However, I'm still struggling just to install the environment. Especially the YAML into it. For information I spent hours to try to install Ubunto shell with YAML on the same machine that already runs HA + mqttt under Oracle VM: issue there is a conflict between them. Virtual box is actually locking (at the bios level) the Virtualization capability.

Can you please recommand a URL for a good tuto to install both products on a WIN10 platform? Further more, the idea of my config is to have a dedicated low power consumption 10w mini PC to run all services

[pesor]

@elecoq129 ——— Especially the YAML into it. For information I spent hours to try to install Ubunto shell with YAML on the same machine ——— I really do not understand what you are doing, yaml and Python,ubunto, does not give any meaning to me.

I will send you links to what is necessary, but I can first do it tomorrow.

So hang on, we will make it work.

Per

[elecoq129]

Thanks then I can get some sleep tonight 👍

[pesor]

@elecoq129 @BeardedTinker

In order to get Autodiscovery to work, you must install Python3 on Windows10.

This is the link for for installing Python3 in windows.:

https://phoenixnap.com/kb/how-to-install-python-3-windows

Then you can install Eric6 from this link:

https://sourceforge.net/projects/eric-ide/files/eric6/stable/

You download the zip file, and unpack it to a location of your choise.

You then have to start a CMD (Commando promt) and change directory/folder to where you have unpacked Eric6

You then follow this:

Windows Download the 64-bit variant of Python 3 available at python.org and install it. Extract the downloaded eric distribution archive (see above) to a temporary location and execute the install.py script. (Your directory for the unpacked Eric6) and write python.exe install.py). Note: If some pre-requisite Python packages are not available, the script will ask for permission to install the missing one. Optional: If Qt development is desired, the respective tools need to be available. This requires the installation of the Qt toolkit using these steps.

(for me it installed QT automatically, but if not follow this) Download the Qt online installer from the Qt download site. Install Qt by executing the installer. Configure the path to the Qt tools on the Qt page of the eric configuration dialog. Note: The Qt development tools may be installed with the command C:\Python39\Scripts\python.exe -m pip install qt5-applications instead.

Hope this helps you.

Per

[elecoq129]

thanks Per for your instructions, I was looking for it. I think I will create a new Win VM in order to install those, as I have a bad feeling with my current Windows environment. (it looks like I screw up a bit of things there...) I am desperate to get this last phase up-and-tuning.

Cheers Emmanuel

[pesor]

Glad to be able to help.

This is where @BeardedTinker is much better than me, I mean to get things explained understandablely from the start.

Hope this will make the success for you, otherwise do not hesitate to ask.

Per

[elecoq129]

So I followed all your installation steps above at the exception of Qt, as the package is currently not available for download (major hard disk crash in the cloud, not even the mirror sites are up). But I assume that I don't really need it? I assume from there you execute the script directly from the (windows) Eric app. that has ben installed?

Ok I have finaly managed to run the script after installing missing package (paho-mqtt). 👍

I still have an error message though: TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

[pesor]

Correct, good luck :-D

[elecoq129]

I got an error during executing the script: Line 831: sock.connect(sa)

Ok I have the explanation, for some security reason, I had deleted the Router configuration Port 8123. So it's working. Just need to tweek the HA configuration the get the sensor information readable.

[pesor]

I need to see some output, and where do you get the error, which program?

[elecoq129]

Big thanks to you Per, for your time effort and patience 🥇 very much appreciated Just need to get your next release in order to integrate module DS18b20. I'm really looking forward to it, this means I could go into production

[elecoq129]

After I ran the autodiscovery program, here's what I get: the program has terminated with an exit status 0. Is this OK, as I seems that I do not get anything to the sensors. yaml ? So I'm not able to pick any entities for the sensor?

[pesor]

@elecoq129

This is not correct, the program should not stop,

below is my output when running the program, with all sensors known:.

With unknown sensors, there will be much more information, as it is sent to Home-Assistant.

With an exit 0, something is wrong, what is the output from the program, can you send that?

Output from program:

Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 22:39:24) [MSC v.1916 32 bit (Intel)] on Pers-PC, Standard []

Connected with result code 0 in on_message Tgrow_HIGrow/fcf5c40cf8f4 b'{"plant":{"Tgrow_HIGrow":"fcf5c40cf8f4","sensorname":"Padron_1_winter","date":"21-01","time":"07:49","sleep5Count":0,"bootCount":22,"lux":0,"temp":11.4,"humid":42.8,"soil":93,"salt":92,"saltadvice":"needed","bat":100,"batcharge":"","batchargeDate":"18-01","batchargeDateCnt":20,"battvolt":2368,"battvoltage":4.198213,"wifissid":"Enterprise_EXT","rel":"4.0.6"}}' in read_yaml back in on_message after write in on_message Tgrow_HIGrow/fcf5c40cfd08 b'{"plant":{"Tgrow_HIGrow":"fcf5c40cfd08","sensorname":"Datura_R\xc3\xb8d","date":"21-01","time":"07:25","sleep5Count":0,"bootCount":181,"lux":0,"temp":12,"humid":90,"soil":81,"salt":0,"saltadvice":"needed","bat":57,"batcharge":"","batchargeDate":"29-12","batchargeDateCnt":179,"battvolt":2039,"battvoltage":3.61493,"wifissid":"Enterprise-pro","rel":"4.0.5"}}' in read_yaml back in on_message after write in on_message Tgrow_HIGrow/fcf5c40d4ff4 b'{"plant":{"Tgrow_HIGrow":"fcf5c40d4ff4","sensorname":"Datura_Gul","date":"21-01","time":"08:56","sleep5Count":0,"bootCount":31,"lux":8.333333,"temp":12.2,"humid":90,"soil":51,"salt":0,"saltadvice":"needed","bat":98,"batcharge":"","batchargeDate":"17-01","batchargeDateCnt":27,"battvolt":2332,"battvoltage":4.134388,"wifissid":"Enterprise-pro","rel":"4.0.6"}}' in read_yaml back in on_message after write in on_message Tgrow_HIGrow/246f289e880c b'{"plant":{"Tgrow_HIGrow":"246f289e880c","sensorname":"Begonie_2","date":"18-01","time":"18:13","sleep5Count":0,"bootCount":0,"lux":34.16666,"temp":10.7,"humid":89.5,"soil":97,"salt":4095,"saltadvice":"too high","bat":64,"batcharge":"","batchargeDate":"29-12","batchargeDateCnt":173,"battvolt":2091,"battvoltage":3.707121,"wifissid":"Enterprise_EXT","rel":"4.0.5"}}' in read_yaml back in on_message after write in on_message Tgrow_HIGrow/fcf5c40cfe50 b'{"plant":{"Tgrow_HIGrow":"fcf5c40cfe50","sensorname":"Palme","date":"21-01","time":"09:08","sleep5Count":0,"bootCount":180,"lux":539.1666,"temp":4.9,"humid":87,"soil":97,"salt":0,"saltadvice":"needed","bat":55,"batcharge":"","batchargeDate":"29-12","batchargeDateCnt":178,"battvolt":2026,"battvoltage":3.591883,"wifissid":"Enterprise","rel":"4.0.5"}}' in read_yaml back in on_message after write in on_message Tgrow_HIGrow/246f28b21590 b'{"plant":{"Tgrow_HIGrow":"246f28b21590","sensorname":"Cattleya","date":"19-01","time":"02:49","sleep5Count":0,"bootCount":1,"lux":0,"temp":18.6,"humid":93,"soil":56,"salt":0,"saltadvice":"needed","bat":-25,"batcharge":"","batchargeDate":"15-12","batchargeDateCnt":775,"battvolt":1459,"battvoltage":2.586652,"wifissid":"Enterprise","rel":"4.0.4"}}' in read_yaml back in on_message after write in on_message Tgrow_HIGrow/246f28b21934 b'{"plant":{"Tgrow_HIGrow":"246f28b21934","sensorname":"Begonie_3","date":"20-01","time":"11:57","sleep5Count":0,"bootCount":0,"lux":185.8333,"temp":22.7,"humid":95,"soil":47,"salt":0,"saltadvice":"needed","bat":-26,"batcharge":"","batchargeDate":"04-01","batchargeDateCnt":107,"battvolt":1454,"battvoltage":2.577788,"wifissid":"Enterprise","rel":"4.0.6"}}' in read_yaml back in on_message after write in on_message Tgrow_HIGrow/3c71bff17118 b'{"plant":{"Tgrow_HIGrow":"3c71bff17118","sensorname":"Begonie_1_winter","date":"21-01","time":"07:16","sleep5Count":0,"bootCount":22,"lux":0,"temp":21.1,"humid":10,"soil":2,"salt":0,"saltadvice":"needed","bat":100,"batcharge":"","batchargeDate":"19-01","batchargeDateCnt":11,"battvolt":2359,"battvoltage":4.182256,"wifissid":"Enterprise","rel":"4.0.6"}}' in read_yaml back in on_message after write

[pesor]

Forgot to mention, you must have at least 1 module online, and running, otherwise you may not get any messages, try and reboot the module.

But, this will first work when the program do not exit with code 0, as it should be continuously running, like you see in my output, the program is waiting for a new sensor.

You can stop the program, you only need to run it when you add new modules.

Waiting for your program output

[elecoq129]

this is what I thouhght. I confirm that I had at least 1 sensor up-and-running during executing the script. One more information I had also installed SSL/TLS security module into my HA server in order to secure the connection using port 443. I had also disabled port 8123. Reason why I got a socket error. I'll send you the requested code in a second message.

[pesor]

@elecoq129

A good thing when testing something new, is NOT to use security, as from experience, I know that you can have everything ready and running, and THEN SECURITY sets in, and stops everything. In my work, this was always the issue, we could have testet internally and everything worked, Publish it, and nothing works.

So try to skip SSL at the moment, and see if it then works.

I still want to see you first Error output ;-)

[BeardedTinker]

this is what I thouhght. I confirm that I had at least 1 sensor up-and-running during executing the script. One more information I had also installed SSL/TLS security module into my HA server in order to secure the connection using port 443. I had also disabled port 8123. Reason why I got a socket error. I'll send you the requested code in a second message.

I may say something wrong, but securing HA doesn't influence anything. If you secured MQTT this is different story. I also use SSL internally but MQTT is plain text. Also haven't tested anything in secured (with certificate) MQTT environment - for this, you need to adjust port number (instead of standard 1883) but it should work if SSL is valid (matching domain name with certificate).

[elecoq129]

I was running Malwarebytes that got hacked: https://www.zdnet.com/article/malwarebytes-said-it-was-hacked-by-the-same-group-who-breached-solarwinds/?ftag=TRE-03-10aaa6b&bhid=122160073&mid=13240826&cid=717931507. I really don't like to have something public, even though under test, reason why I brought some security. For mqtt as you sait BeardedTinker, infor are only text, so not a big deal...

[pesor]

Have tou tried so I can see the output, so I can help

[elecoq129]

I'm rebuilding the HA image, has the old one was corrupted, I'll sens you something shortly

[elecoq129]

Ok here we go, everything is back and running. Except the script! One sensor is rebooting every 30s, connecting correctly to the mqtt broker: SSID name: freebox_noyarey.... Connected to network 7C:9E:BD:FB:F2:38 192.168.1.153 IN VINTERTIME Wire ok BH1750 Advanced begin lux 0.00 0 Soil before map: 1776 needed Volt direct 2715 Battery Voltage: 4.81 Writing file: /batinfo.conf

• file written lux 25.83 Creating JSON document... chipId 7c9ebdfbf238 Tgrow_HIGrow/7c9ebdfbf238 freebox_noyarey Sending message to topic: {"plant":{"Tgrow_HIGrow":"7c9ebdfbf238","sensorname":"","date":"21-01","time":"16:32","sleep5Count":0,"bootCount":13,"lux":25.83333,"temp":23.2,"humid":45,"soil":65535,"salt":81,"saltadvice":"needed","bat":100,"batcharge":"","batchargeDate":"19-01","batchargeDateCnt":2212,"battvolt":2715,"battvoltage":4.813406,"wifissid":"freebox_noyarey","rel":"4.0.6"}}
  Attempting to connect to the MQTT broker: morillesdefrance.fr You're connected to the MQTT broker!

Message published successfully

Boot number: 14 Going to sleep... 30 seconds But still facing this error while executing the script:

Exception "unhandled TimeoutError" [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond File: C:\Users\eleco\AppData\Local\Programs\Python\Python39\lib\socket.py, Line: 831

[elecoq129]

error starting line 831:

try: sock = socket(af, socktype, proto) if timeout is not _GLOBAL_DEFAULT_TIMEOUT: sock.settimeout(timeout) if source_address: sock.bind(source_address) sock.connect(sa)

Break explicitly a reference cycle

        err = None
        return sock

[elecoq129]

Let me know what else do you need?

[elecoq129]

Ok please ignore the socket error (I did not copy the sensor.yaml in the right directory). Now I'm back with were I was yesterday with the error returned while executing the script under Eric

[elecoq129]

[elecoq129]

[elecoq129]

As I did not keep an older release of your previous software (version 1.7.2 I think was the latest generation), could you please send me a link that I can download it and use it to register my sensors into HA, the time we fix the current issue. I really need to start to use "them real life" now.

[pesor]

@elecoq129

First the 30 seconds issue, which I do not understand, as I only have the adjustable seconds in the user-variables.h, and then in the go-to-deep-sleep.h in the src folder, it should be 300 seconds, which is 5 minutes, and that only happens when there is an error.

Please check the bottom of the file for the following:

`void goToDeepSleepFiveMinutes() { Serial.print("Going to sleep... "); Serial.print("300"); Serial.println(" sekunder"); if (logging) { writeFile(SPIFFS, "/error.log", "Going to sleep for 300 seconds \n"); }

WiFi.disconnect(true); WiFi.mode(WIFI_OFF); btStop();

adc_power_off(); esp_wifi_stop(); esp_bt_controller_disable();

// Configure the timer to wake us up! ++sleep5no; esp_sleep_enable_timer_wakeup(300 * uS_TO_S_FACTOR);

// Go to sleep! Zzzz esp_deep_sleep_start(); } `

[elecoq129]

:

I confirm the value in the file is 300. I have changed the value to 30 in the user-variables.h for the time of the tests.

[pesor]

@elecoq129

Now to the Python Autodiscovery.

This is how the sensors.yaml file is located:

SENSORS_FILE = "sensors/sensors.yaml"

This means that the file should be in a subfolder in the folder where the TTGO-T-HIGrow-aut.py program is located, if that is not the case, the .yaml file is not found.

I can see that you have placed the autodiscovery file within your home-assistant directory, that I have not, I do not know if it will help, but you should move the autodiscovery folder outside home-assistant, as it has nothing to do with home-assistant, but is a totally independend program.

Your sensors.yaml looks correct.

I am 99.9 percent sure that your problem lies in contacting your MQTT server.

Try to move autodiscovery out of home-assistant, and lets see what we get.

I promise you we will have this up and running tomorrow, I will be available all the time, and this is helping @BeardedTinker in getting his next video correct in all aspect, as what we want, is also to get none IT people to be able to use our modules. //Per

[elecoq129]

:

I confirm the value in the file is 300. I have changed the value to 30 in the user-variables.h for the time of the tests.

I have changed the value back to 3600 to both of my sensors and restarted them

[pesor]

@elecoq129

I confirm the value in the file is 300. I have changed the value to 30 in the user-variables.h for the time of the tests.

:

I confirm the value in the file is 300. I have changed the value to 30 in the user-variables.h for the time of the tests.

I have changed the value back to 3600 to both of my sensors and restarted them

OK, problem solved

[pesor]

@elecoq129

So we are back to AutoDiscovery, return to me when you have moved the Autodiscovery to its own Folder outside home-assistant.

[elecoq129]

@elecoq129

Now to the Python Autodiscovery.

This is how the sensors.yaml file is located:

SENSORS_FILE = "sensors/sensors.yaml"

This means that the file should be in a subfolder in the folder where the TTGO-T-HIGrow-aut.py program is located, if that is not the case, the .yaml file is not found.

I can see that you have placed the autodiscovery file within your home-assistant directory, that I have not, I do not know if it will help, but you should move the autodiscovery folder outside home-assistant, as it has nothing to do with home-assistant, but is a totally independend program.

Your sensors.yaml looks correct.

I am 99.9 percent sure that your problem lies in contacting your MQTT server.

Try to move autodiscovery out of home-assistant, and lets see what we get.

I promise you we will have this up and running tomorrow, I will be available all the time, and this is helping @BeardedTinker in getting his next video correct in all aspect, as what we want, is also to get none IT people to be able to use our modules. //Per

My mistake when I reinstalled the VM, I copied the sensors.yaml file into the same directory of the autodiscovery. One thing we are sure is the mqtt server is working and the sensors are getting connected to it.

I just followed your installation instructions; _I can see that you have placed the autodiscovery file within your home-assistant directory, that I have not, I do not know if it will help, but you should move the autodiscovery folder outside home-assistant, as it has nothing to do with home-assistant, but is a totally independend program.

Your instructions in the main page:

The Python Part - The Autodiscover - MAGIC

The downloaded folder Autodiscovery, you copy to Home Assistant config folder (where you have your configuration.yaml).

[pesor]

@elecoq129

That is why @BeardedTinker is sooo much better in documenting that I.

Sorry for that, it will be corrected.

But I am not sure if that is the problem, I just wanted you to mirror my production system.

The modules works OK now, no problem with them.

The issue is the Python file.

It has to connect to the MQTT server as well, as it reads what the module sends, if it do not connect, it cannot update the sensors.yaml file.

So now we concentrate on the Python program, if I cannot figure out from your input, where the error is, I will make som output statements in my file, and upload it to the repo, and then you have to download it and run it, in order to get more debug information. That is the procedure I do, when debugging on my own system, now we need to debug on yours to find what goes wrong. --- For a programmer - a normal process ---

cheers Per

[elecoq129]

so I have cleared-up Autodiscovery and subfolder from HA server, checked that the directory/files are in my local hard disk. Still getting the same error?

[pesor]

I feared that, I go into the program, and make some print statements.

Return when new .py file is uploaded

[pesor]

@elecoq129

By the way, do you have an iPhone, if that the case go to App-Store and download MQTTool app, it will make it possible for you to see the MQTT messages when they occur.

I think that it may also be on Android, but I do not know, as I am apart from PC, is fully Apple.

[pesor]

@elecoq129

I just got a better idea, as it turns out I have all my debugging statements still in the program.

Look here, this is the start of the program:

`''' TTGO-HIGrow to MQTT Gateway, release 3.0.0 ''' import json import yaml import paho.mqtt.client as mqtt

Configuration File Locations

SENSORS_FILE = "sensors/sensors.yaml"

Write data to YAML file

def write_yaml_file(mac_id, name): print("in write_yaml_file") sensors = dict()

print("after dict")

sensors['mac_id'] = mac_id
sensors['name'] = name

`

Try to everywhere you see a # print, to remove the #, and then run the program, it should give much more information on how the program progresses, and that will help me in finding out what goes wrong at your end.

//Per

[pesor]

Here is it readable

[pesor]

make sure to make the lining correct, the print, must be in line with sensors - print - sensors, there must be no indentation here

[elecoq129]

can you please tell me exactly which program? main.ccp?

[elecoq129]

ok fund it this is within the script

[pesor]

Thats correct, the TTGO module programs are functioning correctly

[elecoq129]

Here's the first one