Excessive devices - Githubissues

matthiaswh commented 2 weeks ago

I've been monitoring for just under a month but have picked up over 2200 unique device IDs.

These all read 0 values for the measurements. Most frequently they record 1 measurement and then disappear, but sometimes they stick around a few seconds to a minute and pick up multiple measurements.

I don't live in a particularly busy area where I would be catching this many neighbors or people walking by with thermometers in their pockets :slightly_smiling_face:

Any idea what might be happening here?

wcbonner commented 2 weeks ago

I've been monitoring for just under a month but have picked up over 2200 unique device IDs.

Any idea what might be happening here?

@matthiaswh Can you let me know what version your code reports?

There was a problem introduced around 3.20240816.0 that caused this. I think it's been fixed since then. It would require manually deleting all of the bad log files and cache files to make it completely forget the false devices.

matthiaswh commented 2 weeks ago

Version 3.20240816.0 Built on: Aug 22 2024 at 17:14:35

I'll try to build a newer version from master to see if that solves it. Thanks!

matthiaswh commented 2 weeks ago

That was the culprit, thanks!

alzuno commented 1 week ago

Hi,

It seems like I'm encountering the same issue. I recently updated to this version: GoveeBTTempLogger Version 3.20240911.0 Built on: Sep 22 2024 at 17:53:44

I have two Govee Thermometers (H5075), and that’s the only information I expect the code to log. However, today I noticed multiple logs from different device IDs, as shown below:

Do you have any recommendations on how to resolve this? Thanks!

wcbonner commented 6 days ago

Hi,

It seems like I'm encountering the same issue. I recently updated to this version: GoveeBTTempLogger Version 3.20240911.0 Built on: Sep 22 2024 at 17:53:44

I have two Govee Thermometers (H5075), and that’s the only information I expect the code to log. However, today I noticed multiple logs from different device IDs, as shown below:

Do you have any recommendations on how to resolve this? Thanks!

@alzuno Thanks for the feedback. I thought I'd squashed this bug. Obviously not. I'll hopefully have a fix in the next day.

alzuno commented 6 days ago

Thanks @wcbonner ! I look forward to the solution. Should you need anything from my side, I’ll be happy to assist.

wcbonner commented 6 days ago

@alzuno I believe that I fixed this. I'm still doing a couple of tests on different machines.

I've introduced a new problem that's going to take a little thinking on how to fix, at least with producing SVG files.

wcbonner commented 6 days ago

I believe that this is fixed again. It was recognizing and logging devices but not creating SVG files. https://github.com/wcbonner/GoveeBTTempLogger/commit/299715784dac4c16ad226a99c3863ba07289995a caught an omission that was making my internal cache structure declare itself invalid, meaning the SVG files were not newer than what had been written in the past.

alzuno commented 5 days ago

Hi @wcbonner, Thank you very much for your help. I've updated to the latest version and removed all the extra files (SVG and TXT) that were generated previously.

It seems the bug related to multiple logs and SVG files has been resolved. However, I've noticed something odd with the SVG generation. The code reads the log files upon startup, but the generated SVGs don't seem to include all the information from the logs.

Here's an example of an SVG file generated after restarting with the new code:

Please let me know what additional information I can provide to help troubleshoot this issue.

Thanks again!

wcbonner commented 5 days ago

Hi @wcbonner, Thank you very much for your help. I've updated to the latest version and removed all the extra files (SVG and TXT) that were generated previously.

It seems the bug related to multiple logs and SVG files has been resolved. However, I've noticed something odd with the SVG generation. The code reads the log files upon startup, but the generated SVGs don't seem to include all the information from the logs.

If you can look at one of your cache files before trying the solution below, I'd be interested in knowing if the last column is a thermometer type or listed as (ThermometerType::Unknown). Especially farther down the page, the uninitialized information was causing me problems yesterday. (The tail command I just ran looking at my own cache says I may still have the underlying problem lurking)

wim@WimPi4-Sola:~ $ head /var/cache/goveebttemplogger/gvh-E35ECC215C0F-cache.txt
Cache: E3:5E:CC:21:5C:0F GoveeBTTempLogger Version 3.20240924.4 Built on: Sep 24 2024 at 21:58:59
1727280567      16.66   0       0       0       16.66   1.79769e+308    1.79769e+308    1.79769e+308    16.66   -1.79769e+308   -1.79769e+308   -1.79769e+308   99.99   1.79769e+308    -1.79769e+308   100       1       (GVH5074)
1727280567      16.7409 0       0       0       16.65   0       0       0       16.85   0       0       0       99.99   99.99   99.99   100     53      (GVH5074)
1727280300      16.9451 0       0       0       16.85   0       0       0       17.04   0       0       0       99.99   99.99   99.99   100     51      (GVH5074)
1727280000      17.1343 0       0       0       17.02   0       0       0       17.24   0       0       0       99.99   99.99   99.99   100     47      (GVH5074)
1727279700      17.2367 0       0       0       17.21   0       0       0       17.28   0       0       0       99.99   99.99   99.99   100     46      (GVH5074)
1727279400      17.1688 0       0       0       17.14   0       0       0       17.2    0       0       0       99.99   99.99   99.99   100     50      (GVH5074)
1727279100      17.0419 0       0       0       16.94   0       0       0       17.14   0       0       0       99.99   99.99   99.99   100     52      (GVH5074)
1727278800      16.9116 0       0       0       16.85   0       0       0       16.98   0       0       0       99.99   99.99   99.99   100     51      (GVH5074)
1727278500      16.8386 0       0       0       16.78   0       0       0       16.89   0       0       0       99.99   99.99   99.99   100     57      (GVH5074)

wim@WimPi4-Sola:~ $ tail /var/cache/goveebttemplogger/gvh-E35ECC215C0F-cache.txt
1647843114      0       0       0       0       1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0       1.79769e+308    -1.79769e+308     2147483647      0       (ThermometerType::Unknown)
1647756714      0       0       0       0       1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0       1.79769e+308    -1.79769e+308     2147483647      0       (ThermometerType::Unknown)
1647670314      0       0       0       0       1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0       1.79769e+308    -1.79769e+308     2147483647      0       (ThermometerType::Unknown)
1647583914      0       0       0       0       1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0       1.79769e+308    -1.79769e+308     2147483647      0       (ThermometerType::Unknown)
1647497514      0       0       0       0       1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0       1.79769e+308    -1.79769e+308     2147483647      0       (ThermometerType::Unknown)
1647411114      0       0       0       0       1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0       1.79769e+308    -1.79769e+308     2147483647      0       (ThermometerType::Unknown)
1647324714      0       0       0       0       1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0       1.79769e+308    -1.79769e+308     2147483647      0       (ThermometerType::Unknown)
1647238314      0       0       0       0       1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0       1.79769e+308    -1.79769e+308     2147483647      0       (ThermometerType::Unknown)
1647151914      0       0       0       0       1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0       1.79769e+308    -1.79769e+308     2147483647      0       (ThermometerType::Unknown)
1596348714      0       0       0       0       1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0       1.79769e+308    -1.79769e+308     2147483647      0       (ThermometerType::Unknown)

@alzuno I think I was running into this problem and was able to clear it by running the following commands:

sudo systemctl stop goveebttemplogger.service
sudo mv /var/cache/goveebttemplogger/gvh-types-cache.txt ~/gvh-types-cache.txt
sudo rm /var/cache/goveebttemplogger/gvh-*.txt
sudo mv ~/gvh-types-cache.txt /var/cache/goveebttemplogger/gvh-types-cache.txt
sudo rm /var/www/html/goveebttemplogger/gvh-*.svg
sudo systemctl start goveebttemplogger.service
journalctl -f -u goveebttemplogger

I temporarily moved the gvh-types-cache.txt file out of the cache directory, because of my poor choice of filename. It was the one data I didn't want to be deleting just now.

The last journalctl command lets you see the progress as it reads in the log files and creates the SVG files.

wcbonner commented 5 days ago

@alzuno I've figured out the underlying problem.

My log files don't record the thermometer type except in the rare condition of the meat thermometers with multiple probes.

A recent change to test validity in the memory structure is if the thermometer is one of the valid types. This was important because of the change in the way data is delivered under BlueZ/DBus than it was using the BlueZ/HCI interface.

If the log files are read before the program has had a chance to recognize the thermometer types all of that data is treated as invalid. I get around this by having the gvh-types-cache.txt file store a mapping between the address and thermometer type. Because of my poor choice of location and naming, it's easy to delete that file. The data will be recreated, but only after the log files have been read and the memory structures established.

I'll be changing the location/naming of the gvh-types-cache.txt file to take care of this issue soon. I believe that the series of commands I posted in the previous message will work around the problem with the existing code.

alzuno commented 5 days ago

Thanks, @wcbonner! Funny thing, I don't seem to have a cache folder on my system.

pi@rasp3:/var/cache $ ls -l
total 32
drwxr-xr-x  3 root     root     4096 Sep 19  2023 apache2
drwxr-xr-x  3 root     root     4096 Sep 25 10:05 apt
drwxr-xr-x  3 root     root     4096 Dec 13  2023 dbconfig-common
drwxr-xr-x  2 root     root     4096 Sep 22 17:45 debconf
drwx------  2 root     root     4096 Sep 22 17:45 ldconfig
drwxr-xr-x 31 man      man      4096 Sep 25 00:00 man
drwx------  2 root     root     4096 May  3  2023 private
drwxr-xr-x  2 www-data www-data 4096 Mar 15  2020 tcpdf

Maybe i need to implement something different? Or do a clean install?

This is my actual configuration file:

sudo systemctl edit goveebttemplogger.service                        
[Service]
Environment="VERBOSITY=0"
Environment="LOGDIR=/var/log/goveebttemplogger"
Environment="TIME=60"
Environment="SVGARGS=--svg /var/www/html/goveebttemplogger/ --battery 8 --minmax 8 --titlemap /var/www/html/goveebttemplogger/gvh-titlemap.txt"
Environment="EXTRAARGS=--download --celsius"

### Lines below this comment will be discarded

### /etc/systemd/system/goveebttemplogger.service
# # Contents of /etc/systemd/system/goveebttemplogger.service
# [Unit]
# Description=GoveeBTTempLogger service
# After=bluetooth.target dbus-org.bluez.service network-online.target
# Requires=bluetooth.target
# StartLimitBurst=5
# StartLimitIntervalSec=33
# 
# [Service]
# Type=simple
# Restart=always
# RestartSec=5
# Environment="VERBOSITY=1"
# Environment="LOGDIR=/var/log/goveebttemplogger"
# Environment="TIME=60"
# Environment="SVGARGS=--svg /var/www/html/goveebttemplogger/ --battery 8 --minmax 8 --titlemap /var/www/html/goveebttemplogger/gvh-titlemap.t>
# Environment="EXTRAARGS=--download"
# ExecStartPre=/bin/mkdir -p ${LOGDIR}
# ExecStart=/usr/local/bin/goveebttemplogger -v ${VERBOSITY} --log ${LOGDIR} --time ${TIME} $SVGARGS $EXTRAARGS
# KillSignal=SIGINT
# 
# [Install]
# WantedBy=multi-user.target

wcbonner commented 5 days ago

@alzuno You've modified your startup from what's in the distribution.. That makes more sense now. I'm not sure what you've set differently from what I'm using.

I believe that this will revert the startup to what's in the goveebttemplogger.service file I'm using. (https://github.com/wcbonner/GoveeBTTempLogger/blob/master/goveebttemplogger.service)

sudo systemctl revert goveebttemplogger.service
sudo systemctl restart goveebttemplogger.service

I'd let it run for about five minutes to build a gvh-types-cache.txt file, then run the commands I mentioned earlier https://github.com/wcbonner/GoveeBTTempLogger/issues/72#issuecomment-2374638011 and it will probably fix itself.

The Cache was important after I'd gathered more than a years worth of data because when it is working properly it allows restarting much quicker than collating all of the old logged data.

alzuno commented 5 days ago

Great, @wcbonner! Thanks for the help.

I didn’t realize the configuration file had been modified. I installed everything over a year ago and hadn’t made many changes since. I just bought a new thermometer and did an update.

I followed the entire process you outlined in #72 and this comment.

Everything is now working as expected, SVG files were created correctly, no more excessive devices on the log or svg folders

Just to update you, I ran both the tail and head commands as requested, before and after the process. Here are the results:

Before:
In the tail, it was listed as (ThermometerType::Unknown).

head /var/cache/goveebttemplogger/gvh-A4C138043FDE-cache.txt
Cache: A4:C1:38:04:3F:DE GoveeBTTempLogger Version 3.20240924.4 Built on: Sep 25 2024 at 09:54:30
1727293481  23.1    0   0   0   23.1    1.79769e+308    1.79769e+308    1.79769e+308    23.1    -1.79769e+308   -1.79769e+308   -1.79769e+308   63.1.79769e+308 -1.79769e+308   50  1   (GVH5075)
1727293481  23.075  0   0   0   23  0   0   0   23.1    0   0   0   63.85   63.8    63.9    50  4   (GVH5075)
1727293200  26.4356 2.66918e-312    1.78287e-312    2.66918e-312    23.1    2.66918e-312    1.78287e-312    4.94066e-324    26.5    2.7152e-312 1.81715e-312.7152e-312  48.2814 48  48.2814 2   59  (GVH5075)
1727292900  26.4356 2.66918e-312    1.78287e-312    2.66918e-312    23.1    2.66918e-312    1.78287e-312    4.94066e-324    26.5    2.7152e-312 1.81715e-312.7152e-312  48.2814 48  48.2814 2   59  (GVH5075)
1727292600  26.4356 2.66918e-312    1.78287e-312    2.66918e-312    23.1    2.66918e-312    1.78287e-312    4.94066e-324    26.5    2.7152e-312 1.81715e-312.7152e-312  48.2814 48  48.2814 2   59  (GVH5075)
1727292300  26.4356 2.66918e-312    1.78287e-312    2.66918e-312    23.1    2.66918e-312    1.78287e-312    4.94066e-324    26.5    2.7152e-312 1.81715e-312.7152e-312  48.2814 48  48.2814 2   59  (GVH5075)
1727292000  26.4356 2.66918e-312    1.78287e-312    2.66918e-312    23.1    2.66918e-312    1.78287e-312    4.94066e-324    26.5    2.7152e-312 1.81715e-312.7152e-312  48.2814 48  48.2814 2   59  (GVH5075)
1727291700  26.4356 2.66918e-312    1.78287e-312    2.66918e-312    23.1    2.66918e-312    1.78287e-312    4.94066e-324    26.5    2.7152e-312 1.81715e-312.7152e-312  48.2814 48  48.2814 2   59  (GVH5075)
1727291400  26.4356 2.66918e-312    1.78287e-312    2.66918e-312    23.1    2.66918e-312    1.78287e-312    4.94066e-324    26.5    2.7152e-312 1.81715e-312.7152e-312  48.2814 48  48.2814 2   59  (GVH5075)

tail /var/cache/goveebttemplogger/gvh-A4C138043FDE-cache.txt
1659209385  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  0   (ThermometerType::Unknown)
1659122985  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  0   (ThermometerType::Unknown)
1659036585  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  0   (ThermometerType::Unknown)
1658950185  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  0   (ThermometerType::Unknown)
1658863785  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  0   (ThermometerType::Unknown)
1658777385  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  0   (ThermometerType::Unknown)
1658690985  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  0   (ThermometerType::Unknown)
1658604585  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  0   (ThermometerType::Unknown)
1658518185  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  0   (ThermometerType::Unknown)
1626290985  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  0   (ThermometerType::Unknown)

After:
In the tail, it is also listed as (ThermometerType::Unknown).

head /var/cache/goveebttemplogger/gvh-A4C138043FDE-cache.txt
Cache: A4:C1:38:04:3F:DE GoveeBTTempLogger Version 3.20240924.4 Built on: Sep 25 2024 at 09:54:30
1727294251  23.2    -27 10  0   23.2    -27 10  0   23.2    -27 10  0   63.563.5    63.5    50  1   (GVH5075)
1727294251  23.1308 -27 10  0   23.1    -27 10  0   23.2    -27 10  0   63.6385 63.4    63.8    50  13  (GVH5075)
1727294100  23.1    -27 10  0   23.1    -27 10  0   23.1    -27 10  0   64.0636 63.7    64.4    50  22  (GVH5075)
1727293800  23.0962 -27 10  0   23  -27 10  0   23.1    -27 10  0   63.9846 63.6    64.4    50  26  (GVH5075)
1727293500  23.0625 -27 10  0   23  -27 10  0   23.1    -27 10  0   63.6875 63.3    64  50  8   (GVH5075)
1727293200  23  -27 10  0   23  -27 10  0   23  -27 10  0   63.563.3    63.8    50  10  (GVH5075)
1727292900  23  -27 10  0   23  -27 10  0   23  -27 10  0   63.6643 63.4    64  50  14  (GVH5075)
1727292600  23  -27 10  0   23  -27 10  0   23  -27 10  0   63.663.4    63.8    50  9   (GVH5075)
1727292300  23  -27 10  0   23  -27 10  0   23  -27 10  0   63.863.6    64  50  6   (GVH5075)

tail /var/cache/goveebttemplogger/gvh-A4C138043FDE-cache.txt
1659209385  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  (ThermometerType::Unknown)
1659122985  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  (ThermometerType::Unknown)
1659036585  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  (ThermometerType::Unknown)
1658950185  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  (ThermometerType::Unknown)
1658863785  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  (ThermometerType::Unknown)
1658777385  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  (ThermometerType::Unknown)
1658690985  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  (ThermometerType::Unknown)
1658604585  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  (ThermometerType::Unknown)
1658518185  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  (ThermometerType::Unknown)
1626290985  0   0   0   0   1.79769e+308    1.79769e+308    1.79769e+308    1.79769e+308    -1.79769e+308   -1.79769e+308   -1.79769e+308   -1.79769e+308   0   1.79769e+308    -1.79769e+308   2147483647  (ThermometerType::Unknown)

Let me know if you need anything else!

wcbonner commented 5 days ago

@alzuno Glad to know it's been working successfully for you over the past year and the latest stuff fixed the issues. I've been playing around with some optimizations as well as fixing the code today and will push another update to GitHub before the day is over. Hopefully I don't break anything else.

All of my other linux services, I've changed so they run as a unique user instead of root and am thinking of doing that with this project as well. The DBus interface doesn't require root access and it seems a better practice.

wcbonner / GoveeBTTempLogger

Excessive devices #72