search

yo8192 / octo2influx

InfluxDB+Grafana Dashboards of your Octopus Energy usage and tariff data
GNU General Public License v3.0
8 stars 2 forks source link

It was working but now I get errors #10

Open cliver1956 opened 1 week ago

cliver1956 commented 1 week ago

Hi, It's me again. Sorry to raise another issue. My previous issue seemed to be resolve as I had it working albeit with a need to get my tariffs right. I only edited the tariffs section in the config and now I get the errors below.... For clarity I am running this in a debian lxc on proxmox. I created a folder octo2influx and copied octo2influx.py, config.yaml and requirements.txt. Ran pip3 install -r requirements.txt and edited config.yaml to my needs. I run octo2influx.py and get the errors. I'm no expert but it seems to infer that the influx url can't be found. I'm running a separate influx instance in proxmox and using the IP address of it which was working before. Any idea where I'm going wrong?

$python3 ./octo2influx.py /usr/lib/python3/dist-packages/requests/init.py:109: RequestsDependencyWarning: urllib3 (2.2.3) or chardet (5.1.0)/charset_normalizer (3.0.1) doesn't match a supported version! warnings.warn( Traceback (most recent call last): File "/root/octo2influx/./octo2influx.py", line 413, in client = InfluxDBClient(url=cfg['influx_url'],


  File "/root/octo2influx/./octo2influx.py", line 130, in __getitem__
    return self.get_validated(key)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/octo2influx/./octo2influx.py", line 119, in get_validated
    value = super().__getitem__(key).get(self.params[key].cfg_type)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/confuse/core.py", line 308, in get
    return templates.as_template(template).value(self, template)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/confuse/templates.py", line 70, in value
    return self.get_default_value(view.name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/confuse/templates.py", line 79, in get_default_value
    raise exceptions.NotFoundError(u"{} not found".format(key_name))
confuse.exceptions.NotFoundError: influx_url not found
cliver1956 commented 1 week ago

Searching for ideas I found a clue for the warning on stack overflow. In ran pip3 install --upgrade requests I now don't get the first 4 lines of the error output above but still get the remaining output from "Traceback (most recent call last). This, to my untrained eye, looks as if it can't find "influx_url" as that would seem to be the first/earliest call? Am I missing something in the config? I am using the IP address of my influx. Also, what is the relevance of the "/autogen" after the bucket name? The relevant section copied here...

influx_org: "myorgname"
influx_bucket: "octopus/autogen"
influx_tariff_measurement: "octopus-tariffs"
influx_usage_measurement: "octopus-usage"
influx_url: "xx.xxx.xxx.xxx:8086" #the IP address of my influxdb2 instance

I'm a bit confused as to why it was working, I edited the tariffs and now can't get it to work at all :(

thibwk commented 1 week ago

It may be it doesn't find and load your config file.

I do get the same error when I put my config file in a wrong place (here I'm running in a docker container using docker-compose):

cto2influx-1  | 2024-11-02 11:24:10 Starting with FREQ=1h...
octo2influx-1  | 2024-11-02 11:24:10 Sleeping short amount of time out of safety...
octo2influx-1  | Traceback (most recent call last):
octo2influx-1  |   File "/usr/src/app/./octo2influx.py", line 413, in <module>
octo2influx-1  |     client = InfluxDBClient(url=cfg['influx_url'],
octo2influx-1  |                                 ~~~^^^^^^^^^^^^^^
octo2influx-1  |   File "/usr/src/app/./octo2influx.py", line 130, in __getitem__
octo2influx-1  |     return self.get_validated(key)
octo2influx-1  |            ^^^^^^^^^^^^^^^^^^^^^^^
octo2influx-1  |   File "/usr/src/app/./octo2influx.py", line 119, in get_validated
octo2influx-1  |     value = super().__getitem__(key).get(self.params[key].cfg_type)
octo2influx-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
octo2influx-1  |   File "/usr/local/lib/python3.12/site-packages/confuse/core.py", line 308, in get
octo2influx-1  |     return templates.as_template(template).value(self, template)
octo2influx-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
octo2influx-1  |   File "/usr/local/lib/python3.12/site-packages/confuse/templates.py", line 70, in value
octo2influx-1  |     return self.get_default_value(view.name)
octo2influx-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
octo2influx-1  |   File "/usr/local/lib/python3.12/site-packages/confuse/templates.py", line 79, in get_default_value
octo2influx-1  |     raise exceptions.NotFoundError(u"{} not found".format(key_name))
octo2influx-1  | confuse.exceptions.NotFoundError: influx_url not found

Arguably it isn't a very clear error message. The config is loaded by the python confuse module: I'll have to see if I can make it clearer that no config file was found and loaded.

It should find your config file if it is in the same directory as the script, and in that case you should see a log message saying so, e.g. see the "Read configuration from" below:

cto2influx-1  | 2024-11-02 11:29:49 Starting with FREQ=1h...
octo2influx-1  | 2024-11-02 11:29:49 Sleeping short amount of time out of safety...
octo2influx-1  | 2024-11-02 11:29:56    INFO octo2influx.py:400: Read configuration from /usr/src/app/config.yaml.
octo2influx-1  | 2024-11-02 11:29:56    INFO octo2influx.py:420: === Retrieving consumption...

Note it doesn't show that "Read configuration" line when reading the config from /etc/octo2influx/config.yaml or ~/.config/octo2influx/config.yaml.

How about you:

cliver1956 commented 1 week ago

Many thanks. I'll look into your suggestions, give them a go and report back.

yo8192 commented 1 week ago

In the meantime I have made the --help message more explicit about config file locations:

The settings can also be set in a config file (./config.yaml,
/etc/octo2influx/config.yaml, ~/.config/octo2influx/config.yaml,
or $OCTO2INFLUXDIR/config.yaml in a directory of your choice by defining
the env var OCTO2INFLUXDIR).

and I have added a check to say more explicitly if the config file wasn't found, e.g.:

Configuration key "price_types" was not found or empty. Please check you have a valid configuration file at one of ['/usr/src/app/config.yaml', '/etc/octo2influx/config.yaml', '/home/octo2influx/.config/octo2influx/config.yaml'].

You may want to do a 'git pull' to get these updates.

cliver1956 commented 1 week ago

Tried again with config.yaml in same directory as the script I got your new 'configuration key' error but first I got a dependency warning as shown... /usr/lib/python3/dist-packages/requests/__init__.py:109: RequestsDependencyWarning: urllib3 (2.2.3) or chardet (5.1.0)/charset_normalizer (3.0.1) doesn't match a supported version! warnings.warn( Configuration key "price_types" was not found or empty. Please check you have a valid configuration file at one of ['/home/clive/octo2influx/src/config.yaml', '/etc/octo2influx/config.yaml', '/home/clive/.config/octo2influx/config.yaml'].

If I move the config file to etc/octo2influx I get a load of error messages but the first one is the same dependency warning ass above. Is that the main culprit? It seemed to install ok when I run requirements.txt.

yo8192 commented 1 week ago

The urllib is suspicious but probably independent from the config-not-found issue.

The script somehow can't find or load your config file. It might be a permission issue.

Could you put the config file in the same folder as the script, i.e. /home/clive/octo2influx/src/config.yaml, and run a: ls -alh /home/clive/octo2influx/src/ And copy the output.

Could you also confirm you run the script as the user clive?

Also just to confirm, do you run the script directly on your machine, or in a docker container, or in a python venv?

On Mon, 4 Nov 2024, 15:26 CliveR, @.***> wrote:

Tried again with config.yaml in same directory as the script I got your new 'configuration key' error but first I got a dependency warning as shown... /usr/lib/python3/dist-packages/requests/init.py:109: RequestsDependencyWarning: urllib3 (2.2.3) or chardet (5.1.0)/charset_normalizer (3.0.1) doesn't match a supported version! warnings.warn( Configuration key "price_types" was not found or empty. Please check you have a valid configuration file at one of ['/home/clive/octo2influx/src/config.yaml', '/etc/octo2influx/config.yaml', '/home/clive/.config/octo2influx/config.yaml'].

If I move the config file to etc/octo2influx I get a load of error messages but the first one is the same dependency warning ass above. Is that the main culprit? It seemed to install ok when I run requirements.txt.

— Reply to this email directly, view it on GitHub https://github.com/yo8192/octo2influx/issues/10#issuecomment-2455009234, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPHBIRG5RG2NSD2ID2TZT3Z66G2DAVCNFSM6AAAAABQ57IGI6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJVGAYDSMRTGQ . You are receiving this because you commented.Message ID: @.***>

cliver1956 commented 1 week ago

Thanks for coming back so quickly.

clive@debian:~/octo2influx$ ls -alh /home/clive/octo2influx/src/ total 68K drwxr-xr-x 2 clive clive 4.0K Nov 4 15:21 . drwxr-xr-x 7 clive clive 4.0K Nov 4 14:33 .. -rw-r--r-- 1 clive clive 6.3K Nov 4 14:33 config.example.yaml -rw-r--r-- 1 clive clive 15K Nov 4 14:48 config.yaml -rw-r--r-- 1 clive clive 375 Nov 4 14:33 Dockerfile -rwxr-xr-x 1 clive clive 622 Nov 4 14:33 loop.sh -rw-r--r-- 1 clive clive 22K Nov 4 14:33 octo2influx.py -rw-r--r-- 1 clive clive 85 Nov 4 14:33 requirements.txt

I'm running it as clive in a debian lxc container running in proxmox ve

yo8192 commented 1 week ago

Ok thanks. The perms and owernship looks fine. The mystery continues... :)

I am not familiar with proxmox, but it sounds it's essentially a VM, which happens to share the kernel with the host, but otherwise looks and behave like a VM?

And the ls you sent me is from inside the VM you run octo2influx in? (just thinking you might be 'mount' that directory as a 'shared disk' -- which seems to be like a docker volume -- in which case you may have different ownership within the LXD container/VM?)

And you are not using a Python venv or docker container from within the LXC container/VM are you?

Is there any chance you could:

  1. paste the output of pip3 list | grep -e confuse -e requests -e urllib from where you run octo2influx from (i.e. within your LXC container/VM, and within any python venv or docker container if you ran one within LXC)
  2. from your LXC container/VM, copy paste the full octo2influx.py command line you use (including the directory you run it on), and its full output of the up and including to the next shell prompt (you gave a full one in the first message, but I don't think you gave a full one since then, e.g. after fixing your requests issue and updating octo2influx). Please copy and paste this into a markdown code (i.e. one line of triple backticks at the beginning, one at the end) so I can see the line breaks correctly.
  3. If possible, could you try running octo2influx maybe directly on your laptop, outside of proxmox / LXC, to rule out any problem with the proxmox / LXC setup? it's fine if you don't have access to influxdb from there, we just want to make sure you no longer have that "config not found" error but a different one.
cliver1956 commented 1 week ago

I am not familiar with proxmox, but it sounds it's essentially a VM, which happens to share the kernel with the host, but otherwise looks and behave like a VM?

I'm no expert in Proxmox either. I wanted to migrate Home Assistant and other things including InfuxDB off Raspberry Pi's so I googled a bit and thought Proxmox sounded ideal so I purchased a mini PC and installed Proxmox VE. It essentially gives you an OS dedicated to running VM's and LXE containers. You can run a terminal connected to each VM or CT individually which is what I'm doing here with the debian CT I created for this. And the ls you sent me is from inside the VM you run octo2influx in? (just thinking you might be 'mount' that directory as a 'shared disk' -- which seems to be like a docker volume -- in which case you may have different ownership within the LXD container/VM?) Yes all run from inside the container.

And you are not using a Python venv or docker container from within the LXC container/VM are you?

No Python was already installed in the debian container I installed pip manually from the command line. Is there any chance you could:

I'll add these in separate posts.....

  1. paste the output of pip3 list | grep -e confuse -e requests -e urllib from where you run octo2influx from (i.e. within your LXC container/VM, and within any python venv or docker container if you ran one within LXC)
  2. from your LXC container/VM, copy paste the full octo2influx.py command line you use (including the directory you run it on), and its full output of the up and including to the next shell prompt (you gave a full one in the first message, but I don't think you gave a full one since then, e.g. after fixing your requests issue and updating octo2influx). Please copy and paste this into a markdown code (i.e. one line of triple backticks at the beginning, one at the end) so I can see the line breaks correctly.
  3. If possible, could you try running octo2influx maybe directly on your laptop, outside of proxmox / LXC, to rule out any problem with the proxmox / LXC setup? it's fine if you don't have access to influxdb from there, we just want to make sure you no longer have that "config not found" error but a different one.
cliver1956 commented 1 week ago
  1. paste the output of pip3 list | grep -e confuse -e requests -e urllib from where you run octo2influx from (i.e. within your LXC container/VM, and within any python venv or docker container if you ran one within LXC)
clive@debian:~/octo2influx$ ls
docker-compose.example.yml  grafana  images  index.html  LICENSE  README.md  src  tests
clive@debian:~/octo2influx$ pip3 list | grep -e confuse -e requests -e urllib
confuse            1.7.0
requests           2.28.1
urllib3            2.2.3
clive@debian:~/octo2influx$
cliver1956 commented 1 week ago

2. from your LXC container/VM, copy paste the full octo2influx.py command line you use (including the directory you run it on), and its full output of the up and including to the next shell prompt (you gave a full one in the first message, but I don't think you gave a full one since then, e.g. after fixing your requests issue and updating octo2influx). Please copy and paste this into a markdown code (i.e. one line of triple backticks at the beginning, one at the end) so I can see the line breaks correctly.

clive@debian:~/octo2influx$ python3 src/octo2influx.py
/usr/lib/python3/dist-packages/requests/__init__.py:109: RequestsDependencyWarning: urllib3 (2.2.3) or chardet (5.1.0)/charset_normalizer (3.0.1) doesn't match a supported version!
  warnings.warn(
Configuration key "price_types" was not found or empty. Please check you have a valid configuration file at one of ['/home/clive/octo2influx/src/config.yaml', '/etc/octo2influx/config.yaml', '/home/clive/.config/octo2influx/config.yaml'].
clive@debian:~/octo2influx$
cliver1956 commented 1 week ago

3. If possible, could you try running octo2influx maybe directly on your laptop, outside of proxmox / LXC, to rule out any problem with the proxmox / LXC setup? it's fine if you don't have access to influxdb from there, we just want to make sure you no longer have that "config not found" error but a different one.

I tried spinning up a raspberry pi and installing it on that but I'm getting errors trying to install from your requirements.txt. If I try to run pip3 install -r src/requirements.txt it throws an error

error: externally-managed-environment

I'll try running it in docker later.