How can I install this program?

[elmiba]

Hi, I wanted to ask for some advice on how to use your software, because I think this is exactly what I've been looking for!

Regards

[stylpen]

Hi, this project basically has 2 components: a web service implemented as django app and a daemon that gathers and stores data. It started out using sqlite but has since moved to postgresql which is implemented on a separate branch: https://github.com/martinmeinke/rlog/tree/postgres. Since this is what I'm (still) using I'd recommend this branch because all updates and fixes went there. The commit time stamps indicate correctly that this project is no longer actively maintained - though it's still working fine for me ;-) For new installations (especially if publicly accessible from the internet) it would probably be a good idea to invest some effort and move to a still supported version of python and django ...

What you'll need to do is to get postgresql (or another DB at your own risk), python(2), and django running on your computer. I'm using apache with mod_wsgi but for development purposes django itself is fine, too. If everything works out well, django will have created all the tables that the daemon (https://github.com/martinmeinke/rlog/blob/postgres/daemon/rlogd.py) uses. The daemon is accessing the inverters via an RS485 bus. See https://kaco-newenergy.com/index.php?eID=dumpFile&t=f&f=2487&token=612c532f920c22b03c00b10bb48b6b34391f08e7 for advice how to set up the serial bus. Note that you need to configure an ID in the menu of each device so that it knows when to answer a request. If you also have a VSM-102 smart meter (https://asset.conrad.com/media10/add/160267/c1/-/gl/000125439ML01/bedienungsanleitung-125439-voltcraft-vsm-102-drehstromzaehler-digital-80-a-mid-konform-nein-1-st.pdf) you can connect it to another RS485 interface and use the --smart-meter option. If the hardware is set up properly you can start polling the devices using python2 daemon/rlogd.py [--smartmeter] start.

You'll probably have to deal with a lot of software dependency issues - some need to be resolved to get the essential features working but others are required only by optional features like the MQTT part that connects RLog to my home automation system (https://github.com/binarybucks/homA). If you think that you don't need a particular feature which causes an import error or something, it could be best to just comment out a few lines and continue rather than setting everything up.

Unfortunately, I cannot provide a proper setup guide but you are welcome to ask if there is anything I can tell you about my installation that could help yours to get working. My setup is running on a banana-pi (http://www.banana-pi.org/) with Arch Linux.

[elmiba]

Hi, Thanks a lot for your answer! I had kind of lost my hope of getting an answer, but being patient seemd to pay-off.

So I tried for quite some time to find and install al the required packages step by step but I could not get it to run properly on my RPi 3+. The most trouble was trying to get the right versions of django.

So if I understand you correctly, I should just install the postgres version?

If it works you are my saviour because I'm trying to get the Pi running for logging and maybe further extension of sensors connected with an Arduino on the roof. Time will tell.

Thanks a lot again! Regards

[stylpen]

My version of django is still 1.11.27. This is what the postgres branch works with. It is the last one that supports python 2.7. The other branches may expect an even older version. Depending on your distribution it might be hard or even impossible to find django for python2. At least it was removed from the Arch Linux repository: https://security.archlinux.org/package/python2-django. I still have a copy of that package if that helps!

[elmiba]

Thanks! I am using as of today Raspbian and django for Py2.7 is still availible so it should work!

So as I have some time today I will try to get it running! Cheers

[elmiba]

I'm fighting at the moment with the locale settings. I've read the description of this project where it says sudo locale-gen de_DE de_DE.UTF-8

The problem is, it wont change anything...

Do I have to change the code where it sets the locale because I'm useing Raspbian or is it some other fault.

The output from the terminal:

pi@raspberrypi:~/rlog $ python manage.py runserver
Performing system checks...

Unhandled exception in thread started by <function wrapper at 0x74d49fb0>
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/utils/autoreload.py", line 228, in wrapper
    fn(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/runserver.py", line 124, in inner_run
    self.check(display_num_errors=True)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 359, in check
    include_deployment_checks=include_deployment_checks,
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 346, in _run_checks
    return checks.run_checks(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/core/checks/registry.py", line 81, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/usr/local/lib/python2.7/dist-packages/django/core/checks/urls.py", line 16, in check_url_config
    return check_resolver(resolver)
  File "/usr/local/lib/python2.7/dist-packages/django/core/checks/urls.py", line 26, in check_resolver
    return check_method()
  File "/usr/local/lib/python2.7/dist-packages/django/urls/resolvers.py", line 256, in check
    for pattern in self.url_patterns:
  File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python2.7/dist-packages/django/urls/resolvers.py", line 407, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python2.7/dist-packages/django/urls/resolvers.py", line 400, in urlconf_module
    return import_module(self.urlconf_name)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/home/pi/rlog/urls.py", line 12, in <module>
    url(r'^charts/', include('charts.urls')),
  File "/usr/local/lib/python2.7/dist-packages/django/conf/urls/__init__.py", line 50, in include
    urlconf_module = import_module(urlconf_module)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/home/pi/rlog/charts/urls.py", line 4, in <module>
    from charts import views as chart_views
  File "/home/pi/rlog/charts/views.py", line 3, in <module>
    from chart import Chart
  File "/home/pi/rlog/charts/chart.py", line 24, in <module>
    class Chart(object):
  File "/home/pi/rlog/charts/chart.py", line 25, in Chart
    locale.setlocale( locale.LC_ALL, 'de_DE')
  File "/usr/lib/python2.7/locale.py", line 581, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

Thanks!

[elmiba]

I just solved it by entering locale -a into the terminal and then I saw that there's no locale called de_DE as specified in charts.py but only one called de_DE.utf8 so I changed it in charts.py and now I'm one error futher ;-)

[elmiba]

Now the next issue is about authentification, because I think everything should work now, but as I am fairly new to django I would love to get some advice on how to change the general setup of your server. I get this error: django.db.utils.OperationalError: FATAL: Peer-Authentifizierung für Benutzer »stephan« fehlgeschlagen

Luckily I can speek german so that is ok but where does this come from? Do I need to setup something in django? As I have only used django once before I tried to start this in the rlog folder with python manage.py runserver then it completes system check without any error.

Thanks a lot again!

[stylpen]

I'm no django expert either, but the error seems to be related to the database connection. You might just need to change the user name in https://github.com/martinmeinke/rlog/blob/postgres/settings.py#L25 to match your system. Same applies to the daemon script: https://github.com/martinmeinke/rlog/blob/postgres/daemon/rlogd.py#L796 I'm afraid there might be even more cases with hard-coded paths or user names (e.g where coin.mp3 is located a few lines above) but a quick search only revealed https://github.com/martinmeinke/rlog/blob/postgres/wsgi.py as an additional hit.

[elmiba]

So, today I finally got the server somewhat running. (Only the django one) The deamon seams to be running as well. Is there a test I can run or somewhere where I can verify, if the deamon is doing what it should do?

This is how it looks at the moment: (I thik some things are still missing....)

Maybe you could post a picture on how it should look like.

Thanks for your support I really appreciate it!

[elmiba]

After looking at the HTML file of the website, it seems like there should bee some things like a google maps map and some other objects. Some how I won't get anything other than the things shown on the picture above.

When I try to open other pages like the stats for yesterday or similar, it gives me an Error 500.

Here is the output after I try to start the daemon:

pi@raspberrypi:~/rlog/daemon $ sudo python2 rlogd.py start
[2020-05-03 14:56:53.999831+02:00]: RLogDaemon created
[2020-05-03 14:56:54.000565+02:00]: Starting RLog daemon
pi@raspberrypi:~/rlog/daemon $

I think there should be more output. Maybe I'm wrong.

When I type restart instead of start the I get this message:

pi@raspberrypi:~/rlog/daemon $ sudo python2 rlogd.py restart
[2020-05-03 15:01:38.839346+02:00]: RLogDaemon created
[2020-05-03 15:01:38.840229+02:00]: Restarting RLog daemon
pidfile /tmp/rlogd.pid does not exist. Daemon not running?
pi@raspberrypi:~/rlog/daemon $

So I conclude that the deamon is not running.

Things I have changed so far in rlogd.py:

• Path to RS485 device
• some mqtt related stuff because I cant get mosquitto to install correctly because it is migrated to mqtt paho.
• and some mior path changes to the right directory

Where do I need to change the ID as mention in your first post? Mine should be 1 or 01.

[vassilyabramov]

выход хороший

[vassilyabramov]

Упс, я заберу свои слова обратно. Похоже, у тебя еще осталась пара проблем.

[stylpen]

Maybe you could post a picture on how it should look like.

python2 manage.py collectstatic should fetch all the assets like CSS and Javascript files. After that you should be able to see the plots of live data and statistics. There shouldn't be errors in your browser's developer console and new value should be coming in every few seconds in the /charts/live view when rlogd.py is running. Typical log output of the Javascript console looks something like this:

chart.js:184 autoUpdate
chart.js:188 lastTick is: 1588633929000
chart.js:193 newData is :{"timeseries":[{"data":[]},{"data":[]},{"data":[]},{"data":[[1588633938000,"163.500"],[1588633929000,"166.700"]]},{"data":[[1588633938000,"277.300"],[1588633929000,"278.400"]]},{"data":[[1588633938000,"279.300"],[1588633929000,"279.200"]]}]}

The statistics section could for example look like that: (I know that something weird is going on with inverter 1 which is not properly responding to the requests at daytime (thermal issue??) so it is missing in the plots. Total energy production value is still recorded, though, because there seems to be at least one proper response every evening and that is most important information for accounting purposes!)

The google maps feature on the "home" view requires MQTT (over WebSockets) to be set up.

Here is the output after I try to start the daemon I think there should be more output. Maybe I'm wrong.

Rlogd should log something like the following in /var/log/RLogDaemon.log if everything is discovered properly (time stamps omitted for brevity):

daemon running
database connected
Using Parameters:     KWHPERRING: 1.00000     NEXTRING: 1.00000     SOUND: coin.mp3
starting MQTT
Searching for device where the smart meter responds
Checking if /dev/ttyUSB0 exists ...
trying device: /dev/ttyUSB0 as smart meter device
discovering smart meter
Using /dev/ttyUSB0 for smart meter
Searching rs485 device for WR
Checking if /dev/ttyUSB1 exists...
Using device: /dev/ttyUSB1
looking for WR
Running discovery on ID 1
Device 1 answered on type request  *019 3002IN `
Running discovery on ID 2
Device 2 answered on type request  *029 3002IN a
Running discovery on ID 3
Device 3 answered on type request  *039 3002IN b
loading eigenverbrauch
starting normal execution

When I type restart instead of start the I get this message So I conclude that the deamon is not running.

This is a known bug. Have a look the output of ps aux | grep rlogd.py to find out if it is actually running. If you need to restart rlogd I'd suggest to kill it manually and then start it again. (stop isn't working either).

Where do I need to change the ID as mention in your first post? Mine should be 1 or 01.

You can change the device's ID in its menu. Have a look at the manual for information on the menu structure and search for Inverter address. It is described on page 35 in the documentation I've linked above.

[elmiba]

Hi, Thanks again for the great help!

I have used collectstatic before in order to make it possible to show anyting on the website. Sadly it did not help running it again. The website looks still the same as shown in my previous post.

EDIT: Adding pictures of the website: This error shows up on these sites:

• Today
• Last 7 Days
• This Month
• This Year It does not happen with the Custom Period one. It looks like this:

2nd EDIT: I have found that when I plug in some date into the Custom Period fields, I get a cookie related error: (I have tried Chrome Edge and Firefox and all have he same result.)

At the moment I'm fighting with mqtt because mosquito is not availible anymore so I would have to use paho. Due to the fact that I dont have any MQTT devices setup. I have commented out a lot of code in rloged.py

I figured out what the first issue with the deamon was. I had to change/add in rlogd.py the password for the database.

Now I am getting a new error concerning a missing table in postgresql. Or I think that at least.

delpid[2020-05-05 14:09:36.254563+02:00]: daemon running
[2020-05-05 14:09:36.313284+02:00]: database conencted
Traceback (most recent call last):
  File "rlogd.py", line 1228, in <module>
    daemon.start()
  File "/home/pi/rlog/daemon/daemon.py", line 87, in start
    self.run()
  File "rlogd.py", line 806, in run
    self._db_cursor.execute("SELECT * FROM charts_settings WHERE active = TRUE ORDER BY id DESC LIMIT 1;")
psycopg2.ProgrammingError: FEHLER:  Relation »charts_settings« existiert nicht
LINE 1: SELECT * FROM charts_settings WHERE active = TRUE ORDER BY i...
                      ^

The tables at the moment contained in the rlog database are the following ones:

rlog=> \dt
                    Liste der Relationen
 Schema |            Name            |   Typ   | Eigentümer
--------+----------------------------+---------+------------
 public | auth_group                 | Tabelle | pi
 public | auth_group_permissions     | Tabelle | pi
 public | auth_permission            | Tabelle | pi
 public | auth_user                  | Tabelle | pi
 public | auth_user_groups           | Tabelle | pi
 public | auth_user_user_permissions | Tabelle | pi
 public | django_content_type        | Tabelle | pi
 public | django_migrations          | Tabelle | pi
 public | django_session             | Tabelle | pi
 public | django_site                | Tabelle | pi

Is there a SQL file which I have to load into psql manually? By the way I have run the migrate command in django as well, maybe this caused some issue.

Thanks for the info on the ID. I first though you meant that I had to change the ID in the code some where.

[stylpen]

Is there a SQL file which I have to load into psql manually? By the way I have run the migrate command in django as well, maybe this caused some issue.

No, there is no SQL file required. Please try python2 manage.py makemigrations charts followed by python2 manage.py migrate. Afterwards, tables for the charts app should be created and rlogd can use them, too. It should look like this (except with your username as owner):

rlog=# \dt
                     Liste der Relationen
 Schema |             Name             |   Typ   | Eigentümer 
--------+------------------------------+---------+------------
 public | auth_group                   | Tabelle | stephan
 public | auth_group_permissions       | Tabelle | stephan
 public | auth_permission              | Tabelle | stephan
 public | auth_user                    | Tabelle | stephan
 public | auth_user_groups             | Tabelle | stephan
 public | auth_user_user_permissions   | Tabelle | stephan
 public | charts_device                | Tabelle | stephan
 public | charts_eigenverbrauch        | Tabelle | stephan
 public | charts_reward                | Tabelle | stephan
 public | charts_settings              | Tabelle | stephan
 public | charts_smartmeterdailymaxima | Tabelle | stephan
 public | charts_smartmeterentryday    | Tabelle | stephan
 public | charts_smartmeterentryhour   | Tabelle | stephan
 public | charts_smartmeterentryminute | Tabelle | stephan
 public | charts_smartmeterentrymonth  | Tabelle | stephan
 public | charts_smartmeterentrytick   | Tabelle | stephan
 public | charts_smartmeterentryyear   | Tabelle | stephan
 public | charts_solardailymaxima      | Tabelle | stephan
 public | charts_solarentryday         | Tabelle | stephan
 public | charts_solarentryhour        | Tabelle | stephan
 public | charts_solarentryminute      | Tabelle | stephan
 public | charts_solarentrymonth       | Tabelle | stephan
 public | charts_solarentrytick        | Tabelle | stephan
 public | charts_solarentryyear        | Tabelle | stephan
 public | django_admin_log             | Tabelle | stephan
 public | django_content_type          | Tabelle | stephan
 public | django_migrations            | Tabelle | stephan
 public | django_session               | Tabelle | stephan
 public | django_site                  | Tabelle | stephan
(29 Zeilen)

I have used collectstatic before in order to make it possible to show anyting on the website. Sadly it did not help running it again. The website looks still the same as shown in my previous post.

You could try setting DEBUG=True in settings.py. collectstatic should not be necessary in debug mode and additional stack traces are much more helpful than just an error 500.

Due to the fact that I dont have any MQTT devices setup. I have commented out a lot of code in rloged.py

That's fine. MQTT is completely optional and not required if you just want to see the plots in the web interface.

[elmiba]

The data tables are now correctly set up! It worked with your command.

I some how still get only a text version of the website when I run it without debug mode but when I run it with debug turned on the website is shown correctly. I only get one warning, which I still have to check on. The warning is the yellow text on the right:

The deamon is still not running correctly ...

Here is the picture of the website when entering a custom period with some more information but I have not checked on it yet. I want to get the deamon running first maybe it resolves some issues.

[elmiba]

Ok here new post to separate the deamon topic a bit from the website one.

I have successfully started the deamon mor or less. I have added a zero behind the base path to the device, in my case I had used /dev/ttyUSB0 instead of /dev/ttyUSB.

Now it tries to read messages from the inverter and the inverter answers correctly but the deamon says that my answer is too long. Here is the Log of the deamon:

delpid[2020-05-06 12:28:10.785621+02:00]: daemon running
[2020-05-06 12:28:11.211567+02:00]: database conencted
[2020-05-06 12:28:11.363049+02:00]: no settings in database. using defaults
[2020-05-06 12:28:11.363873+02:00]: Using Parameters:     KWHPERRING: 1     NEXTRING: 1     SOUND: /home/pi/rlog/sound/coin.mp3
[2020-05-06 12:28:11.364364+02:00]: starting MQTT
[2020-05-06 12:28:11.364841+02:00]: Searching rs485 device for WR
[2020-05-06 12:28:11.365320+02:00]: Checking if /dev/ttyUSB0 exists...
[2020-05-06 12:28:11.365843+02:00]: Using device: /dev/ttyUSB0
[2020-05-06 12:28:11.368894+02:00]: looking for WR
[2020-05-06 12:28:11.369693+02:00]: Running discovery on ID 1
[2020-05-06 12:28:11.499870+02:00]: Read type message with invalid length. message is:  *019 5500I ^W  length was: 14
[2020-05-06 12:28:12.464261+02:00]: Running discovery on ID 2
[2020-05-06 12:28:14.468750+02:00]: Read type message with invalid length. message is:  length was: 0
[2020-05-06 12:28:17.134450+02:00]: Running discovery on ID 3
[2020-05-06 12:28:19.138336+02:00]: Read type message with invalid length. message is:  length was: 0
[2020-05-06 12:28:21.803851+02:00]: loading eigenverbrauch
[2020-05-06 12:28:21.810591+02:00]: starting normal execution
[2020-05-06 12:28:22.446254+02:00]: integrity error shit is going on: FEHLER:  Einfügen oder Aktualisieren in Tabelle »charts_solarentrytick« verletzt Fremdschlüssel-Constraint »charts_solarentrytick_device_id_66c69a05_fk_charts_device_$
[2020-05-06 12:28:32.365147+02:00]: integrity error shit is going on: FEHLER:  Einfügen oder Aktualisieren in Tabelle »charts_solarentrytick« verletzt Fremdschlüssel-Constraint »charts_solarentrytick_device_id_66c69a05_fk_charts_device_$
[2020-05-06 12:28:42.372116+02:00]: integrity error shit is going on: FEHLER:  Einfügen oder Aktualisieren in Tabelle »charts_solarentrytick« verletzt Fremdschlüssel-Constraint »charts_solarentrytick_device_id_66c69a05_fk_charts_device_$
[2020-05-06 12:28:52.380642+02:00]: integrity error shit is going on: FEHLER:  Einfügen oder Aktualisieren in Tabelle »charts_solarentrytick« verletzt Fremdschlüssel-Constraint »charts_solarentrytick_device_id_66c69a05_fk_charts_device_$
[2020-05-06 12:29:02.387908+02:00]: integrity error shit is going on: FEHLER:  Einfügen oder Aktualisieren in Tabelle »charts_solarentrytick« verletzt Fremdschlüssel-Constraint »charts_solarentrytick_device_id_66c69a05_fk_charts_device_$
$evice_id« DETAIL:  Schlüssel (device_id)=(1) ist nicht in Tabelle »charts_device« vorhanden.

Now it is constantly trying between WR 2 and WR 3...

At least it is running now somewhat ;-)

But I think it exspecting some other inverter. Which inverter do you have? Mine is a Powador 5500i.

I will try to find out where the problem is. But concerning the log output where it is trying to write into the table and fails, I have no clue.

Thanks again!

[stylpen]

Hi,

I have added a zero behind the base path to the device, in my case I had used /dev/ttyUSB0 instead of /dev/ttyUSB.

I don't understand why you had to do this in order to get it working because that because what discover_devices() is supposed do around here: https://github.com/martinmeinke/rlog/blob/postgres/daemon/rlogd.py#L957 Alternatively, you can skip the discovery and pass the --debugInverterPort argument to rlogd.

Now it tries to read messages from the inverter and the inverter answers correctly but the deamon says that my answer is too long. Which inverter do you have?

I've got three Powador 3002 with IDs 1, 2, and 3 on the RS485 bus (see the log in my previous answer https://github.com/martinmeinke/rlog/issues/48#issuecomment-623728802). A valid type message of these device is 15 bytes long. You might have to adjust the type_valid() methods of the WR class to match your device. Possibly data messages of your inverter look differently, too, so you might have to adopt data_valid() and possibly other bits and pieces as well.

Now it is constantly trying between WR 2 and WR 3...

Since there are 3 devices on my bus (one for each phase) the default is to look for all of them. Therefore, the --participants parameter has a default value of 3. Set it to 1 in your setup.

But concerning the log output where it is trying to write into the table and fails, I have no clue.

Looking at DETAIL: Schlüssel (device_id)=(1) ist nicht in Tabelle »charts_device« vorhanden., my guess is that the foreign key issue is a consequential error of findWRs() not working properly with your device type. It should have inserted the inverter's ID into charts_device table after successful discovery. For reference, this is how it looks in my setup:

rlog=# select * from charts_device;
 id | model  
----+--------
  1 | 3002IN
  2 | 3002IN
  3 | 3002IN
(3 Zeilen)

I some how still get only a text version of the website when I run it without debug mode but when I run it with debug turned on the website is shown correctly.

The documentation has a paragraph on static files with runserver: https://docs.djangoproject.com/en/3.0/howto/static-files/#serving-static-files-during-development. I run the web interface on apache using mod_wsgi (or ages ago with runserver always in debug mode) so there might be some work to be done to use runserver without debug. The yellow warning is not your fault but the app uses a deprecated django feature. I'm happy to accept your pull request if you want to work on it. I'm not sure why you get the CSRF cookie error. I'd suggest that you check everything according to the hints on the error page.

Good luck!

[elmiba]

Thanks a lot for all of your help!!!

I have now figured everything out.

The documentation has a paragraph on static files with runserver: https://docs.djangoproject.com/en/3.0/howto/static-files/#serving-static-files-during-development. I run the web interface on apache using mod_wsgi (or ages ago with runserver always in debug mode) so there might be some work to be done to use runserver without debug.

That worked out for me. I'm still using this at the moment because I did not get the apachie mod_wsgi to run correctly by today.

I'm not sure why you get the CSRF cookie error. I'd suggest that you check everything according to the hints on the error page.

This one was also resolved quite easily. The CSRF check force parameter was set to True and had to be turned off.

I have it now running since two days and it works like a charm. Do you know by any chance where I can remove the smartmeter outputs on the "live" chart?

[stylpen]

I have now figured everything out.

Nice!

Do you know by any chance where I can remove the smartmeter outputs on the "live" chart?

Try following the ticksSM variable at https://github.com/martinmeinke/rlog/blob/postgres/charts/views.py#L45 https://github.com/martinmeinke/rlog/blob/postgres/charts/views.py#L58 (and optionally for the statistics page) https://github.com/martinmeinke/rlog/blob/postgres/charts/views.py#L173. I think it should be enough to comment these lines and all occurrences of the variable or you could introduce a parameter in the settings to (dis|en)able smart meter plotting and act accordingly in those places.