search

mincequi / SBFspot

Yet another tool to read power production of SMA® solar/battery inverters
Other
3 stars 0 forks source link

Cannot compile SBFspot #2

Open rucksman opened 3 years ago

rucksman commented 3 years ago

Continued from https://github.com/SBFspot/SBFspot/issues/50 ...

@mincequi Had to compile cmake because your CMakeLists.txt states cmake_minimum_required(VERSION 3.10) In the raspbian repo only version 3.6.2 is available (I am on jessie).

Probably it is a naming problem? Because looking at my installed packages I see

pi@raspberrypi:~/mincequi/SBFspot/SBFspot/build $ dpkg -l | grep 'msgpack'
ii  libmsgpack-dev                         0.5.7-3                          armhf        binary-based efficient object serialization library
ii  libmsgpack3                            0.5.7-3                          armhf        binary-based efficient object serialization library
ii  libmsgpackc2                           0.5.7-3                          armhf        binary-based efficient object serialization library
pi@raspberrypi:~/mincequi/SBFspot/SBFspot/build $ dpkg -l | grep 'libmosquittopp'
ii  libmosquittopp-dev                     1.3.4-2+deb8u4                   all          MQTT version 3.1 client C++ library, development files
ii  libmosquittopp1                        1.3.4-2+deb8u4                   armhf        MQTT version 3.1 client C++ library

So there is no package "msgpack" (but libmsgpack3) and also no package "libmosquittopp" (but libmosquittopp1).

rucksman commented 3 years ago

So, for interval 60, you have to wait until a full minute is reached

This is what I get at around a full minute:

15:54:57.503     MqttExporter_qt.cpp:90       1| Publishing topic: sbfspot_mitem/live, payload size: 80 bytes
15:54:58.505     MqttExporter_qt.cpp:90       1| Publishing topic: sbfspot_mitem/live, payload size: 80 bytes
15:54:59.502     MqttExporter_qt.cpp:90       1| Publishing topic: sbfspot_mitem/live, payload size: 80 bytes
15:55:00.405          SmaManager.cpp:126   INFO| Polling inverters, timestamp: 1618667700
15:55:00.500     MqttExporter_qt.cpp:90       1| Publishing topic: sbfspot_mitem/live, payload size: 80 bytes
15:55:01.405          SmaManager.cpp:144      1| Polling timed out
15:55:01.406     MqttExporter_qt.cpp:90       1| Publishing topic: sbfspot_mitem/live, payload size: 109 bytes
15:55:01.502     MqttExporter_qt.cpp:90       1| Publishing topic: sbfspot_mitem/live, payload size: 80 bytes
15:55:02.502     MqttExporter_qt.cpp:90       1| Publishing topic: sbfspot_mitem/live, payload size: 80 bytes

No additional data in sbfspot_/live.

Ehm, i could have defined plenty of keys. So, what data are you expecting?

Well, I checked #3 and in the list I can see "Static properties" with keys 1-6 and "Dynamic properties" with keys 8-17. As you can see in my Node Red payload, I only receive 0 (which is always 0), 8-12, 14-15 and 17. That is why I am wondering if I am missing data or not.

mincequi commented 3 years ago 
15:55:01.406     MqttExporter_qt.cpp:90       1| Publishing topic: sbfspot_mitem/live, payload size: 109 bytes
15:55:01.502     MqttExporter_qt.cpp:90       1| Publishing topic: sbfspot_mitem/live, payload size: 80 bytes

This looks actually quite promising. I guess you are now receiving the inverter (109 bytes) AND energy meter data (80 bytes). I will continue to add some properties, but some of the properties will not be exported via the "live" topic, but some "stats" and/or "config" topic. I try to keep the live data as compact as possible. So, i guess, you should now be able to see most relevant data for your devices, correct? :)

I will continue to work on the database a little bit.

rucksman commented 3 years ago

Just noticed that you already made some commits regarding DB. For me personally SQLite ist not of any interest. I use MySQL to store the data. And: What I would also love to see is the possibility to run the software only at certain intervals (eg. EM data every 10 seconds, inverter data every 5 minutes) and/or at certain times (like the original SBFspot as a cronjob). But both of these wishes reflect only my personal needs. It is only you who decides about features. Thanks for your effort!

mincequi commented 3 years ago

The good thing of Qt is, that it abstracts database access as well. So, SQLite and MySQL will share the same code base and once i am finished with SQLite, you get MySQL (PostgreSQL and others) for free :) I already mentioned the inverval settings in SBFspot.cfg, right? There you can set the interval to whatever you want. However, the EM data is not requested. The energy meter simply broadcasts values each second and i just catch those values. Export rate to DB can be set to whatever you like, but MQTT will be updated more often, since it is consideres a "live" exporter without causing disk io. However, i also might rate limit that a little bit.

When i am ready with MySQL, i can give you a ping for beta testing ;)

mincequi commented 3 years ago

@rucksman I have an update with MySQL support. Hopefully, it succeeds to compile on your machine, because i am using recent C++ features.

rucksman commented 3 years ago

Compiled without problems. But after SBFspot_qt -finq -nocsv -v1 -d5 -mqtt

ist says:

SBFspot V3.8.0
Yet another tool to read power production of SMA solar inverters
(c) 2012-2021, SBF (https://github.com/SBFspot/SBFspot)
Compiled for Linux (LE) 32 bit with SQLite support
Commandline Args: -finq -nocsv -v1 -d5 -mqtt
time                            file:line     v|
QSqlDatabase: QMYSQL driver not loaded
QSqlDatabase: available drivers: QSQLITE
Stack trace:
4   0x763aa718 __libc_start_main + 268
3      0x71278 /home/pi/sbfspot_20210501/SBFspot/SBFspot/build/SBFspot_qt() [0x71278]
2      0x5a580 /home/pi/sbfspot_20210501/SBFspot/SBFspot/build/SBFspot_qt() [0x5a580]
1      0x93e04 /home/pi/sbfspot_20210501/SBFspot/SBFspot/build/SBFspot_qt() [0x93e04]
0      0xa03a8 /home/pi/sbfspot_20210501/SBFspot/SBFspot/build/SBFspot_qt() [0xa03a8]
13:13:32.893      SqlExporter_qt.cpp:84    FATL| Driver not loaded Driver not loaded:

Steps to compile did not change? I did "cmake .." and "make" as I did with the older versions.

mincequi commented 3 years ago

It seems, that you do not have the MySQL driver installed: apt install libqt5sql5-mysql should do the trick.

rucksman commented 3 years ago

Installed the driver. But ist still fails because it seems that it does not respect SQL_Hostname in SBFspot.cfg. My database is running on a different machine which is reflected in SBFspot.cfg:

SQL_Hostname=192.168.101.100

Nevertheless I get this error message: Error creating table: CREATE command denied to user 'usernamefromsbfspotcfg'@'raspberrypi.homedomain.local' for table 'LiveDataDc' QMYSQL: Unable to execute query

So the username is coming from SBFspot.cfg, but not the hostname because it tries to reach a database server on the raspberry itself.

mincequi commented 3 years ago

Actually, the hostname is read correctly. Just try to enter a random string, then it will fail hard. It seems, that your user is not allowed to create a database table. As of now, this is mandatory for SBFspot v4, because i use a slightly different DB schema. MySQL simply uses a fully qualified username (so, the host of the user is attached to the username).

rucksman commented 3 years ago

You are absolutely right, I was "multitasking" at the moment when I saw the error message and did not think about it properly. Of course ist ist the hostname from the raspberry and not the target database server. Sorry.

I used the sql statement from SBFspot to create database, tables, views etc. and also the user. As your version is creating tables: is using the sql statements not necessary for your version (creating tables, views etc.)?

CreateMySQLUser.sql does not grant CREATE rights out of the box. After granting CREATE rights, it works.

What I get:

LiveInterval is set to 60, ArchiveInterval is set to 500. So why is there an entry in the database only every 5 minutes, and why are there 2 (different) entries at the same time? Homemanager issue from above?

EDIT: There is no direct relation between what the log says and what I can see in the database. Example:


- entries in the database have the timestamp: 18:10:00, so there is a gap of about 1 minute

I finally realized why there are 2 entries: one is for inverter, the other one for EM.
mincequi commented 3 years ago

I try to make my version as easy as possible to use. That's also why you do not need so specify the inverters by ip. They shall be entirely auto-discovered. Also the database tables will be created "on-demand". So, no need to execute the sql statements to create tables manually.

I guess you mean 300 instead of 500? The "Live" in LiveInterval does not refer to LiveData, but refers to the interval for "live" exporters (like mqtt). Please also read the comments in SBFspot.cfg. So, the 5 minute entry is correct, since it is configured as 300 for ArchiveInterval.

SBFspot also fetches day/month (historical) data. That happens lazily and the timestamps of the logs do not necessarily match the entries in database.

rucksman commented 3 years ago

Understood. "500" was indeed a typo ...

In "LiveInterval" it says: See ProcessInterval how often data shall actually be persisted.

Is this a new variable waiting to be introduced, or is ProcecssInterval=ArchiveInterval?

mincequi commented 3 years ago

Oh, that means: ProcessInterval = ArchiveInterval. One day, i will also update/reimplement config file parsing....

rucksman commented 3 years ago

I do not want to be impatient, just a quick question: any idea about the timeline for the development of your fork?

mincequi commented 3 years ago

Hi @rucksman, there is no timeline for further development. There is just little time and little support. However, I recently finished obtaining archive data using SBFspot v4, which is big step ahead. Is there any feature from your side you would actually need, to use SBFspotv4 in production environment? So, I could continue working on that.

rucksman commented 3 years ago

Hi,

good to hear from you. Sorry for no support from my side, but I have absolutely no clue about C++ ...

Well, for a smooth transition from the original SBFspot to your fork of course it would be great to have identical features. What is missing from your point of view? MySQL support is there, obtaining archive data as well, so it seems there is almost everything convered? What about the data itself? It seems as if this is the biggest difference as you use a totally different database/table scheme, and you do not provide the same amount of data. So it won't be possible to switch from the original SBFspot to your fork. So in a way it is a trade-off for me between less inverter data and additional data from EM ...

deHarro commented 3 years ago

Hi @rucksman , @mincequi ! I followed your discussion with high interest, since I am on reading energy meter and converters (and battery converter) to replace the ever seizing Sunny Portal dashboard by an own (in house) solution.

But I don't have and don't want to use MQTT (as for now :)

Am I able to run SBFspot_qt on its own to display live data in any way? [edit] ... by passing the output of SBFspot_qt to a "program" or a "visualizer" of any sort, obviously. [\edit]

Thanks!

Harry

mincequi commented 3 years ago

Well, for a smooth transition from the original SBFspot to your fork of course it would be great to have identical features. What is missing from your point of view? MySQL support is there, obtaining archive data as well, so it seems there is almost everything convered? What about the data itself? It seems as if this is the biggest difference as you use a totally different database/table scheme, and you do not provide the same amount of data. So it won't be possible to switch from the original SBFspot to your fork. So in a way it is a trade-off for me between less inverter data and additional data from EM ...

Ok, I try to keep the current database scheme as far as possible. There are some things i have to change (or want to change). I will also provide the same data as the original SBFspot. SBFspot computes some data itself which is not necessary and just increases database size, so i remove it. You can check the current status of my fork at #3 . Probably the biggest change is, that SBFspot v4 will be a daemon itself (and not just a command tool).

mincequi commented 3 years ago

I followed your discussion with high interest, since I am on reading energy meter and converters (and battery converter) to replace the ever seizing Sunny Portal dashboard by an own (in house) solution.

But I don't have and don't want to use MQTT (as for now :)

Am I able to run SBFspot_qt on its own to display live data in any way? [edit] ... by passing the output of SBFspot_qt to a "program" or a "visualizer" of any sort, obviously. [\edit]

Hey @deHarro , good to know, that there are more interested persons of my fork :) How do you plan to realize your in-house visualizer? Apparently, we need some kind of IPC and i would go for MQTT (using MessagePack), because it is easy to implement and is widely accepted. What else did you have in mind to implement communication to your visualizer? You can also check the status of my App at #4 once i update this.

deHarro commented 3 years ago

Hi Manuel,

I am in a very early stage and no explicit programmer. I think, I should have all tools on hands to get started before I dive into deep.

My first idea was to use an ESP9266 and attach a display of any sort (BW or color). Secondly I though, ok I already have a RPi controlling the wall box for my E-Scooter (Nico Treffkorn's SmartCharger). So why not using this as base. Now you propose MQTT...

Maybe I should give MQTT a try? So I will start studying... :)

Concerning the technically part of your implementation: Does SBFspot_qt provide all information to make up a display like Sunny Portal? (Just the graphics, no statistics. This part is maintained by Sunny Portal very well).

I want to add another "bar" showing the charging state and value for the wall box. SmartCharger delivers this information as XML output..

So what readings do I need?

Am I missing something?

Thanks! Harry

mincequi commented 3 years ago

Hi Harry, it depends on how "fancy" you want your UI designed. An ESP9266 (or ESP8266?) does not provide much power. So, if you have a RPi available, this sounds much more promising to me and you can implement a very sophisticated UI. Where can i find more info about this SmartCharger you mentioned? So, does the SmartCharger already provide a GUI (which can also be extended)? Or do you want to start your own UI?

And yes, SBFspot_qt can provide all of these data sets (from converters and energy meters, battery and charging state can be added). MQTT is definitely recommended to communicate data for home automation. Many manufacturers already support it out of the box.

rucksman commented 3 years ago

@deHarro You should definitely deal with MQTT, this is a must in the field of home automation from my point of view. MQTT itself is quite simple (at least with JSON, not so much with MessagePack, but that depends on your IT infrastructure) and it is also easy to operate your own MQTT broker (recommended: Mosquitto). A RPi is more than enough for Mosquitto.

rucksman commented 3 years ago

@mincequi Daemonizing SBFspot is not a showstopper for me. More important is kind of compatibility with the original database and amount of data. As soon as there is a new version/new commit I will be happy to compile and test.

deHarro commented 3 years ago

Ok, having read some pages... some steps ahead... MQTT is only the transport layer, no visualization.

The SmartCharger home is located here and here. (in German)

The visualization should more or less resemble the original:

grafik

Between battery and the three other boxes (or perhaps at any other place, fitting better?) I will implement another graph displaying the charging of the car/bike.

The GUI of SmartCHarger is text based only, not fancy in any way:

grafik

mincequi commented 3 years ago

Yes, MQTT serves only as transport to transfer data of one process (or machine) to another one.

There are a lot of ways to realize a visualization. So, it depends what you want to do. You said, you want to drive a display using an ESP9266 or RasPi. So, i assume you do not want to provide an UI via Webinterface (to be shown by any browser)?.

You could have a look at Qt/Qml to implement an UI/Dashboard and orient yourself on material design: https://doc.qt.io/qt-5/qtquickcontrols2-material.html. My App uses this approach and i also wanted so start a similar UI as you intended to do. We might collaborate on that. So, whats your experience with programming?

deHarro commented 3 years ago

The ESP approach with dedicated display only was the very first idea that came to mind since I recently did some projects with those µCs.

During my activities in reading about getting hands on data from the SMA devices, I realized, that the RPi approach is not the dumbest way to go. So at the moment I preferably would want to use the already running RPi as device for the visualization.

If this is the way, then a web based UI is perfectly well for me. At the moment I use the SunnyPortal, also web based, so this isn't a reorientation.

My only wish is, please no line charts with tons of parameters shown at the same time with thin lines of colors hardly to distinguish from another. I'm 63 now and my eyes are getting worse ;) I'm acquainted and somewhat addicted to the SunnyPortal visualization shown above, so I want it to resemble this picture. The WAF (wife acceptance factor) is also important, and my wife uses and understands this presentation, too ;-)

I'm some sort of a hobby programmer. In my times as employee I had several crash courses in C, C++ and C# (but sadly no chance to use the knowledge "in the wild" extensively), some experience with BASIC and am used to Arduino stile programs. WEB programming of any sort is not on my schedule, sadly. If it comes to that, I cobble together what I find in the net.

To get an idea of my capabilities you might want to have a look at my homepage (in German, Google translate option at the bottom of each page).

Collaboration would be highly welcome. Harry

mincequi commented 3 years ago

Ok, C++ sounds good and QML for UI design is pretty simple. You could try to install Qt and get familiar with it: https://www.qt.io/download (Please use latest version from Qt5, Qt6 is not supported yet). Then you can check out my other project at https://github.com/mincequi/Smarf as a base for your UI design. Let me know, if this sounds good to you.

deHarro commented 3 years ago

Yes, sounds good :-) I will start tomorrow with first my steps. And thanks!

deHarro commented 3 years ago

Hi @mincequi! And here are my first questions... Qt open source is the correct version for me? To install it, I have to equip my RPi with keybord, mouse and monitor, right? ("Download Qt online installer"). At the moment I drive my RPi with SSH from my Windows machine.

mincequi commented 3 years ago

You better use your native Windows machine for Development and later compile it for raspberry pi.

Yes, go for the Open Source Version.

deHarro commented 3 years ago

Ok, thanks!

deHarro commented 3 years ago

Huh! My i7 computer is struggling since more than an hour and has reached only 36% of the Qt installtion. Many thanks for the hint to use the "big" computer ;-) Not to think about, how long the RPi would have taken...

mincequi commented 3 years ago

Oh, really? This should not take such a long time....are you compiling it by yourself? The online installer should just download some packages (can be several GBs) and you should be fine...

deHarro commented 3 years ago

It just finished now. Yes, several GB, but the download was not the time consuming task. The "installation" needed the time. And no, I didn't compile it, just used the download option.

I just had a short look at the GUI, browsed the demo apps, clicked on one of it and was told to configure the app... On the search on how to close this one to browse on, the complete GUI shuts down ;-(

At the moment I am feeling lost, no clue how to proceed. (That is always the situation when I come to new software, no need to worry about ;-)

deHarro commented 3 years ago

Here is my first idea on how to display the wanted information:

grafik

Surely, the boxes corners will be rounded and no grid from Excel will be underlying ;) The "V" and "<", ">" will be super duper arrow visualizations, perhaps animated as in SunnyPortal, but that is candy and perhaps obsolete.

mincequi commented 3 years ago

Yes, that looks great and is easily doable in QML.

You can find material design icons for any of your boxes at: https://materialdesignicons.com The flow animation can also be done in QML.

I can try to prepare something for you, if that helps.

How do you get along with Qt/QML and QtCreator?

deHarro commented 3 years ago

Hi Manuel, sounds good.

How do you get along with Qt/QML and QtCreator?

I installed it and staggered around a bit, but "getting along" is something different ;)

I have some other tasks in parallel at the moment and as written, it is not easy for me to get acquainted to a new programming language. So, if you may prepare a starting point for me, that would be very helpful. Thanks!

deHarro commented 3 years ago

Just to keep you posted... The colors of the boxes (have to) change with their values, also associated with each other (not all, but some). E.g. as this:

grafik

or this:

grafik

or this:

grafik

There might be other combinations.

[edit] Changed the pictures to resemble the actual design [\edit]

deHarro commented 3 years ago

First success in QT Creator (Qt Quick) ;)

grafik

And questions... Have I to use "Animation" to implement color changes as shown above in the Excel pictures? Or is it possible to change the background color of a Rectangle by code?

Connnected directly to this problem is my aim to change the color of the "Gesamtverbrauch" tile dependent on the usage of battery power and/or grid power:

How is this accomplished?

And another one: How can I activate/display my icons in the resulting GUI? I placed the icons in QT Creator ...but when executing the project, they are unvisible:

grafik

Thanks!

mincequi commented 3 years ago

Nice, you are getting there :)

Animation is something that you add later. If you want to change colours (of any item), I suggest to not implement that logic in QML, but in C++ instead. Therefore you add some "Model" classes. You can check https://github.com/mincequi/Smarf as an example. Have a look at "ThemeModel" (for colouring) and "InverterModel" (for data).

For your case, I would add a "PowerNodeModel" class and reuse that for each single box, that you have. The properties could be: backgroundColor, foregroundColor, power, remainingBattery,.... For the flows, you can add another model "PowerFlowModel". This holds ALL your flows.

deHarro commented 3 years ago

Nice to read, but hard to understand ;)

Perhaps I should have emphasized on "crash" when mentioning my crash courses of C++ (and the others). Here in Germany we would say something like: Most of your advices are "Böhmische Dörfer" for me :( It translates to "more or less unknown".

But I get to it... hold on!

mincequi commented 3 years ago

Yes, this can be a little awkward to read...but it pays out ;)

Hmm, i am used to hear that. I am from Bavaria, no one understands me north of the "Weißwurstäquator". ;)

You can share your repo and I can add some TODOs with appropriate implementation hints.

deHarro commented 3 years ago

I am from Bavaria, no one understands me north of the "Weißwurstäquator".

This is why we communicate in english ;)

-> Share repo... is what and how? Sorry for my dumb beginners questions :( I work offline, no connection to Github in any way. I download code by ZIP. So how can I share my repo?

Just attach the ZIPped project? PV-Anzeige_1.zip

mincequi commented 3 years ago

No, worries, i appreciate, that you are willing to learn. ;)

Well, just share it via github:

Unfortunately, using git is another step to learn for some serious development. However, then we have our bits and pieces together.

deHarro commented 3 years ago

I made it private since I don't want to "fight" against numerous good will collaborators who have the full knowledge of all ;) You should have gotten an invitation, I suppose.

-> I think we should communicate further over there, right?

mincequi commented 3 years ago

OK, understood. I have the invitation. I will try to contribute, once i find the time.

deHarro commented 3 years ago

Ok, thanks!

deHarro commented 3 years ago

@mincequi Just to be sure, that I did it all right... Did you get my invitation to the dicussion over in my Git? Thanks! Harry

broke23 commented 3 years ago

so, is this working now, sma-energy meter is getting read out alongside sbfspot?

mincequi commented 3 years ago

Yes, that should work.