Problems with Middleware

[RaptorSDS]

Hello team

I get despite all the instructions on wiki.volkszaehler not a single version to transfer the data from vzlogger into the DB again and again I face the problem invalid UUID or DB invalid I have tried it on 2 linux boxes , Rk3229 TV-Box with Debian Buster and Kernel 5.10 and PHp7.3 and nginx Rasberry 3b with dietpi-image ( latest) Buster lite and the Volkszaehler image I have several servers running ->blynk , owncloud , windows server with WSUS , Pihole etc i can handle linux but still i despair with this DB and middleware problem

can you please a) write the instructions differently and adjust the install script if necessary and / or b) similar to the VZlogger .log also a log file for the middleware with more than just the html request from the PHP log ( I miss information if middleware accesses the DB at all and if so on which ) and / or c) also create a Docker image for armhf instead of just arm64

Translated with www.DeepL.com/Translator (free version)

Hallo Team

ich bekomme trotz aller anleitung auf wiki.volkszaehler keine einzige version dazu die daten von vzlogger in die DB zu übernehmen immer wieder stehe ich vor dem Problem ungültige UUID oder DB ungültig Ich habe es auf 2 linux boxen probiert , Rk3229 TV-Box mit Debian Buster und Kernel 5.10 sowie PHp7.3 und nginx Rasberry 3b mit dietpi-image ( latest) Buster lite und dem Volkszaehler image Ich habe verschiedene Server am laufen ->blynk , owncloud , windows server mit WSUS , Pihole etc ich kann mit linux umgehen aber trotzdem verzweifel ich bei diesem DB und middleware problem

könnt ihr bitte a) die Anleitung anders Schreiben und ggf das install script anpassen und / oder b) ähnlich dem VZlogger .log auch eine log datei für die middleware mit mehr als nur den html request aus dem PHP log ( ich vermisse information ob middleware überhaupt auf die DB zugreift und wenn ja auf welche ) und / oder c) auch ein Docker image für armhf erstellen statt nur arm64

[r00t-]

your issue report does not contain ANY technical information (configurations, logfiles, ...). (it also appears to be a support request, which should go to the -users mailing list, but even for that it's useless.) you are referring to some documentation that you consider inadequate, but don't even say which documentation you read, or what you did according to it. what exactly do you expect us to do with this?

[RaptorSDS]

soory if my question was inadequate but after about 6 day of searching and testing i am frustrate

i used https://wiki.volkszaehler.org/software/middleware/installation[https://wiki.volkszaehler.org/software/middleware/installation](url) script install and config nginx as proxy aand ppm as server both UUID are config in as channel in frontend

in my opinion this guide should be a good start to equip a fresh installation with a middleware and database and to get it running my vzlogger log

[Jun 02 20:29:21][main] vzlogger v0.8.0 based on heads/master-0-g8e8b515573 from Wed, 21 Apr 2021 00:49:45 +0200 started. [Jun 02 20:29:21][main] log level is 3 [Jun 02 20:29:22][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:22][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:23][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:23][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:24][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:24][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:25][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:25][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:26][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:26][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:27][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:27][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:28][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:28][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:29][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:29][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:30][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:30][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:31][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:31][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:32][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:32][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:33][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:33][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:34][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:34][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:35][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:35][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:36][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:36][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:37][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:37][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:38][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:38][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:39][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:39][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:40][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:40][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:41][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:41][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:42][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb'' [Jun 02 20:29:42][chn0] CURL Error from middleware: 'Exception': 'No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa'' [Jun 02 20:29:43][chn1] CURL Error from middleware: 'Exception': 'No entity with UUID 'eed2da70-c30c-11eb-a6ce-5903bd9741bb''

my config

`"retry": 0, "verbosity": 3, "log": "/var/log/vzlogger.log",

"local": { "enabled": false, "port": 8082, "index": true, "timeout": 0, "buffer": 0 }, "meters": [ { "enabled": true, "allowskip": false, "interval": -1, "aggtime": -1, "aggfixedinterval": true, "channels": [ { "api": "volkszaehler", "uuid": "c8b1a070-c30c-11eb-a500-b110c2dbffaa", "identifier": "1-0:1.8.0", "middleware": "http://localhost/api", "aggmode": "none", "duplicates": 0 }, { "api": "volkszaehler", "uuid": "eed2da70-c30c-11eb-a6ce-5903bd9741bb", "identifier": "1-0:2.8.0", "middleware": "http://localhost/api", "aggmode": "none", "duplicates": 0 } ], "protocol": "sml", "device": "/dev/ttyUSB0", "baudrate": 9600, "parity": "8n1", "use_local_time": true`

also use https://wiki.volkszaehler.org/howto/debug but there are missing info where is the access.log , how i can check db -> has middleware access to db , its look like frontend do not write the uuid inside db -> its that frontend problem or middleware ?

my wish or request is a better log file for middleware also some more infos on wiki --> maybe link from install to debug, some more infos about middleware to db access, more infos where are log of db , middleware, frontend

it was for me more a github problem because i also use the ready rpi-image and have same problem

[J-A-U]

Please test these URLs against your installation (examples are towards Demoserver, 2&3 for comparison) and post the answers 1) https://demo.volkszaehler.org:8080/capabilities/database.json 2) https://demo.volkszaehler.org/api/capabilities/database.json 3) https://demo.volkszaehler.org/middleware/channel/c8b1a070-c30c-11eb-a500-b110c2dbffaa.json

[RaptorSDS]

thanks for answer

1. and 2. localhost:8080/... and localhost/api/. .. show my middleware this
   An exception occurred while executing 'EXPLAIN SELECT COUNT(id) FROM data USE INDEX (PRIMARY)':\n\nSQLSTATE[42S22]: Column not found: 1054 Unknown column 'id' in 'field list'" and InvalidFieldNameException

for 3. now that point to the error localhost/middleware......dffaa.json show No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa but your link to the demo.volkszahler.org show uuid is there on the demo not localhost -->conclusion my frontend build the uuid to demo.volkszaehler.org and not localhost

i check option.js on this build and also rpi_image both config right with this title: 'Local (default)', url: ''

in config.js are also a secound entry with url:"demo....." is also to change ?

[J-A-U]

Against what type of installation/hardware are we actually testing?

An exception occurred while executing 'EXPLAIN SELECT COUNT(id) FROM data USE INDEX (PRIMARY)':\n\nSQLSTATE[42S22]: Column not found: 1054 Unknown column 'id' in 'field list'"

Seems like database (DB) is connected but empty. Didn't you allow the script to build the structure (per doctrine)?

No entity with UUID 'c8b1a070-c30c-11eb-a500-b110c2dbffaa

With empty DB you should have got an error when you created the channel.

in config.js are also a secound entry with url:"demo....." is also to change ?

You may leave or remove it, it's usually not needed. But with this in configuration you have the option to select between these two middlewares (Local vs. Demo) when adding or creating a channel. Are there two entries? Which one did you choose when creating the channel with the UUID above?

[RaptorSDS]

now a RP3B with dietpi image with root user (i have a seperate SD-Card with rpi-image) as spare i have a rk3229 tv box with buster

yes i allowed the script build DB and also demo entrys ; i did not use doctrine no i did not get a error at create channel -> on demo.volkszaehler.org both channels are there and not local --> demo.volkszahler.org i used the Local(default) for create channels and both entry demo and local(default) are visiable at frontend

[J-A-U]

Please show the content of ./etc/config.yaml

[RaptorSDS]

root@DietPi: ~/volkszaehler.org/etc# ls config.dist.yaml config.yaml.bak middleware.json config.yaml dbcopy.dist.yaml root@DietPi: ~/volkszaehler.org/etc# nano config.yaml

# general settings overriding php configuration
php:
  timezone: Europe/Berlin
  locale: ['de_DE', 'en_US', 'C']

# Doctrine database configuration
# http://www.doctrine-project.org/projects/doctrine-dbal/en/2.6
db:
  driver: pdo_mysql
  host: localhost
  # port: 3306
  user: vz
  password: demo
  charset: utf8
  dbname: volkszaehler
  path: volkszaehler # only used for sqlite

  # db admin credentials (used by doctrine cli and setup script)
  admin:
    user: vz-admin
    password: secure

  # database optimizer - leave empty for automatic
  # optimizer: Volkszaehler\Interpreter\SQL\MySQLOptimizer # provides additional group=15m s$

# push server for realtime frontend updates
push:
  server: 5582
  broadcast: 8082
  # routes for wamp access
  wamp:
    - /
    - /ws
  # routes for plain web socket access
  websocket:
    - /socket

network:
  # limit maximum POST body size, e.g. 4096
  postlimit: false

# enable debug messages by default if true
debug: false

/var/www/volkszaehler.org/etc/config.yaml

general settings overriding php configuration
php:
  timezone: Europe/Berlin
  locale: ['de_DE', 'en_US', 'C']

# Doctrine database configuration
# http://www.doctrine-project.org/projects/doctrine-dbal/en/2.6
db:
  driver: pdo_mysql
  host: localhost
  # port: 3306
  user: vz
  password: demo
  charset: utf8
  dbname: volkszaehler
  path: volkszaehler # only used for sqlite

  # db admin credentials (used by doctrine cli and setup script)
  admin:
    user: vz-admin
    password: secure

  # database optimizer - leave empty for automatic
  # optimizer: Volkszaehler\Interpreter\SQL\MySQLOptimizer # provides additional group=15m s$

# push server for realtime frontend updates
push:
  server: 5582
  broadcast: 8082
  # routes for wamp access
  wamp:
    - /
    - /ws
  # routes for plain web socket access
  websocket:
    - /socket

network:
  # limit maximum POST body size, e.g. 4096
  postlimit: false

# enable debug messages by default if true
debug: false

[J-A-U]

At his point i have no idea how you managed to create channels on demo.volkszaehler.org instead of localhost.

[RaptorSDS]

without both entry demo and local(default) i now get channels add to local mybe it was my fault and i forget () to close but the localhost/api/capabilitys i still get same error

An exception occurred while executing 'EXPLAIN SELECT COUNT(id) FROM data USE INDEX (PRIMARY)':\n\nSQLSTATE[42S22]: Column not found: 1054 Unknown column 'id' in 'field list'"

with reinstall complete system i got this ( complete reflash of emmc of my testbox, clean armbian buster minimal rom )

 !
 ! [CAUTION] This operation should not be executed in a production environment!
 !

 Creating database schema...

 [OK] Database schema created successfully!

 Processing entity "Volkszaehler\Model\Aggregate"
 Processing entity "Volkszaehler\Model\Entity"
 Processing entity "Volkszaehler\Model\Channel"
 Processing entity "Volkszaehler\Model\Aggregator"
 Processing entity "Volkszaehler\Model\Data"
 Processing entity "Volkszaehler\Model\Property"

 Proxy classes generated to "/root/volkszaehler.org/lib/Model/Proxy"
~

create volkszaehler.org database user? [y] y
creating db user vz with proper rights...

allow channel deletion? [n] y
granting db user vz delete rights...

insert demo data in to database? [n] y
ERROR 1054 (42S22) at line 28: Unknown column 'id' in 'field list'

with rpi-image -> on my rpi3b i did not get any measument from middleware to db also no uuid is create , middleware response with http200 there is also some db realate issue

is there any log from middleware to db ?

[RaptorSDS]

I hope that the question may ask again, I do not want to appear impatient

is there any log from middleware to db ?

Next step for next few days build database and database user step by step , -->Maybe someone can tell me why these Database errors seem to occur only with me -> is there one who can test the RPI image again. I can not explain why even on the finished image no data is written to the database.

[andig]

is there any log from middleware to db ?

There isn't. I'd suggest to switch the discussion to the mailing list, it's probably easier to have it in German, too.