Missing things in the smarthome hub!

[Smarthome-Creator]

Describe the bug Since version 2.5.0 I miss a lot of channels in the smarthome hub.

SEPIA client and server versions

• SEPIA-Server version: 2.5.0

To Reproduce Steps to reproduce the behavior:

1. Go to 'Smart Home in Sepia Control HUB'
2. Click or say 'Load Hub Info'
3. See: Not more then 10 things, in version 2.4.1.b I still had all I had in openhab.

Expected behavior For example, I can no longer send my vacuum cleaner off or wake up my home server because the channels are no longer there.

[fquirin]

Oh :astonished: ... is it exactly 10 ? I will check that immediately

[fquirin]

So the good news is I can still load dozens of devices at the same time using for example "http://demo.openhab.org:8080" for testing. The bad news is that will make it harder to find out why its not working for you :-/

• Which version of openHAB is this?
• Are you using item files for setup or Paper UI?
• Can you reach the devices via voice/chat input?

[EDIT] Some more things to try ^^

• Are there any "hidden" devices shown at the bottom of the list?
• Are there any errors appearing in "~/SEPIA/sepia-assist-server/log.out" after loading the devices?
• Does it make a difference when you select a specific group of devices ("Get Devices" selector)?

[Smarthome-Creator]

I actually only have 9 channels left. :-( I use the current version 2.5.5 of openHAB. I use item files and Paper UI. Only the existing channels can be controlled via voice and chat. For example: "fish tank light" on/off works fine but "start the ps4" channel is no longer available.

No hidden devices are shown. No errors after loading devices. If I load all at once, everything is ok, but if I just want to load the devices, for example, the following message appears: "result": "fail", "error": "no devices found or failed to contact HUB" AND "No items found or no access to smart home system." For example, I have defined my home server as a device and it is no longer there.

[sepia-assistant]

Is the PS4 configured via item files? Could you post the device configuration somehow (item file or REST API result)?

If I load all at once, everything is ok, but if I just want to load the devices, for example, the following message appears: "result": "fail", "error": "no devices found or failed to contact HUB"

So you see the PS4 when you select "Get Devices: All" but not if you choose "Device"? When you load it via "All" what does it show in the "Type" field?

[Smarthome-Creator]

I think maybe I found the problem. I no longer have the role of smarthomeadmin after the update. But how do I assign a role correctly? The last time I did it, all the other roles were gone.

Here's the way I would do it: User Management - uid1003 - Get Roles - (write) smarthomeadmin - Set Roles - ERROR: Cannot set Core Roles!

The last time I did it: User Management - uid1003 - (write) smarthomeadmin - Set Roles - Get Roles - all other roles were gone and I was just Smarthomeadmin.

Hence the question ... how do I assign an additional role?

[sepia-assistant]

• Login as admin
• enter the user ID you want to edit
• press 'Get Roles' to see what roles are already there
• edit the result to add smarthomeguest and optionally smarthomeadmin
• press 'Set Roles'

Example:

Note: You cannot modify the roles of the admin! This is what ERROR: Cannot set Core Roles! means.

The thing is, if your user doesn't have the roles you will see no devices at all. What you have is something like a partial result, that shouldn't be related to roles.

So you see the PS4 when you select "Get Devices: All" but not if you choose "Device"? When you load it via "All" what does it show in the "Type" field?

Could you check those two points again please :-)

[Smarthome-Creator]

Is the PS4 configured via item files? Could you post the device configuration somehow (item file or REST API result)?

If I load all at once, everything is ok, but if I just want to load the devices, for example, the following message appears: "result": "fail", "error": "no devices found or failed to contact HUB"

So you see the PS4 when you select "Get Devices: All" but not if you choose "Device"? When you load it via "All" what does it show in the "Type" field?

Sorry, we got it wrong. The PS4 no longer appears. However, this is also controlled via the Harmony Hub and, unfortunately, it no longer appears. But my TV, which is also controlled via the Harmony Hub. :-(

That's what it looks like to me. So the devices that Sepia still gets from openHAB are all ok, but the rest are simply gone.

Because of the REST API, I have to see how it works.

[Smarthome-Creator]

* Login as admin

* enter the user ID you want to edit

* press 'Get Roles' to see what roles are already there

* edit the result to add `smarthomeguest` and optionally `smarthomeadmin`

* press 'Set Roles'

Example:

Note: You cannot modify the roles of the admin! This is what ERROR: Cannot set Core Roles! means.

The thing is, if your user doesn't have the roles you will see no devices at all. What you have is something like a partial result, that shouldn't be related to roles.

So you see the PS4 when you select "Get Devices: All" but not if you choose "Device"? When you load it via "All" what does it show in the "Type" field?

Could you check those two points again please :-)

Unfortunately, the user role was not the solution.

[Smarthome-Creator]

I removed openHAB from sepia as a test and added it again, unfortunately that didn't help either. But I noticed that the button (register sepia) is no longer there. He is there at FHEM or the others.

[fquirin]

I removed openHAB from sepia as a test and added it again, unfortunately that didn't help either. But I noticed that the button (register sepia) is no longer there. He is there at FHEM or the others.

Yes this button actually didn't do anything for openHAB anyways. It's only required for FHEM, so I removed it for other HUBs ;-)

Sorry, we got it wrong. The PS4 no longer appears. However, this is also controlled via the Harmony Hub and, unfortunately, it no longer appears. But my TV, which is also controlled via the Harmony Hub. :-(

That's what it looks like to me. So the devices that Sepia still gets from openHAB are all ok, but the rest are simply gone.

Because of the REST API, I have to see how it works.

So do you think that is an issue between openHAB and Harmony HUB? Does openHAB itself still see the devices in question?

[Smarthome-Creator]

I've done the update dozens of times in the past three days and the result is always the same. In openHAB everything works great but as soon as I update to 2.5.0 I only get the 9 devices displayed in the Smarthome HUB. And it's always the same 9 devices.

Before I made the update, I checked everything again. User roles, devices in the Smarthome HUB and and and. I noticed a few things that happened after the update. I also tested it with a clean sepia installation and got exactly the same result.

I made you a few screenshots so that you might be able to understand it better.

After the first start after the update, I get two error messages about Smart Home. But I think they are normal because they don't come after restarting Sepia.

2020-06-17 20:47:57 ERROR - Elasticsearch - Missing index: smart-interfaces 2020-06-17 20:47:57 LOG - Elasticsearch - Trying to create missing index: smart-interfaces 2020-06-17 20:48:02 LOG - Elasticsearch: created index 'smart-interfaces' 2020-06-17 20:48:04 LOG - Elasticsearch: Index ready for work. 2020-06-17 20:48:04 ERROR - Elasticsearch - Missing index: smart-devices 2020-06-17 20:48:04 LOG - Elasticsearch - Trying to create missing index: smart-devices 2020-06-17 20:48:05 LOG - Elasticsearch: created index 'smart-devices' 2020-06-17 20:48:06 LOG - Elasticsearch: Index ready for work. 2020-06-17 20:48:06 LOG - Elasticsearch: found 14 of 14 mapped indices. All good.

Or will the existing database be destroyed here? If I look in the server settings after the update, everything is as it was before the update.

Then I go to the Smart Home section, select openHAB and click on load hub info. Sepia automatically deletes my selection here and brings this error message. The same applies to a clean installation.

If I go back to the core settings and write openhab "small" ...

... and then go back to the Smart Home HUB, select openHAB, click Load Hub info, everything is okay.

But then it just doesn't load all of my smart things. I suspect it is breaking up the database somewhere here.

[fquirin]

Hi and thanks for the detailed info. Now lets try to find out whats going on :-)

2020-06-17 20:47:57 ERROR - Elasticsearch - Missing index: smart-interfaces [...] 2020-06-17 20:48:04 ERROR - Elasticsearch - Missing index: smart-devices

These errors are perfectly fine. v2.5.0 introduced the new "internal" HUB und thus requires 2 new database indicies. If SEPIA doesn't find them it will simply create them. This is whats happening here.

Or will the existing database be destroyed here?

No, the existing database is actually the openHAB server (except for one config entry in SEPIA) since SEPIA writes the configuration into the "tag" field of the openHAB devices. The new database is only used for the new options. You can read more about it ->here<-.

Then I go to the Smart Home section, select openHAB and click on load hub info [...] If I go back to the core settings and write openhab "small" ...

Actually you can forget about the "LOAD" button in v2.5.0, its just to check your settings. Also you don't need to go back to the "Core Settings" page anymore since you can use the new "WRITE" button now to change your settings :-) When you press "Get Devices" the interface will now automatically fill the fields above with the stored settings of your server.

But then it just doesn't load all of my smart things. I suspect it is breaking up the database somewhere here.

We need to look further into the items itself. Can you get the data from openHAB REST interface for one items that shows up and another that is missing in the list please?

[EDIT] Here is an example call to get all items via the REST API of openHAB: http://192.168.0.10:8080/rest/items?recursive=false

[Smarthome-Creator]

So here is my switch for my home server from Sepia 2.4.1:

{"link":"http://192.168.0.25:8080/rest/items/Home_Server","state":"ON","stateDescription":{"readOnly":true,"options":[]},"editable":true,"type":"Switch","name":"Home_Server","label":"Home_Server","tags":["sepia-name\u003dHomeserver","sepia-room\u003dhallway","sepia-type\u003ddevice"],"groupNames":["Status","Network"]}

And here is the string for my vacuum:

{"link":"http://192.168.0.25:8080/rest/items/Saugroboter_ControlVacuum","state":"dock","stateDescription":{"pattern":"%s","readOnly":false,"options":[{"value":"vacuum","label":"Vacuum"},{"value":"spot","label":"Spot Clean"},{"value":"pause","label":"Pause"},{"value":"dock","label":"Dock"}]},"commandDescription":{"commandOptions":[{"command":"vacuum","label":"Vacuum"},{"command":"spot","label":"Spot Clean"},{"command":"pause","label":"Pause"},{"command":"dock","label":"Dock"}]},"editable":true,"type":"String","name":"Saugroboter_ControlVacuum","label":"Control Vacuum","tags":[],"groupNames":[]}

Still unconfigured. Appears in the Smart Home Hub.

And here is the switch for my home server from Sepia 2.5.0:

{"link":"http://192.168.0.25:8080/rest/items/Home_Server","state":"ON","stateDescription":{"readOnly":true,"options":[]},"editable":true,"type":"Switch","name":"Home_Server","label":"Home_Server","tags":["sepia-name\u003dHomeserver","sepia-room\u003dhallway","sepia-type\u003ddevice"],"groupNames":["Status","Network"]}

And here is the string for my vacuum:

"link":"http://192.168.0.25:8080/rest/items/Saugroboter_ControlVacuum","state":"dock","stateDescription":{"pattern":"%s","readOnly":false,"options":[{"value":"vacuum","label":"Vacuum"},{"value":"spot","label":"Spot Clean"},{"value":"pause","label":"Pause"},{"value":"dock","label":"Dock"}]},"commandDescription":{"commandOptions":[{"command":"vacuum","label":"Vacuum"},{"command":"spot","label":"Spot Clean"},{"command":"pause","label":"Pause"},{"command":"dock","label":"Dock"}]},"editable":true,"type":"String","name":"Saugroboter_ControlVacuum","label":"Control Vacuum","tags":[],"groupNames":[]}

Still unconfigured. Is no longer displayed in the Smart Home Hub.

For me it all looks the same, but sepia seems to see a difference here, so it loads some devices and some don't. I also found out that if I configure unconfigured devices in version 2.4.1, they will also be loaded in version 2.5.0. But no unconfigured devices are loaded in 2.5.0. This is of course a problem because I then cannot configure it via the Smart Home Hub but manually via the REST API.

[fquirin]

Thanks for the data.

So the items look the same before and after update and thats to be expected since the SEPIA update will not modify any openHAB data.

The data itself actually looks OK too. I ran through the SEPIA code a few times to see if it would break at some point using this "Saugroboter" JSON result but I don't see any reason why it should. Though there is something that I'll have to check again. Comparing results to data I get from "demo.openhab.org" it could be that devices that have an empty tags field are skipped completely ... not even showing as "Type: - Choose -" anymore.

I also found out that if I configure unconfigured devices in version 2.4.1, they will also be loaded in version 2.5.0. But no unconfigured devices are loaded in 2.5.0. This is of course a problem because I then cannot configure it via the Smart Home Hub but manually via the REST API.

This somewhat confirms my suspicion from above. Could you try to set any "tag" value (e.g. "test") via the openHAB rest API to see if the device shows up in the SEPIA Control HUB afterwards? In the meantime I'll try to reproduce the error on my test system.

[fquirin]

I found the bug! The empty "tags" field was just an coincidence, it was an issue with the new device filter that failed when both deviceType and filter were "null" :beetle: :facepalm: . This can happen when the device has no "category" field and no previously defined SEPIA tags.

Could you please test this updated file: sepia-assist-v2.5.1.jar.zip

Simply unzip it to SEPIA/sepia-assist-server/ (next to the current version 'sepia-assist-v2.5.1.jar') and restart the server.

[Smarthome-Creator]

Hey Florian, thx 4 the Hotfix Update. Works fine, all Smart Things dislplayed again.

Great Job

[sepia-assistant]

:+1:
I will leave this issue open until the fix arrives in the next official release.

[Smarthome-Creator]

Hello Florian,

thx 4 the new great feature update... but...i have the same issue since i have updatet from 2.6.0 to 2.6.1.

:-(

[fquirin]

Hi @Smarthome-Creator , that's surprising. I didn't really change anything in the interface. Do you see any errors in the sepia-assist-server/log.out ? Did something else change as well? Can you ping the URL from the server?

[Smarthome-Creator]

Hello Florian,

2022-02-15 16:20:03 ERROR - Service:OpenHAB - failed to get devices from server!

That's all what i get back from the log. Yes i can ping the server.

Also i renewed my basic auth key for OH3... but the same result.

[fquirin]

Do you still have the v2.6.0 backup folder? If so could you maybe try to run the old server once to see if this is actually a version problem? (you can simply shutdown the server, rename the existing folder, name the backup 'SEPIA' and start the old server again).

[fquirin]

@Smarthome-Creator quick question: Do you see no devices anymore or did just some disappear? (like in the initial bug)

[Smarthome-Creator]

@fquirin, I won't be able to test it until the weekend.

[Smarthome-Creator]

Hallo Florian,

tut mir leid aber ich muss hier mal kurz auf deutsch switchen um dir zu beschreiben was ich versucht habe. Ich bun nun wieder auf Version 2.6.0 zurückgeswitcht und ich bekomme nun einfach gar keine Verbindung mehr zu meinem OH3 zustande. Ich nutze openHAB 3.2.0.

Möglicherweise liegt es ja auch an OH selbst, ich kann es nun aber nicht mehr überprüfen, da ich mit meinem Latein am Ende bin.

2 Sachen sind hierbei recht umständlich... will ich ich den Basic Auth Key von OH auf dem Sepia Server ablegen, muss ich zwingend über die https URL, will ich Smarthome Devices abfragen, geht das nur wenn ich es über die http URL mache, da ich über die https URL ja keine http URL's abfragen kann.

Hast du vieleicht noch irgendeine Idee wo ich nach dem Fehler suchen kann? Ich bin echt aufgeschmissen.

[fquirin]

2 Sachen sind hierbei recht umständlich... will ich ich den Basic Auth Key von OH auf dem Sepia Server ablegen, muss ich zwingend über die https URL, will ich Smarthome Devices abfragen, geht das nur wenn ich es über die http URL mache, da ich über die https URL ja keine http URL's abfragen kann.

Moment, hier komme ich noch nicht ganz mit ... ... oder vielleicht doch :sweat_smile: . Dein OH3 ist http ja? Nutzt also keine self-signed SSL Zertifikate oder Ähnliches? Das heißt wenn du den SEPIA Control HUB mit HTTPS öffnest kriegst du auf der Smart-Home Seite einen mixed-content Error? Ich glaube eigentlich sollte das nicht passieren, weil der SEPIA Server die Requests macht und nicht die Webseite. Wenn du dich also einloggen kannst sollte der Rest auch klappen :thinking: .

Du könntest vielleicht den Request mal via Terminal probieren (cURL example):

curl -X POST \
  http://[assist-server-host]/integrations/smart-home/getDevices \
  -H 'Content-Type: application/json' \
  -d '{"hubName": "openhab",  "hubHost": "http://openhab:8080", "deviceTypeFilter": "", "GUUID": "uid1007", "PWD": "myClearTextPassword123" }'

Mit entsprechend angepassten URLs (http oder https) und GUUID + PWD natürlich.

[fquirin]

Hi @Smarthome-Creator , motivated by the discussion here: https://github.com/SEPIA-Framework/sepia-docs/issues/174 I've installed OH 3.2.0 and tested a few things. So far everything seems to work fine you just have to consider a few things:

• If you use basic authentication (SEPIA type Basic) make sure it is actually activated in your OH settings
• If you use API token authentication please set SEPIA auth. type to Bearer and then add your token without any other modifications
• If you've created items via .items file keep in mind that they might be write protected for access via REST API

Hope this helps!

[Smarthome-Creator]

Hello @fquirin,

here my actually settings in Sepia Server: In this case i tryed it with "Bearer" and my OH3 (3.2.0) Token. Before I always had it with "Basic" and my OH3 (3.1.0) token and that worked.

Here's the output from curl request: curl -X POST \ http://stts:20721/integrations/smart-home/getDevices \ -H 'Content-Type: application/json' \ -d '{"hubName": "openhab", "hubHost": "http://openhab:8080", "deviceTypeFilter": "", "GUUID": "uid1007", "PWD": "MyPW ;-)" }'

{"result":"fail","error":"401 not authorized"}

It's the same like:

I don't know how to find the problem.

Current configuration that no longer works: openHAB 3.2.0 / Sepia 2.6.0 / Basic Auth / OH API Token

Old configuration with which it worked:

openHAB 3.1.0 / Sepia 2.6.0 / Basic Auth / OH API Token

Test fail:

openHAB 3.2.0 / Sepia 2.6.0 / Bearer / OH API Token

[fquirin]

Hm this is really strange. So here is what I've observed:

• No auth. data at all: Devices are shown but cannot be edited
• Type "Basic" with correct API token: No items found
• Type "Bearer" with correct API token: Everything works
• Type "Bearer" with wrong API token: No items found

So a wrong token seems to be worse than no token :sweat_smile:

Can you remove the auth. data and see what happens please?

[Smarthome-Creator]

Hm this is really strange. So here is what I've observed:

* No auth. data at all: **Devices are shown** but cannot be edited -> Checked, same result

* Type "Basic" with correct API token: **No items found** -> Checked, same result: No items found or no access to smart home system.

* Type "Bearer" with correct API token: Everything works -> Checked, result: No items found or no access to smart home system.

* Type "Bearer" with wrong API token: **No items found** -> Checked, same result. No items found or no access to smart home system.

So a wrong token seems to be worse than no token 😅

Can you remove the auth. data and see what happens please?

[fquirin]

No auth. data at all: Devices are shown but cannot be edited -> Checked, same result

So this is actually working for you as well? Did you try to create a new API Token and use this one? Also I'm not entirely sure what effects the "scope" option has, I left it blank and depending on the user you use to create the token you might get different results as well (I used the primary account because it was the only one I had).

[Smarthome-Creator]

@fquirin do we want to close this issue and continue with this one?

https://github.com/SEPIA-Framework/sepia-docs/issues/143

Yes I created a new token but all attempts failed.

I haven't changed anything. SEPIA standard installation, openHAB standard installation, no proxies or certificates.

With VSCode it works without any problems.

Unfortunately, I can't really test efficiently either, because with SEPIA I always have to switch between the https page (to make changes to the server) and the http page to be able to test there to see if anything has worked. Log out and log in again each time.

You can't really test efficiently there.

[SHU-red]

Did you try to NOT use "openhab" as hostname but the IP itsself? Are you using openhab via http or https?

Im using also newest OH and Sepia but everything only http No problems here

What is your environment? HomeServer? PC? Docker? RaspberryPi?

[Smarthome-Creator]

Did you try to NOT use "openhab" as hostname but the IP itsself? Are you using openhab via http or https?

Yes, I tried it directly via the IP.

Im using also newest OH and Sepia but everything only http No problems here

I've had it running over http since the first hour since it doesn't work over https to http.

What is your environment? HomeServer? PC? Docker? RaspberryPi?

Everything runs on Raspis.

[SHU-red]

Just a guess Did you take a look at frontail/logs after attempting to login?

Absolutely not sure but maybe there is something pointing to the root cause ...

[fquirin]

@fquirin do we want to close this issue and continue with this one? https://github.com/SEPIA-Framework/sepia-docs/issues/143

I think it's the same problem, maybe we keep going here and I'll post the results in the other issue later as well.

Unfortunately, I can't really test efficiently either, because with SEPIA I always have to switch between the https page (to make changes to the server) and the http page to be able to test there to see if anything has worked. Log out and log in again each time.

I'm not sure what you mean. You should be able to use the non-SSL page to change server settings as well :thinking: . If you need to login 2 different accounts at the same time (admin and test user for example) I'd suggest to use 2 browsers (FF and Chrome or Edge) or private/non-private tab or the same browser with 2 different profiles :-).

Some ideas:

• If you can read the devices without auth. data then I don't think HTTPS or HTTP is the problem, the connection to openHAB in general seems to be fine
• The failed CURL request indicated that your SEPIA user wasn't properly authenticated: curl -X POST http://[sepia-ip]:20721/integrations/smart-home/getDevices -H 'Content-Type: application/json' -d '{"hubName": "openhab", "hubHost": "http://openhab:8080/", "deviceTypeFilter": "", "GUUID": "uid1007", "PWD": "[password]" }' - Can you double-check if the password was correct and the user has the smarthomeadmin role
• Your openHAB settings look something like this now? smarthome_hub_auth_type=Bearer and smarthome_hub_auth_data=oh.sepia.zcjTS41WMsrPTD2xmWGdO1GDwtBH46JyiVELlYSEgksJPQngv0r5GwdSpakCFywH3OjebOLBtBypNQPKcJw?
• Did we check the SEPIA/sepia-assist-server/log.out data already? I can't find it anymore.

[Smarthome-Creator]

@fquirin, when i start the sepia server, the proxy start fail, is this normal?

When I try to make server settings via the Sepia http URL I get the following error message:

I tried the CURL request again with uid1003 and 1007. 1003 is the smart home admin and 1007 my smart home guest. Same output like:

I can't check anemone my admin (uid1003). Error Message:

And here is the log:

2022-04-03 20:33:21 ERROR - Service:OpenHAB - failed to get devices from server!

[fquirin]

when i start the sepia server, the proxy start fail, is this normal?

This is SEPIA-Home v2.6.1 right? The most recent run-script should detect if you have a valid Nginx config at /etc/nginx/sites-enabled/sepia... and decide what proxy URL to ping. If you did install Nginx once but don't use it anymore you can delete the config to avoid the test (there is a setup option to clean up old configs). If you were actually planning to use it then there seems to be a problem, maybe it never started?

"not allowed to access service from outside the private network"

That is weird. From where did you send this request? If you check your sepia-assist-server/log.out file you should see something like this: "Unauthorized access attempt to server config! Host: ...". You can deactivate this feature in the core settings, I think its called allow_global_dev_requests, but usually the hostname/IP-check should recognize that your origin is one of: localhost, .local domain, internal IP.

I tried the CURL request again with uid1003 and 1007. 1003 is the smart home admin and 1007 my smart home guest. Same output like:...

The error indicates that your request is refused BEFORE the get devices method is called. That can happen in 2 cases:

• Wrong username or password
• User is not 'superuser' and not 'smarthomeadmin'

uid1003 should be 'superuser' unless you created a new admin on purpose at some point. If this user can edit the core settings than has to still be admin. 'smarthomeguest' users can control the smart-home items via SEPIA client but cannot change any settings.

can't check anemone my admin (uid1003). Error Message:

In your screenshot it says "1003" instead of "uid1003". Anyway the admin cannot be edited so the server will refuse to open the settings. If this is the case that its definitely the 'superuser' though ;-).

Just to double-check this is the curl request that works for me:

curl -X POST http://192.168.0.10:20721/integrations/smart-home/getDevices -H 'Content-Type: application/json' -d '{"hubName": "openhab", "hubHost": "http://192.168.0.10:8080", "deviceTypeFilter": "", "GUUID": "uid1003", "PWD": "supersafe123" }'

[Smarthome-Creator]

I think this is probably looking for a needle in a haystack. I suspect that something went wrong with the update to version 2.6.0. And not accidentally fixed with 2.6.1.

I'm closing the ticket here because the original bug in this ticket has been fixed. I'll probably just have to reinstall the thing.

Nevertheless thanks for your support.

[fquirin]

ok, feel free to continue in https://github.com/SEPIA-Framework/sepia-docs/issues/143 ;-)