search

Domi04151309 / HomeApp

HomeApp is a small and easy to use smart home app with a simple framework.
https://f-droid.org/packages/io.github.domi04151309.home/
GNU General Public License v3.0
106 stars 24 forks source link

[Feature Request] Support for Shelly switches #27

Closed cweiske closed 2 years ago

cweiske commented 2 years ago

Shelly relay devices (Shelly Plus 1, Shelly 1, see https://shelly.cloud/ ) are installed in power sockets and light switches and can be used without any cloud services. They have a publicly documented API at https://shelly-api-docs.shelly.cloud/gen1/ (Shelly 1) and https://shelly-api-docs.shelly.cloud/gen2/ (Shelly Plus 1).

It would be nice if HomeApp would support their API.

I would like to support that by buying you some test devices. Please contact cweiske+shelly@cweiske.de if you are interested and let me know your postal address.

Service Discovery

Shelly devices announce themselves via MDNS (Bonjour/Zeroconf) and not SSDP. One can use the linux tool "avahi-browse" to find them.

$ avahi-browse --cache --no-db-lookup --parsable --all
+;wlp1s0;IPv4;ShellyPlus1-A8032ABD2342;_http._tcp;local
+;wlp1s0;IPv4;ShellyPlus1-A8032ABD2343;_http._tcp;local
+;wlp1s0;IPv4;ShellyPlus1-A8032ABE2344;_http._tcp;local
+;wlp1s0;IPv4;shellyplus1-a8032abd2342;_shelly._tcp;local
+;wlp1s0;IPv4;shellyplus1-a8032abd2343;_shelly._tcp;local
+;wlp1s0;IPv4;shellyplus1-a8032abe2344;_shelly._tcp;local

Android natively supports mDNS discovery also with Kotlin: https://developer.android.com/training/connect-devices-wirelessly/nsd#discover

Device information

All devices (v1 and v2) have a public (unprotected) JSON information file at /shelly:

v1

{
    "type": "SHSW-1",
    "mac": "483FDA82687A",
    "auth": false,
    "fw": "20211109-124958/v1.11.7-g682a0db",
    "longid": 1,
    "num_outputs": 1
}

https://shelly-api-docs.shelly.cloud/gen1/#shelly

v2

$ curl -s shelly1.home.cweiske.de/shelly|jq .
{
  "id": "shellyplus1-a8032abd1bcc",
  "mac": "A8032ABD2342",
  "model": "SNSW-001X16EU",
  "gen": 2,
  "fw_id": "20210921-202758/0.8.1-g52de872",
  "ver": "0.8.1",
  "app": "Plus1",
  "auth_en": false,
  "auth_domain": null
}

The "gen" key is only available on v2 devices.

https://shelly-api-docs.shelly.cloud/gen2/Overview/CommonServices/Shelly#http-endpoint-shelly

Number of switches

v1

The JSON object in GET /status has a "relays" array property: https://shelly-api-docs.shelly.cloud/gen1/#shelly1-1pm-status

{
    "relays": [
        {
            "ison": false,
            "has_timer": false,
            "timer_started": 0,
            "timer_duration": 0,
            "timer_remaining": 0,
            "overpower": false,
            "source": "http"
        }
    ],
    ...
}

v2

We have to look for "switch:*" keys in the device configuration object for v2-devices: https://shelly-api-docs.shelly.cloud/gen2/Overview/CommonServices/Shelly#shellygetconfig

$ curl -s shelly1.home.cweiske.de/rpc/Shelly.GetConfig | jq .
{
  ...
  "switch:0": {
    "id": 0,
    "name": "wohnzimmer_1",
    "in_mode": "follow",
    "initial_state": "restore_last",
    "auto_on": false,
    "auto_on_delay": 60,
    "auto_off": false,
    "auto_off_delay": 60
  },
  ...
}

Detecting on/off device state

v1 returns the ison state in the /status route as well.

Both v1 and v2 devices have the /relay/0 URL, which tells us if it is switched on or off via ison:

$ curl -s shelly1.home.cweiske.de/relay/0|jq .
{
  "ison": true,
  "has_timer": false,
  "timer_started_at": 0,
  "timer_duration": 0,
  "timer_remaining": 0,
  "source": "http"
}

v2 also returns all states in /rpc/Shelly.GetStatus:

$ http -p b shelly1.home.cweiske.de/rpc/Shelly.GetStatus
{
   ...
    "switch:0": {
        "id": 0,
        "output": true,
        "source": "http",
        "temperature": {
            "tC": 58.8,
            "tF": 137.8
        }
    },
    ...
}

Switching on/off

Both v1 and v2 devices have the /relay/0 URL, which allows turning the device on, off or toggling the device state with a GET parameter:

$ curl http://shelly1.home.cweiske.de/relay/0?turn=on
$ curl http://shelly1.home.cweiske.de/relay/0?turn=off
$ curl http://shelly1.home.cweiske.de/relay/0?turn=toggle

Note: Some devices have more that one relay. See above for detecting the number and names of switches.

Temperature sensors

v1

It's possible to attach a temperature sensor to a Shelly 1 device. /status gives us the values:

{
    ...
    "ext_sensors": {
        "temperature_unit": "C"
    },
    "ext_temperature": {
        "0": {
            "hwID": "0300",
            "tC": 23,
            "tF": 73.4
        }
    },
    "ext_humidity": {
        "0": {
            "hwID": "0300",
            "hum": 52.3
        }
    },
    ...
}

Authentication

v1

https://shelly-api-docs.shelly.cloud/gen1/#common-http-api

All resources except for /shelly will require Basic HTTP authentication when it is enabled

v2

https://shelly-api-docs.shelly.cloud/gen2/Overview/CommonDeviceTraits#authentication

Communication through HTTP and Websocket channels is secured by a digest authentication mechanism using the SHA256 hmac algorithm as defined in RFC7616. When enabled, all communication is protected except:

  • the RPC method Shelly.GetDeviceInfo
  • the HTTP endpoint /shelly
Domi04151309 commented 2 years ago

Sounds great! Sadly, I am currently unable to accept your offer because of the limited time on my side. Since this is more of a hobby project I am working on, getting test devices would, in my opinion, lift this to a project with obligations. I, however, feel like I am currently not able to invest enough time in this project to deliver reliable production-ready implementations of new APIs, which I do not use in my day-to-day life. It would be possible to experiment with implementations of new APIs on separate branches and work closely together to test these implementations.

Domi04151309 commented 2 years ago

You can check out the Shelly branch to see whether the basic info request is working already.

cweiske commented 2 years ago

I tried d1c30fe, and it shows the shelly information. At first it showed me "Das Gerät ist momentan nicht verfügbar", but that was because I was missing the trailing slash at the end of the address ("http://shelly1.home.cweiske.de" instead of "http://shelly1.home.cweiske.de/"). Maybe that could be added automatically.

Domi04151309 commented 2 years ago

I implemented some methods with the latest commit but I don't really get how authentication works. Do you know what headers to send or what data to post? Also, automatic addition of trailing slashes is back.

cweiske commented 2 years ago

Authentication v1: Basic Auth

The HTTP request needs to contain a header

"Authorization: Basic " + base64encode($user . ":" . $pass)

That's all.

https://datatracker.ietf.org/doc/html/rfc7617#section-2

Authentication v2: Digest auth

Digest auth is not nice because you have to do two http calls. The first to get told that you need authentication details and to fetch the "nonce" needed in the auth data, the second to do the request again and this time with authorization data.

It's pretty complex (but it doesn't leak the password on plaintext connections, and prevents replay attacks).

https://datatracker.ietf.org/doc/html/rfc7616

I suggest to simply not implement it right now.

cweiske commented 2 years ago

I made a crude attempt to add service discovery for shelly devices: https://github.com/cweiske/HomeApp/tree/shellydisco It kinda works because it shows the shelly devices in my network, and I can add them.

What does not work:

I'm a kotlin and android noob. The code is hacked together from the manual at https://developer.android.com/training/connect-devices-wirelessly/nsd#discover Please have a look at it and take what makes sense and dump the rest.

Domi04151309 commented 2 years ago

Log.e(Global.LOG_TAG, "gen" + serviceInfo.attributes["gen"].decodeToString()) should show what you are looking for. What you are logging is that your variable of type ByteArray at 3417e53 is called gen.

cweiske commented 2 years ago

I'll try that this evening. Btw, you can simply rebase the shelly branch against master, so that the changes from master get into the shelly branch without picking the single commits manually. It also keeps history linear: git checkout shelly && git rebase master

cweiske commented 2 years ago

Ah, I just saw that you fixed it yourself. The "gen" attribute in discovery is only available for gen2 devices. Gen1 do not have that attribute.

Domi04151309 commented 2 years ago

Authentication should work for gen 1 devices with the latest commit.

Domi04151309 commented 2 years ago

What I don't get is whether authentication is required by default or whether it has to be enabled by the user.

cweiske commented 2 years ago

By default auth is disabled. The user has to enable it manually. The /shelly response tells us if auth is enabled or not.

v2:

"auth_en": false,

v1:

"auth": true,
Domi04151309 commented 2 years ago

Will useless auth headers be ignored by the device or will they throw an error. For example if auth is disabled but the auth header is set. For Gen 2 devices I guess the first request will be 200 instead of 401, so that is easily checkable.

cweiske commented 2 years ago

ServiceDiscovery:

Shelly support:

Domi04151309 commented 2 years ago

Discovery should work now and relays should show up with their name sorted by their id for gen 1 and gen 2.

Domi04151309 commented 2 years ago
  • When adding a device via service discovery, I get an "OK" popup. In the list, I don't see any change. It would be nice if the recently added device got a new green checkmark icon, a grey text and/or moved to the end of the list, so that I can concentrate on the ones that I did not add yet.
  • devices that are already in the devices list should be shown in grey text and/or with a green checkmark, too.

I will work on this once the shelly api is working as intended.

cweiske commented 2 years ago

Thank you so much @Domi04151309 for adding shelly support, and for building HomeApp!

For the record, I tested the following devices:

Domi04151309 commented 2 years ago

Now only digest authentication is left for gen 2 devices.

Domi04151309 commented 2 years ago

It would be great if you could send me the log outputs of the latest commit so I can verify if it works as intended.

cweiske commented 2 years ago

No debug output is shown but an exception/error:

E/Volley: [1823] NetworkUtility.shouldRetryException: Unexpected response code 401 for http://shelly1.home.cweiske.de/rpc/Shelly.GetConfig
W/HomeApp: com.android.volley.AuthFailureError
        at com.android.volley.toolbox.NetworkUtility.shouldRetryException(NetworkUtility.java:189)
        at com.android.volley.toolbox.BasicNetwork.performRequest(BasicNetwork.java:145)
        at com.android.volley.NetworkDispatcher.processRequest(NetworkDispatcher.java:132)
        at com.android.volley.NetworkDispatcher.processRequest(NetworkDispatcher.java:111)
        at com.android.volley.NetworkDispatcher.run(NetworkDispatcher.java:90)

(Why has the gradle debug type minification, shrinking and proguard enabled? Is there a reason the obfuscation is enabled at all?)

Domi04151309 commented 2 years ago

It's enabled because of app size on production builds and to detect errors in the process for debug builds.

Domi04151309 commented 2 years ago

The latest commit overrides an additional method to try catch the error.

cweiske commented 2 years ago 
E/Volley: [2004] NetworkUtility.shouldRetryException: Unexpected response code 401 for http://shelly1.home.cweiske.de/rpc/Shelly.GetConfig
E/HomeApp: Try catching the error
Digest qop="auth", realm="shellyplus1-a8032abd1bcc", nonce="61a91ab4", algorithm=SHA-256
E/HomeApp: shellyplus1-a8032abd1bcc
E/HomeApp: 61a91ab4
E/AndroidRuntime: FATAL EXCEPTION: Thread-3
    Process: io.github.domi04151309.home, PID: 16637
    java.lang.Error: javax.xml.datatype.DatatypeConfigurationException: Provider org.apache.xerces.jaxp.datatype.DatatypeFactoryImpl not found
        at javax.xml.bind.DatatypeConverterImpl.<clinit>(DatatypeConverterImpl.java:907)
        at javax.xml.bind.DatatypeConverter.initConverter(DatatypeConverter.java:155)
        at javax.xml.bind.DatatypeConverter.printHexBinary(DatatypeConverter.java:640)
        at io.github.domi04151309.home.custom.JsonObjectRequestDigestAuth.parseNetworkResponse(JsonObjectRequestDigestAuth.kt:59)
        at io.github.domi04151309.home.custom.JsonObjectRequestDigestAuth.parseNetworkError(JsonObjectRequestDigestAuth.kt:31)
        at com.android.volley.NetworkDispatcher.parseAndDeliverNetworkError(NetworkDispatcher.java:174)
        at com.android.volley.NetworkDispatcher.processRequest(NetworkDispatcher.java:160)
        at com.android.volley.NetworkDispatcher.processRequest(NetworkDispatcher.java:111)
        at com.android.volley.NetworkDispatcher.run(NetworkDispatcher.java:90)

I think that debug builds should not have shrinking, minification and proguard enabled. Only production builds. I did not get proper stacktraces until I modified

--- app/build.gradle
+++ app/build.gradle
@@ -12,9 +12,6 @@ android {
     }
     buildTypes {
         debug {
-            minifyEnabled true
-            shrinkResources true
-            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
         }
         release {
             minifyEnabled true
Domi04151309 commented 2 years ago

The latest commit could fix the issue. Shrinking is enabled on debug builds because of past crashes of production builds caused by errors in the shrinking process, so that those errors will get detected before they are pushed to production.

cweiske commented 2 years ago

Still a similar error:

E/HomeApp: Try catching the error
E/HomeApp: Digest qop="auth", realm="shellyplus1-a8032abd1bcc", nonce="61aa307a", algorithm=SHA-256
E/HomeApp: shellyplus1-a8032abd1bcc
E/HomeApp: 61aa307a
E/AndroidRuntime: FATAL EXCEPTION: Thread-3
    Process: io.github.domi04151309.home, PID: 4025
    java.lang.Error: v2.a: Provider org.apache.xerces.jaxp.datatype.DatatypeFactoryImpl not found
        at u2.b.<clinit>(:907)
        at u2.a.a(:155)
        at u2.a.b(:640)
        at p2.c.M(:59)
        at p2.c.L(:31)
        at y0.i.b(:174)
        at y0.i.d(:160)
        at y0.i.c(:111)
        at y0.i.run(:90)
     Caused by: v2.a: Provider org.apache.xerces.jaxp.datatype.DatatypeFactoryImpl not found
        at v2.b.a(Unknown Source:22)
        at u2.b.<clinit>(:905)
        at u2.a.a(:155) 
        at u2.a.b(:640) 
        at p2.c.M(:59) 
        at p2.c.L(:31) 
        at y0.i.b(:174) 
        at y0.i.d(:160) 
        at y0.i.c(:111) 
        at y0.i.run(:90) 
     Caused by: java.lang.ClassNotFoundException: Didn't find class "org.apache.xerces.jaxp.datatype.DatatypeFactoryImpl" on path: DexPathList[[zip file "/data/app/io.github.domi04151309.home-xchsc-C1ChIui-yCcpoTgw==/base.apk"],nativeLibraryDirectories=[/data/app/io.github.domi04151309.home-xchsc-C1ChIui-yCcpoTgw==/lib/arm64, /system/lib64, /system/product/lib64, /system/vendor/lib64]]
        at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:196)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:379)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:312)
        at v2.c.f(Unknown Source:9)
        at v2.c.c(Unknown Source:236)
        at v2.b.a(Unknown Source:4)
        at u2.b.<clinit>(:905) 
        at u2.a.a(:155) 
        at u2.a.b(:640) 
        at p2.c.M(:59) 
        at p2.c.L(:31) 
        at y0.i.b(:174) 
        at y0.i.d(:160) 
        at y0.i.c(:111) 
        at y0.i.run(:90) 
Domi04151309 commented 2 years ago

You can retry with the latest commit.

cweiske commented 2 years ago

It's better now. Auth doesn't work though, and HomeApp tries to parse the response despite the second 401:

E/Volley: [2424] m.e: Unexpected response code 401 for http://shelly1.home.cweiske.de/rpc/Shelly.GetConfig
E/HomeApp: Try catching the error
E/HomeApp: Digest qop="auth", realm="shellyplus1-a8032abd1bcc", nonce="61aa3a43", algorithm=SHA-256
E/HomeApp: shellyplus1-a8032abd1bcc
E/HomeApp: 61aa3a43
E/HomeApp: {"auth":{"realm":"shellyplus1-a8032abd1bcc","username":"admin","nonce":"61aa3a43","cnonce":1291756127,"response":"a6d69882242e1dfa4ce6121fa124d01ae886bdf14fa581259b6b34faa8a38799","algorithm":"SHA-256"}}
E/Volley: [2432] m.e: Unexpected response code 401 for http://shelly1.home.cweiske.de/rpc/Shelly.GetConfig
E/Volley: [2432] m.e: Unexpected response code 401 for http://shelly1.home.cweiske.de/rpc/Shelly.GetConfig
E/AndroidRuntime: FATAL EXCEPTION: main
    Process: io.github.domi04151309.home, PID: 5053
    java.lang.NullPointerException: Attempt to invoke virtual method 'org.json.JSONArray org.json.JSONObject.names()' on a null object reference
        at s2.z.s(:51)
        at s2.z.i(Unknown Source:0)
        at s2.w.b(Unknown Source:6)
        at z0.l.l(:100)
        at y0.f$b.run(:102)
        at android.os.Handler.handleCallback(Handler.java:883)

I'll have a look at your digest implementation this evening and try to find out what's wrong.

cweiske commented 2 years ago

I don't think that building an own digest auth implementation is it worth. e.g. that one is huge already: https://github.com/rburgst/okhttp-digest/blob/master/src/main/java/com/burgstaller/okhttp/digest/DigestAuthenticator.java

Also, digest auth allows doing multiple sequential requests by increasing a nonce count variable, which requires more work.

Just don't support auth on shelly v2 devices until an official digest auth for volley http lib is available: https://github.com/google/volley/issues/433

cweiske commented 2 years ago

I'd like to add more data from shelly switches (internal temperature, external temperature sensor, current power usage), but I feel that the current implementation hinders this.

ShellyAPI.kt basically takes the response to a request, modifies it a bit and handles it back to onSwitchesLoaded, which tries to makes sense of the json response data and displays some bits of information as list items. Now ShellyAPI.kt supports v1 and v2, and the v2 code mangles the v2 json response so that it looks like a v1 response, and handles it back to onSwitchesLoaded.

This makes implementing the other data kinda hard, because I have to do the same mangling for them as it's done for the switch info.

What do you think, would it make sense if ShellyAPI would just return a list of ListViewItem objects? That way JSON handling would only needed to be done in ShellyAPI and not MainActivity. Another way would be to let ShellyAPI return a ShellyData object containing arrays of SwitchData and TemperatureData and PowerUsageData objects back to onSwitchesLoaded. This would leave rendering to MainActivity, and ShellyAPI would really only return the extracted data.

Domi04151309 commented 2 years ago

You can check out the latest commit. I removed auth for gen 2 devices and made the onSwitchesLoaded method accept a RequestCallbackObject with the response type set to ArrayList<ListViewItem>.

cweiske commented 2 years ago

@Domi04151309 Please have a look at https://github.com/cweiske/HomeApp/commit/42da3b1cf60c561d1b51828d7ee8b90c7c1546bc

To make it easier to add new code I wanted to have tests with static JSON files. Running the tests without any Android environment is kinda hard, so I reduced the dependencies to the bare minimum - the Resources class so we can use getString(). I also had to move the parsing code out into a separate class so that the dependencies (selected device URL from "Devices(c).getDeviceById(deviceId).address") are not needed for the test.

What do you think? Is the approach ok, or is there a better way?

Maybe it's somehow possible to even get rid of getString in the parser, so we wouldn't need roboelectric, and tests would run faster.

Domi04151309 commented 2 years ago

Looks good to me but sadly I do not have a lot of experience with writing tests. Are there some resources you would recommend to read about writing tests?

cweiske commented 2 years ago

I've not yet found a tutorial that I'd call "good". There is the generic official android testing documentation at https://developer.android.com/training/testing whose examples sadly don't all work.

You can do approach it from the several sides:

There is junit which is/(was?) the de-facto standard for writing and running tests for java, and roboelectric that simulates the android API without actually running an emulator, which cuts down on test time while allowing me to use Resources.getString() properly. This is the first time I'm writing tests for an android project, so I'm a newbie here, too.

cweiske commented 2 years ago

I'd call shelly integration finished.