COVIVA with Apps Script

[Komet2005]

Do you think possible to convert your code to be use in an apps script ?

[tylerrmit]

I imagine it's possible. What did you have in mind? What are you trying to do?

As long as the platform allows you to make web requests and MD5 hashes and store your credentials somewhere secure, it would be possible.

I've never used Apps Scripts for anything. I'm time poor, so I might not be the person to do it. It seems like Apps Script is something where you probably want to be a JavaScript coder to use it, maybe the issue is you need some example code to get started?

The logic of how my Homebridge plugin talks to Coviva is open, but maybe it's not the clearest way to read what's going on, with all the plug-in housekeeping. I do have some very simple Python scripts somewhere to do things like "login to Coviva and list all devices" or "login to Coviva and turn a specific light off". They might be easier to read if one is trying to convert this stuff into Apps Script.

[Komet2005]

I've got an STR and my goal is to connect my TRM694G with an apps script in order to automatically turn on and turn off in function of the rental. To do this, do you think that you can share your Python script ?

[tylerrmit]

Looks like I already included some basic python scripts in the repository under the "tools" directory.

Start with "get_all.py"

The scripts are very primitive, they expect you to hard-code your user, password and "koala" ID for your coviva hub. These are the exact same three pieces of information that you use to login to the Coviva app. The "koala" ID is probably physically recorded on your hub. Before running the script, update the first line to the path to your python command, and update user/password/koala from their default 'xxx' values.

The first stage, the script takes your credentials and transforms them into the format required by Coviva:

1. Use "URL encoding" on the username (which is probably your email address) to handle any characters that would be "special" from the perspective of the HTTP protocol
2. Take a SHA512 hash of your password (because the hash is sent, not the plain-text password!
3. Concatenate 1 and 2 together with a ':' in between
4. Then convert those bytes to ascii
5. ... then apply base64 encoding
6. ... and then transform back to ascii
7. ... And then add a "Basic " prefix to it!

The Coviva protocol expects you to send a "device hardware id" hexadecimal string, I've hard-coded a value in the demo script, but in real life you'd probably do something like transform the mac address of the client to hex in some way. The Coviva hub seems to keep track of all the devices that have connected to it, so you don't want it to be different/random every time, it'll end up tracking hundreds of different "devices".

Once you have that credential string in the required form "Basic " it is time to connect to the Coviva API. You need to provide those credentials to get an access token. Parse the response to get the token and info on when it expires.

Then, open a "web socket", including the access token in the URL.

You can send a "command" and then receive a response.

"GET:all" -> This will get a dump of everything about your hub, including clients, all the "nodes", etc. The "nodes" are the Coviva devices that you're trying to control. Fun fact, it's possible to link your Netatmo devices to your Coviva account, if you do, you'll see those devices, too, as nodes.

"GET:nodes" -> This will get you slightly less info, just focussing on the "nodes" part of the "GET:all" command.

Either way, you're receiving JSON. It helps to use a service or tool that will parse it into something "pretty" rather than a dense long string, it's easier for a human to read.

The nodes will be numbered 1,2,3,... etc. and they will have some attributes. From memory, attribute 4 is whether it's on or off. Attribute 5 is a brightness. Reading the JSON will tell you all the attributes that your device has -- this was enough to tell me how to control the blinds for someone who had Coviva blinds, if I recall correctly. There are other attributes to tell you the name of the device (assigned by the human owner) and the type of the device, which would give you a hint as to which attribute numbers are what.

"PUT:nodes/1/attributes/4?target_value=0" would turn off node #1 if it is a light.

"PUT:nodes/1/attributes/4?target_value=1" would turn it on

"PUT:nodes/1/attributes/5?target_value=50" would set its brightness to 50%

"ping" -> this will send a little bit of activity to keep the websocket alive. In the demo script, we create a web socket connection, send one command, then close the web socket. But you could just as easily keep it open if necessary, and send a "ping" command every minute or five, to ensure that it stays active and isn't timed out by some firewall somewhere.

[tylerrmit]

If you get an error when opening the web socket relating to having too few arguments, you have to "pip uninstall websocket" and then "pip install websocket-client" to ensure you're using the expected web socket library

[tylerrmit]

So, to make it all work for apps script you'd hope that environment allows you to:

• use a url encode function
• create a sha512 hash of a string
• apply base64 encoding to a string and then output that as ascii
• make an HTTP POST request to authenticate and gain an access token
• Use a "websocket" to communicate with your Coviva devices via the hub

You'd have to figure out how to make it do these things one-by-one, and make sure the output looks similar to what you're seeing in the Python demo

[tylerrmit]

I don't believe the nodes tend to change their node number once they're paired to the Coviva hub. So once you know the node id from the "GET:all" or "GET:nodes" command, you can just have a script that asks to turn off/on node N each time. No need to get info each time if all you want to do is send a command to turn a switch off if it is not already off.

In the Homebridge plugin, I might be periodically sending a "GET:all" or "GET:nodes" to get the latest status of each device and pass it onto HomeKit. Because someone might come along and physically operate the switch -- this will send a message back to the Coviva Hub, so Coviva knows what's on and what's off. I can't remember whether I'm periodically polling with "GET:nodes" or whether Coviva actually pushes the update to the WebSocket when it happens. I feel like it pushes it.

[tylerrmit]

I think that's about as much as I can reasonably do to help -- I don't do Javascript in my day job, and I probably don't have the time to go exploring this app script stuff myself. But hopefully the python demo makes it easier to see how to interact with it.

If you're struggling to find the Javascript code to do some part of the sequence, you'll probably find it in the source of the Coviva Web Application if you stare hard enough. Somewhere in that site, it's including a "coreSDK.min.js" file on their server, and if you download it you'll see what they did. There's a lot of noise looking through their web application source, though.

I'll leave this "issue" open for a few days, let me know if you have any more questions.

[Komet2005]

Thanks a lot for your feedback. I have start to code my function. One think that I don't understand at this time is koala id. Can you please help me on this subject?

[tylerrmit]

The "koala id" is the same as the "Coviva id" that you used to login to the Coviva phone app. You'd also use it to login to Coviva's own web interface at mycoviva.net, or to use my homebrige plugin. It's a hardware id permanently associated with your Coviva hub.

On my hub, there is a cross section at the back that usually makes it stand up, but can be pulled off to reveal a USB port, a QR code, and the coviva id aka koala id written in hex above the QR code. That thing.

[Komet2005]

Ok thanks a lot. I've got also a problem with the Host parameter in the url when I try to get a response. Can you explain it to me please ?

[tylerrmit]

I'm sorry, I don't understand the question here.

The first phase of the script creates the authentication credential string in the required format, after all those urlencode/sha512 hash/base64 encoding operations.

The second phase is where it requests an access token. It is a "POST" style HTTP request. Line 67 of tools/get_all.py sets the URL for the POST request as http://XXX.koalabox.net/access_token, where XXX is the "coviva id" aka the "koala id" for your Coviva Hub. Lines 68-76 set the "headers" for the HTTP request. One header parameter is "Host", which is set to "XXX.koalabox.net", where XXX, again, is the "coviva id". Was that what you were asking about? Lines 77-83 show the "payload" that is actually "POST"ed to the URL along with the headers. Lines 84-88 execute the actual HTTP request. Lines 90-108 parse the response from the server into some expected name/value pairs, one of which is the critical "access_token" that was being requested.

Later, the "access_token" becomes part of the URL for the web socket request.

You will need to construct Javascript code in the Apps Script environment that will create and execute and exactly equivalent HTTP POST operation to the same URL with the same headers and the same payload, then parse the response to pluck out the "access_token".

[tylerrmit]

I imagine that Hager own the koalabox.net domain. (On "whois", all the contact details for the domain are redacted for privacy.) When the Coviva hub phones home to Hager, Hager are then able to ensure that requests to XXX.koalabox.net get diverted back to the appropriate hub. No idea why they called it "koalabox.net" but that's invisible to the ordinary consumer, so I guess they can call it whatever they want.

[Komet2005]

I understand but I used my "coviva id" to replace XXXXXXXXXXXX (12 caracthers) in the POST request with http://XXXXXXXXXXXX.koalabox.net/access_token for exemple but it returned a 401 error code instead oh a 200 (my coviva hub is an TKP100A).

[tylerrmit]

"The HyperText Transfer Protocol (HTTP) 401 Unauthorized response status code indicates that the client request has not been completed because it lacks valid authentication credentials for the requested resource."

Seems like you're getting through, but it is rejecting your request to authenticate. Perhaps you have not exactly replicated the request from the python demonstration code in your own code.

When you take tools/get_all.pay, edit the three hard-coded parameters to specify your email address, password, and Coviva id (set to 'xxx' by default so that I don't give away my own credentials)... Then run it. Does that work?

If the python works but your code doesn't, you will probably need to find someone or something that can help you translate the python into JavaScript.

I imagine the App Script stuff is running remotely on Google infrastructure so it's harder to trace with something like the Charles Web Debug Proxy. First thing I'd do is write a version that outputs the "Basic xxx" authentication string that is assembled with all those base64 etc operations and sent in the header. Print what your Apps Script generates, and print what the Python generates. If they don't match then you know there's a problem in that area.

If the python didn't work I'd ask you to check whether the web interface at "mycoviva.net" works for you.

Again, that web interface has JavaScript code to do everything if you disentangle it enough to understand it. I trawled through that JavaScript code to understand it and translate it into the Python test programs, then translated again into a Homebridge plugin.

[tylerrmit]

If you give it garbage for the Coviva Id, do you still get a 401? If you don't, that lends support to the idea that the 401 means you are getting through but your credentials aren't being accepted.

[Komet2005]

When i use garbage for the Coviva Id my code error is now 404. You're right, I'm getting through but my credentials aren't being accepted. I found the error (it was in the conversion code in base 64)

[tylerrmit]

Ok good. I remember I had to carefully trace things with "Charles" Web Debug Proxy until I had the encoding of the credentials byte-for-byte exactly the same as the official web application was doing. If you have that right, you should be well on your way.

Please let me know when you succeed and everything is working, otherwise I'll keep this issue open for a few more days in case you have more questions. If you end up posting your App Script code somewhere for others to enjoy, please respond here with a link to it.

[Komet2005]

No problem. I'll do it when I've tested all the elements.

[Komet2005]

Unfortunately, impossible to create a websocket in Google Apps Script.

I try to find a solution but it's really not easy. In order to try to solve the problem without, can you explain me the used system to send an instruction to the TRM694G for exemple ?

[tylerrmit]

I'm sorry, I think I'm at the limit of what I can do to help, if Google Apps Script doesn't support websocket requests.

The purpose of a websocket is to open up a persistent communication channel, where client and server can send messages back and forth. E.g. the server has an open channel in which it can tell the client "hey someone just physically turned the device on, it's now on", without being polled. Whereas a HTTP/HTTPS is transactional, and there'd be no way for the server to update the client out of the blue. So Coviva's own web interface -- a web version of the phone client -- uses a websocket to tell the Coviva hub when you want to switch a light on or off. It naturally uses a websocket so that the web interface can be kept up-to-date. So, in the Homebridge plugin, I'm replicating the communication method used by Coviva's own web interface, and I only know of one way to issue instructions to turn a Coviva device on or off: via the websocket. I imagine their phone apps and supported integrations like Alexa.

You don't really need the "stateful" connection, you'll know the node ID and you just want to turn it off. But the protocol is, you create a websocket connection, your commands go to your Coviva Hub, and your Coviva Hub is updating your TRM694G device via the KNX protocol. Everything goes via the hub.

I quickly tried adjusting the python script to see if there was an easy way to issue a "GET:all" command via HTTPS:// instead of WSS:// but it didn't work. 404 error. The HTTPS service at the Coviva hub doesn't recognise the "connection" page/service being requested.

I think you're going to have to keep researching ways to make Google Apps Script talk to a websocket URL, if only in a stateless way.

https://stackoverflow.com/questions/41370977/google-app-scripts-and-websocket

That stack overflow page has a comment suggesting a workaround:

You can however poll server-side functions using client-side asynchronous JavaScript API using google.script.run

Otherwise, worst case you might find yourself having to set up your own web service somewhere outside of the Google Apps Script platform, that can accept HTTPS requests from Google Apps Script and then forward them to the websocket.

[tylerrmit]

(Just in case it wasn't clear, I have no association with Hager/Coviva, this is not an official Homebridge plugin from them. I'm also not someone who can help with Javascript coding, the best I can do is explain how the official Hager/Coviva web application operates and how I have applied that to make an unofficial Homebridge plugin. If there is a way to talk to Coviva devices without using a websocket, I am not in a position to know.)

[Komet2005]

Thank you very much for your explanation.

[Komet2005]

Hello, Can you explain me the notion of Coviva profile numbers ?

[tylerrmit]

That's explained at least to some extent in the readme. I suggest you run the Python program that sends a "GET:all" or "GET:nodes" and have a close look at the JSON response (with something to give it "pretty" formatting for clarity)

Each device that is registered to your hub is a "node".

Each node has an "id" by which you refer to them. The first node id is typically "1", then "2"... it's probably allocated in the same order you or your electrician installed/registered them.

If you look at the JSON data structure for a node, you should see something that tells you it's "profile" id. This tells you what type of device it is, and THAT is going to influence which attributes the device has, what they represent, and what you can control.

E.g. I have devices in my house that control lights. Some of them have a "dimmer" module, some of them don't => that's two different hardware profiles. I registered Netatmo cameras and a Netatmo weather station to the Coviva app, so they have different profiles again -- one device can tell me if it's raining, but my dimmer switch sure won't!

So for each node you want to first look in its part of the JSON at the name -- what is the human assigned name for the device with node id "1"? And you want to look at the bit where it tells you the profile -- what type of device is it? Then, there will be one or more attributes.

The profile helps you understand what "attribute 4" and "attribute 5" are in the context of the device. What information can you read from a device under which attribute? Which of those attributes can be set?

There's no manual for which attributes are supported on which device, or what each attribute means. Look at the readme and I think you'll find that if a user has a device whose profile I don't recognise, they should send me some information from the logs so that o can help them figure it out, then add support for it in the Homebridge plug-in. I went through this process with a user to add support for blinds/curtains -- a type of device I don't have. It helps to e.g set a device to 100% open, 50% open, 0% open and request the status in each state to see which attributes varies depending on that. Then you have a candidate attribute to send a new % to, and the device should change.

You can imagine that a simple on/off device might only have one attribute that is set to 0/1. A dimmer switch might have on/off represented by 0/1 and a second attribute for % brightness from 0-100. A set of blinds might only have one attribute to represent % openness, since you don't really switch them off and on. A weather station might have many attributes to represent temperature, wind direction, wind speed... etc.

[tylerrmit]

I've tried to "set" the temperature and rain attributes on my Netatmo weather station, but those can't be set, and it continues to be too hot/too cold/too wet outside despite my request. So while many attributes can be set, some set requests will be ignored or rejected.

[Komet2005]

Hello, I have publish my code on https://github.com/Komet2005/GAS-Coviva-Hager

[tylerrmit]

You managed to get everything working to your satisfaction? Congratulations!

What approach did you take to solve the websocket issue?

Your repo is currently private, only you can see it. But if you ever release it I'll be sure to check it out. (Be careful making it public IF you have ever hard coded your credentials into it - even an old version of a file. One solution to that is to make a new repo from files that don't have the credentials, and making that one public. You don't want people digging through old versions and finding them.)

I'll leave this issue open for a few days in case you have anything else to add.

[Komet2005]

Unfortunately, I do not have a solution yet. I have make the code to get the access token and to use it but right now it's not possible to send on command to a TRM with the payload way. I hope to find a solution in days.