cannot parallelize light activation

[rkitover]

When I do

bridge.set_light([1,2,3,4], { 'hue': 33862, 'sat': 50, 'bri': 254, 'on': True })

What actually happens, is that every light slowly turns on in sequence, instead of all four lights turning on.

I tried to get around this by using Pool like so:

from multiprocessing import Pool
LIGHTS = 4
def change_state(light):
    b.set_light(light, { 'hue': 33862, 'sat': 50, 'bri': 254, 'on': True })

if on_state != current_state:
    pool = Pool(LIGHTS)
    pool.map(change_state, range(1, LIGHTS+1))

What happens when I do this is even worse, first one of the lights slowly turns on, then the other three turn on in parallel, and this takes even longer than the first example.

[natcl]

I can't seem to reproduce here, can you post your full code ?

[rkitover]

Sure, here is my script:

#!/usr/local/bin/python

import sys, logging
from multiprocessing import Pool
from phue import Bridge

LIGHTS = 4

logging.basicConfig()

b = Bridge('rafael_living_hue')
b.connect()
b.get_api()

current_state = [b.get_light(i, 'on') for i in range(1, LIGHTS+1)].count(True) == LIGHTS

on_state = True if current_state == False else False

arg = sys.argv[1].lower() if len(sys.argv) > 1 else None

if arg == 'on':
    on_state = True
elif arg == 'off':
    on_state = False

def change_state(light):
    b.set_light(light, { 'hue': 33862, 'sat': 50, 'bri': 254, 'on': on_state })

if on_state != current_state:
    b.set_light(range(1, LIGHTS+1), { 'hue': 33862, 'sat': 50, 'bri': 254, 'on': on_state })
#    pool = Pool(LIGHTS)
#    pool.map(change_state, range(1, LIGHTS+1))

This uses the first method, but if you comment out the third line from the bottom and uncomment the two lines from the bottom then it will use the Pool method.

[rkitover]

I was looking at the set_light code, and I had some thoughts.

What if we used an async IO lib to spawn the PUT requests simultaneously (using Twisted or the new builtin thing or whatever.) AND we did not request /state in the same request, but first fired off the commands without /state, and only then sent new /state requests and compared the result.

I could probably work on this and send you a PR, what do you think?

[natcl]

Let me do some tests on my end before. I can't seem to see that much lag on my network when setting multiple lights.

Another alternative which would probably be a better idea than to modify the lib would be to use the group functionality which is meant to control multiple lights at the same time with one call...

[rkitover]

@natcl

Thank you for that very excellent tip, this solved the problem I was having!

I also optimized my script some, and now things are working much better.

Here is my new script:

#!/usr/local/bin/python

import sys, logging
from phue import Bridge

logging.basicConfig()

b = Bridge('rafael_living_hue')
b.connect()
b.get_api()

on_state = None

arg = sys.argv[1].lower() if len(sys.argv) > 1 else None

if arg == 'on':
    on_state = True
elif arg == 'off':
    on_state = False

if on_state is None:
    current_state = b.get_group(1, 'on')

    on_state = True if current_state == False else False

b.set_group(1, { 'hue': 33862, 'sat': 50, 'bri': 254, 'on': on_state }, transitiontime=0)

This turns my lights on at the same time.

One thing that still bothers me is that the whole process of running this script to toggle the lights takes about a second, if I use a parameter it's a bit quicker but still about half a second or so, it'd be nice to make it faster.

I'm going to look for opportunities for optimization and reducing startup time, because I also have a WeMo switch running this script and it feels a but unnatural to have a delay when turning lights on or off.

Another issue is that the philips standard hue app does not allow creating or editing groups, you have to get the third party "Hue Lights" app or some such. But the group with id "1" seems to be the system "All Lights" group, so that can probably be relied upon.

[rkitover]

@natcl

I did some benchmarks, this initialization code:

#!/usr/local/bin/python

import sys, logging
from phue import Bridge

logging.basicConfig()

b = Bridge('rafael_living_hue')
b.connect()
b.get_api()

takes approximately 0.6s on average on my mac mini.

The full script I posted above takes on average 1.5s with no parameter for a toggle and between 0.7s and 1.1s with an explicit on or off parameter.

[DeastinY]

Hi @rkitover, This does not solve the issue with the long time the initialization takes, but what I did was having the script run on a raspberry, connecting once and then keeping that connection. This way you only have to to the connection once on startup. Maybe something along those lines is a feasible solution for you as well :) Cheers, DeastinY

[natcl]

You do not need to call b.connect() and b.get_api(), removing those should help.

[natcl]

@rkitover you can also create your groups with phue using the create_group method, no need for an app.

[rkitover]

@DeastinY

Funny thing is, right before you posted that I was already starting screw around with writing a server and client. So I have something working reasonably nicely now for this, except for the WeMo switch daemon, still working on that.

Here is the server:

#!/usr/local/bin/python

import os, os.path, sys, logging
from phue          import Bridge
from gevent        import socket
from gevent.server import StreamServer

# This is adapted from
# https://github.com/gevent/gevent/blob/master/examples/echoserver.py
#
# this handler will be run for each incoming connection in a dedicated greenlet
def handler(socket, address):
    global bridge
    fileno = socket.fileno()
    log.debug('New connection on %d' % fileno)
    # using a makefile because we want to use readline()
    rfileobj = socket.makefile(mode='rb')
    while True:
        line = rfileobj.readline()

        if not line:
            log.debug('client %d disconnected' % fileno)
            break

        msg = line.strip().lower()

        if msg == 'on':
            on_state = True
        elif msg == 'off':
            on_state = False
        elif msg == 'toggle':
            on_state = not bridge.get_group(1, 'on')
        else:
            log.error("received invalid message '%s' on %d" % (msg, fileno))
            socket.sendall("ERROR: invalid message '%s'" % msg)
            continue

        log.debug("processing '%s' on %d" % (msg, fileno))

        bridge.set_group(1, { 'hue': 33862, 'sat': 50, 'bri': 254, 'on': on_state }, transitiontime=0)

    rfileobj.close()

logging.basicConfig(format='%(asctime)s %(message)s')
log = logging.getLogger('toggle_hue_daemon')
log.setLevel(logging.DEBUG)

bridge = Bridge('rafael_living_hue')

if not os.path.exists(os.path.expanduser('~/run')):
    os.mkdir(os.path.expanduser('~/run'), 0700)

sock_file = os.path.expanduser('~/run/toggle_hue.sock')

if os.path.exists(sock_file):
    os.remove(sock_file)

server_sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
server_sock.bind(sock_file)
server_sock.listen(25)

log.debug('Starting hue control daemon on %s' % sock_file)
server = StreamServer(server_sock, handler)

try:
    server.serve_forever()
except KeyboardInterrupt:
    log.debug('hue control daemon exiting')
    exit(0)

And here is the client:

#!/usr/local/bin/python

import socket, os.path, sys

sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.connect(os.path.expanduser('~/run/toggle_hue.sock'))

arg = sys.argv[1].lower() if len(sys.argv) > 1 else 'toggle'

if arg not in ['on', 'off', 'toggle']:
    raise ValueError("command must be 'on', 'off' or 'toggle'")

sock.sendall(arg + '\015\012')

[rkitover]

Here is my daemon for listening to the WeMo switch, things are finally working nicely:

#!/usr/bin/env python

import logging, os, os.path, time, sys, gevent
from gevent               import socket
sys.path.insert(1, os.path.expanduser('~/src/ouimeaux'))
from ouimeaux.environment import Environment
from ouimeaux.signals     import statechange, receiver

def connect():
    global log, sock
    sock      = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
    connected = False

    while not connected:
        try:
            log.debug('connecting to hue daemon')
            sock.connect(os.path.expanduser('~/run/toggle_hue.sock'))
            connected = True
            log.debug('connection to hue daemon successful')
        except:
            log.error('connection to hue daemon failed, retrying')
            gevent.sleep(5)

def send(msg):
    global log, sock
    while True:
        try:
            log.debug("sending '%s' message to hue daemon" % msg)
            sock.sendall(msg + '\012')
            log.debug("message '%s' sent to hue daemon successfully" % msg)
            return
        except (socket.error, IOError):
            log.error('not connected to hue daemon, reconnecting')
            connect()

logging.basicConfig(format='%(asctime)s %(message)s')
log = logging.getLogger('wemo_switch_daemon')
log.setLevel(logging.DEBUG)

log.debug('starting wemo switch daemon')

connect()

log.debug('retrieving wemo environment')

env = Environment()
env.start()
env.discover(5)

switch = env.get_switch('rafael_living_wemo')

@receiver(statechange, sender=switch)
def switch_toggle(*args, **kwargs):
    if kwargs['state'] == 1:
        log.debug('received switch ON state event')
        gevent.spawn(send, 'on')
    else:
        log.debug('received switch OFF state event')
        gevent.spawn(send, 'off')

try:
    log.debug('entering event loop')
    env.wait()
except (KeyboardInterrupt, SystemExit):
    log.debug('exiting wemo switch daemon')
    exit(0)

[rkitover]

@natcl

Even though I don't have the original problem by switching to using groups, I was learning how to use gevent recently and I realized that if you wanted to do parallel requests for e.g. set_light() it is really, really easy with gevent (because, gevent is awesome!)

So here is an example of doing set_light in parallel with very little changes to the code:

diff --git a/phue.py b/phue.py
index 2512f9d..5a9931c 100755
--- a/phue.py
+++ b/phue.py
@@ -14,11 +14,13 @@ I am in no way affiliated with the Philips organization.

 '''

+import gevent, gevent.monkey
+from gevent import socket
+gevent.monkey.patch_all()
 import json
 import os
 import platform
 import sys
-import socket
 if sys.version_info[0] > 2:
     PY3K = True
 else:
@@ -687,11 +689,11 @@ class Bridge(object):
             if isinstance(light_id, int) or isinstance(light_id, str) or isinstance(light_id, unicode):
                 light_id_array = [light_id]
         result = []
+
         for light in light_id_array:
             logger.debug(str(data))
             if parameter == 'name':
-                result.append(self.request('PUT', '/api/' + self.username + '/lights/' + str(
-                    light_id), json.dumps(data)))
+                result.append(gevent.spawn(self.request, 'PUT', '/api/' + self.username + '/lights/' + str(light_id), json.dumps(data)))
             else:
                 if PY3K:
                     if isinstance(light, str):
@@ -703,9 +705,13 @@ class Bridge(object):
                             converted_light = self.get_light_id_by_name(light)
                     else:
                         converted_light = light
-                result.append(self.request('PUT', '/api/' + self.username + '/lights/' + str(
-                    converted_light) + '/state', json.dumps(data)))
-            if 'error' in list(result[-1][0].keys()):
+                result.append(gevent.spawn(self.request, 'PUT', '/api/' + self.username + '/lights/' + str(converted_light) + '/state', json.dumps(data)))
+
+        gevent.wait(result)
+        result = [x.value for x in result]
+
+        for r in result:
+            if 'error' in list(r[0].keys()):
                 logger.warn("ERROR: {0} for light {1}".format(
                     result[-1][0]['error']['description'], light))

And this is my test script:

#!/usr/local/bin/python

import sys, os.path
sys.path.insert(1, os.path.expanduser('~/src/phue'))
from phue import Bridge

bridge = Bridge('rafael_living_hue')

bridge.set_light([1, 2, 3, 4], { 'hue': 33862, 'sat': 50, 'bri': 254, 'on': (not bridge.get_light(1, 'on')) }, transitiontime=0)