Add server-push functionality to REST API

[GoogleCodeExporter]

The REST API should be enhanced to allow server-push, i.e. notification of the 
client when some server event is happening.

For this, we would like to use Atmosphere for this and will hence have to 
upgrade Jersey as well to a more recent version (and dropping Java5 support 
along its way). This is a follow-up on issue 25.

Original issue reported on code.google.com by kai.openhab on 17 Sep 2011 at 11:26

[GoogleCodeExporter]

Hi Kai,
I think that this was a hard decision. Please let me know if you or someone 
else is allready working on atmosphere integration. I can do this immediately.

greetings from Bochum
Oliver

Original comment by oliver.m...@gmail.com on 21 Sep 2011 at 8:46

[GoogleCodeExporter]

Hi Oliver, no, neither me nor anybody else has started with this feature. Feel 
free to have a try :-)
The final API might need some discussion and mutual agreement - do you already 
have concrete ideas about the how and what to push from the server?

Greetings from holidays in Locarno ;-)

Original comment by kai.openhab on 21 Sep 2011 at 3:30

[GoogleCodeExporter]

Hi Kai, I will check what is possible at the weekend. 
My idea is to push everything what's now accessible over the REST API.
So if one connects over the REST channel he gets same result as before.
If one connect via websocket or comet he gets the push notifications.

Original comment by oliver.m...@gmail.com on 23 Sep 2011 at 10:22

[GoogleCodeExporter]

Sounds like a valid approach for the start and should hopefully be quite 
straight forward. Looking forward to it!

Original comment by kai.openhab on 23 Sep 2011 at 7:26

• Changed state: Started
• Added labels: Version-0.9.0
• Removed labels: Version-later

[GoogleCodeExporter]

Any news on this issue?

Original comment by mishoboss on 19 Oct 2011 at 11:01

[GoogleCodeExporter]

I'm working on this.  I used the last weeks to get an deeper look into 
atmosphere framework and to find the best way how to integrate it into openHAB 
without changeing the current REST API. You can expect the final release by end 
of november.

The new REST API will support push notifications over websocket, HTTP streaming 
and cometD.

Original comment by oliver.m...@gmail.com on 19 Oct 2011 at 1:21

[GoogleCodeExporter]

Hi Oliver, could you give us an update on the state of things? 

Original comment by kai.openhab on 27 Nov 2011 at 9:44

[GoogleCodeExporter]

Hi, nearly finished ;) I still have to review and edit the maven dependencies. 
I will check-in my code in the next days.

Original comment by oliver.m...@gmail.com on 28 Nov 2011 at 12:53

[GoogleCodeExporter]

The next days are over - where can I see the code? :-)

Original comment by kai.openhab on 6 Dec 2011 at 9:03

[GoogleCodeExporter]

Hi Kai,
sorry about the delay too much work and not enough time in the last few days. 
If I get home earlier today I will check in today, otherwise tomorrow.

Original comment by oliver.m...@gmail.com on 9 Dec 2011 at 12:53

[GoogleCodeExporter]

Hi Kai,
check my repository olivermazur-websocket-comet.
You can use the REST API as before, but if you add the HTTP-Header 
"X-Atmosphere-Transport" the request will be suspended. It will be resumed when 
an item has changed. You can subscribe to different topics.
The first one is the all items channel
http://localhost:8080/rest/items
the other one is the single item channel e.g
http://localhost:8080/rest/items/Light_GF_Corridor_Ceiling/

When subscribing to the all items channel you get notified when an arbitary 
item has changed.
When subscribing to the single item channel you get notified when the selected 
item has changed.

I will add a small test page tomorrow or on monday.

Original comment by oliver.m...@gmail.com on 10 Dec 2011 at 8:31

[GoogleCodeExporter]

Hi Oliver,

Thank you very much! I just had a brief look at it and have some questions, if 
you don't mind:
- What would you think of extending the support from only items to also 
sitemaps? The current UI uses a kind of push support for sitemaps as it 
requires to refresh whenever some item of the current page changes - I guess, 
this use case would also be interesting to other UI implementations, e.g. the 
planned Sencha tablet UI or also native clients.
- I guess the slf4j-api-1.6.1.jar was committed by mistake in the lib folder - 
or do we really need this for something?
- The MANIFEST.MF still says "Bundle-RequiredExecutionEnvironment: J2SE-1.5" - 
Afaik, the new Jersey version required Java6, right?
- Although I have added the required header, the response does not seem to get 
suspended - in the log, I get a "ERROR o.a.c.BlockingIOCometSupport[:213] - 
Unable to resume the suspended connection". Do you have any idea, what might be 
wrong here?

Cheers,
Kai

P.S.: Thanks as well for rebasing your change on the latest default commits - 
that made it very easy for me to pull your code!

Original comment by kai.openhab on 11 Dec 2011 at 10:15

[GoogleCodeExporter]

Thanks Oliver. I also have some questions as all this push and websockets 
technology is new to me, but I'm definitely looking to build a Sencha 2.0 based 
openHAB interface.

This is how I see things:
1. The interface looks for a given sitemap at /rest/sitemaps/<sitemap> via 
regular REST request, no need for server push here I think
2. The server returns the XML of the items and the client builds the visual 
interface
3. Every item has a "link" tag which is used for status updates or sending new 
values. And here comes my question about the two approaches I think about:

----- 3.1. Every item makes a new connection to the server via regular REST 
polling with given interval or via websocket with server push if it's supported 
by the client. This way if there are 100 items, you have to make 100 HTTP 
requests every few seconds or you have to maintain a 100 socket connections.

------ 3.2. You use /rest/items and poll only this resource that feeds with 
data all items in the UI. This way you have only one HTTP request every few 
seconds (or you could do it even every second) or only one socket connection. 
However you need to do additional logic to parse and update the items on client 
side.

Which of this approaches you see as the better approach and why?

Original comment by mishoboss on 12 Dec 2011 at 1:37

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Mihail, very simple, the best approach is to use the sitemap and that's why I 
have asked Oliver to extend the server push support to it.
As explained on the REST wiki, you can access parts of sitemaps (I call it 
"pages") like this: http://localhost:8080/rest/sitemaps/demo/FF_Bath
This request will return the items and their states of a certain page the user 
is looking at. If you add a suspend here, it means that the client only has to 
open this single connection and will be notified if (and only if!) there is a 
status change for an item of the current page.

Original comment by kai.openhab on 12 Dec 2011 at 1:44

[GoogleCodeExporter]

Ahhh, yes. This simplifies a lot of things. Looking forward to see it.

Original comment by mishoboss on 12 Dec 2011 at 2:32

[GoogleCodeExporter]

No prob. I will extend the sitemap asap. From wednesday I have more time to do 
this.

Original comment by oliver.m...@gmail.com on 12 Dec 2011 at 10:54

[GoogleCodeExporter]

Hi all, 

As Kai and Thomas know I am working on an iOS native interface for openHAB. I 
would be really interested in a brief explanation of the technology involved in 
this and  a little how-to to do some initial tests with iOS. How do i suscribe 
to a notification? How do I work with it? Can I choose which items do I want to 
get notifications for?

Thanks in advance,

Pablo Romeu

Original comment by pablo.ro...@gmail.com on 13 Dec 2011 at 9:52

[GoogleCodeExporter]

Hi Pablo,

for what you can request via REST, please see the wiki: 
http://code.google.com/p/openhab/wiki/REST
This issue here will only add a "suspend" mechanism, so that the server only 
returns the response as soon as some values have changed.

Original comment by kai.openhab on 13 Dec 2011 at 11:16

[GoogleCodeExporter]

Hi Kai,

And if the value continuously changes, it will send the values? 

Also, keep in mind that the push notification service for iOS is fully managed 
by an Apple server. Then, the app cannot suscribe to a third party websocket 
for remote notifications, that's why I am asking for the functional details. 

If a remote push notification (not in local network) is needed then we will 
need a server capable of registerong to the desired websocket and then, sending 
the push notification through the Apple Notification Service

Original comment by pablo.ro...@gmail.com on 13 Dec 2011 at 11:43

[GoogleCodeExporter]

Hi Oliver,

> I will extend the sitemap asap. From wednesday I have more time to do this.

When do you expect to finish that? Please note that I meanwhile did a couple of 
changes to the REST API, I hope it does not cause any bigger merge problems for 
you. Let me know, if you have any questions regarding my changes or if I can be 
of any help.

Cheers,
Kai

Original comment by kai.openhab on 18 Dec 2011 at 9:47

[GoogleCodeExporter]

Hi Kai,

you can expect the next update on thursday. A testapp will be included. 
I've stil some issues with websocket support. HTTP streaming and long-polling 
will work. Moreover I will switch back to an earlier version of the atmosphere 
framework. The current version throws Message Body writer exceptions when 
dealing with List. objects. 

Original comment by oliver.m...@gmail.com on 20 Dec 2011 at 10:46

[GoogleCodeExporter]

Hi,
adding the sitemap subscription feature to the test app took a littel more time 
than expected so I'm not able to merge today, but tomorrow.

Original comment by oliver.m...@gmail.com on 22 Dec 2011 at 10:10

[GoogleCodeExporter]

Ok, so I won't review it tonight anymore and will go to bed instead :-)

Original comment by kai.openhab on 22 Dec 2011 at 10:14

[GoogleCodeExporter]

I just noticed now that you have pushed your changes on Dec 24 - thanks for the 
christmas present ;-)
So I assume that it is ready for me to review and integrate? Will try to do so 
tonight.

Original comment by kai.openhab on 30 Dec 2011 at 10:10

• Added labels: Priority-MediumI
• Removed labels: Priority-Medium

[GoogleCodeExporter]

There is a bug with the Comet suspended connection. Please see this issue: 
http://code.google.com/p/openhab/issues/detail?id=60

Original comment by mishoboss on 4 Jan 2012 at 8:12

[GoogleCodeExporter]

I'm back from my ski trip. I will check the posted issues at the weekend.

Original comment by oliver.m...@gmail.com on 6 Jan 2012 at 9:45

[GoogleCodeExporter]

Give me until tonight, I am just doing a major refactoring of the code myself - 
would be nice if you could review this on the weekend then!

Original comment by kai.openhab on 6 Jan 2012 at 9:49

[GoogleCodeExporter]

Ok, just committed my changes: 
http://code.google.com/p/openhab/source/detail?r=bfb3c8a1e22a8dfc5fdde4c61340858
5ba0999f1

Oliver, fyi: I mainly changed two things (which resulted in rewriting most of 
the code :-():
1. The way state change listeners are used - instead of using the 
RESTApplication as a listener on ALL available items, I only register single 
requests on the items that are concerned by this request. This is imho 
"lighter" - if a user does not use REST/atmosphere at all, he won't have any 
additional listeners on all his items, so it is least intrusive.
2. As far as I could see, your implementation always only broadcasted an 
itemBean (the one where the status had changed). Instead, the new 
implementation always returns the same result as a normal response to the uri - 
just that it is suspended and resumed inbetween. The response will a a plain 
text item status, a full item info, a page of a sitemap and its format might be 
XML, JSON or JSONP, just as on normal requests (that gave me most headache...)

As I have never used the atmosphere framework before, I am not sure if my code 
could be improved or simplified - so I welcome any feedback you might have!

P.S.: I still have this "unable to resume" error on normal (unsuspended) 
requests - do you have any idea where this comes from and how we could get rid 
off it?

Original comment by kai.openhab on 6 Jan 2012 at 9:48

[GoogleCodeExporter]

Hi Kai,

you should remove the resumeOnBroadcast = true attribute from the @Suspend 
command.
By removing the resumeOnBroadcast the Atmosphere framework will decide when to 
resume the suspended connection. HTTP streaming should work again if you do so.

to 1. full agree :)

to 2. I'm not sure if this is the best solution. If I subscribe to a page and 
the server publishes new values I have to check / compare the state of each 
item in the subscribing application. Normally a subscribing application is only 
intrested in items with changed values. To handle both approches we can add 
some (sub)resources to the API.
Like:
http://localhost:8080/rest/sitemaps/demo/FF_Corridor  -  just return the result
http://localhost:8080/rest/sitemaps/demo/FF_Corridor/SubscribePage  -  
subscribe and broadcast pages to this channel
http://localhost:8080/rest/sitemaps/demo/FF_Corridor/SubscribeItems  - 
subscribe and broadcast only items on this page

this should also resolve the "unable to resume" error. The "unable to resume" 
error 
is thrown when a client connects through a classic request. In this case the 
request should be suspended first. Then the TransportListener.OnSuspend() 
method should resume the request, but it seems that the connection is not 100% 
suspended at that point. There are a few possible solutions for this:
1. separate classic and subscribing requests as described above
2. an own AtmosphereResourceHandler
3. suspend and broadcast immediately to this request
4. try to suspend only subscribing connections
...

Original comment by oliver.m...@gmail.com on 10 Jan 2012 at 11:08

[GoogleCodeExporter]

Hi, Oliver. As you probably know I develop a new openHAB UI and I use the 
suspended connections. What I do is this:

On page change, the app first cancel the previous ajax call (if any) and make a 
regular ajax request to get items data immediately. Then it makes a suspended 
connection of "long-poll" type and waits for new data. On new data or 3 minutes 
timeout it makes another suspended connection. This works fine so far.

> to 2. I'm not sure if this is the best solution. If I subscribe to a page and 
the server publishes new values I have to check / compare the state of each 
item in the subscribing application. Normally a subscribing application is only 
intrested in items with changed values.

Yes, you're absolutely right here. In fact what I suggested before some time is 
a new channel that supply with new data only changed items and only with data 
that changes in a nice and easy to parse format. You can see my suggestion 
here: http://code.google.com/p/openhab/issues/detail?id=56 

Right now I update all page widgets with the new data if only one has changed 
it's value. I don't compare anything, just update them all. It works pretty 
good I must say. There is no noticeable delay because of performance issues. 
However to make what you suggest we need some unique widget identification, so 
the app will know which widget to update with the new data. Such an ID doesn't 
exists right now.

And a little bit different topic - I need some info about the websocket 
implementation. URL to connect, port, data format (HTTP packet or...)? Please, 
feel free to contact me on my mail. Thank you.

Original comment by mishoboss on 11 Jan 2012 at 6:52

[GoogleCodeExporter]

> you should remove the resumeOnBroadcast = true attribute
> HTTP streaming should work again

Hm, I actually do not want streaming to work for these URIs as it imho does not 
make sense for them (they are simply supposed to return the same result as on a 
normal request).

> I'm not sure if this is the best solution.

It is a first step to introduce server-push by simply not adding any new kind 
of resources/data types, but simply adding suspend support to what is already 
there.
This already gives UIs the possibility to get notified by the server and they 
can receive all required information. As the page data is not too much, the 
performance overhead is usually also acceptable (as you can see from 
mishoboss's comment).

Still, I understand your point that other subscription types (which then could 
also support streaming mode) make sense. But I would like move this 
implementation to after the 0.9.0 release (which is scheduled for in 10 days 
from now), especially as it might need some further discussions and 
specifications (see mishoboss's idea around item IDs etc.)

> The "unable to resume" error is thrown when a client connects through a 
classic request.

Right. As a solution, I would probably prefer your solution 4 (try to suspend 
only subscribing connections). The problem is that atmosphere automatically 
suspends in any case as there is the @Suspend annotation. Do you know if one 
can achieve the suspension by code as well? Then we could check for the 
X-Atmosphere-Transport parameter and decide on this, whether we want to suspend 
the request or not.

Original comment by kai.openhab on 12 Jan 2012 at 3:49

[GoogleCodeExporter]

Hi Kai,
some supplementary notes from my side. As I wrote in comment 22 there are some 
issues in the websocket support. In the current setup websocket won't work and 
the server returns "501 not implemented". Now I was able to solve this by 
adding org.eclipse.jetty.continuation, org.eclipse.jetty.util, 
org.eclipse.jetty.websocket to the Imported-Packages (Manifest.mf). With this 
jetty will find the libs and is able to handle websockets. 

But that's not all. The Websocket connections will run in the same troubles as 
the HTTP streaming connections. To solve this you can perform the following 
steps:

1. don't resume Websocket requests on broadcast (remove @resumeOnBroadcast)
2. when the connection is not resumed don't unregister the StateChangeListener 
in the onBroadcast event. 

Moreover I've updated my workspace to atmosphere 0.8.3 without issues yet 
(still testing) and I'm looking for a nice "unable to resume" solution.

Original comment by oliver.m...@gmail.com on 13 Jan 2012 at 4:57

[GoogleCodeExporter]

Hi Oliver,
Thanks for this analysis - feel free to do these changes in your clone, I will 
then pull the changeset from you!
Would be nice if we could have this stable by mid of next week, so it is in the 
right shape for the 0.9.0 release.

Original comment by kai.openhab on 13 Jan 2012 at 9:48

[GoogleCodeExporter]

hi Kai you can find the modifications in my clone. With atmosphere 0.8.3 it 
seems that the "unable to resume" error is reduced or gone.

Original comment by oliver.m...@gmail.com on 17 Jan 2012 at 2:27

[GoogleCodeExporter]

Very good news, thanks! Will pull the changes either today or tomorrow.

Original comment by kai.openhab on 17 Jan 2012 at 2:34

[GoogleCodeExporter]

I have just reviewed, tested and pulled your changes into the official repo. 
Thanks a lot!

Original comment by kai.openhab on 17 Jan 2012 at 5:32

• Changed state: Implemented

[GoogleCodeExporter]

Something was broken with http long-polling between 24.04.2012 and today. The 
main problem is when several long-polling requests are open for the same 
sitemap page the whole mechanism goes very wrong. Unpredictably it can send or 
not send responses to selective requests when changes are happening. 
Additionally, is there any documentation for server push available?

Original comment by belovic...@gmail.com on 24 Jun 2012 at 7:36

[GoogleCodeExporter]

I'd guess that this is related to the merge of issue 62, which I did yesterday.
@Oliver: Could you please check if there is now a problem with long-polling? If 
so, I might better revert the merge.

Original comment by kai.openhab on 24 Jun 2012 at 2:06

[GoogleCodeExporter]

you are right. The mechanism to prevent double broadcasts for group events also 
blocks some long-polling clients. If I remove the double broadcast protection 
everything works like expected, but it's possible that you will receive several 
answers if an group event occured.
Hope I can post a fix tomorrow otherwise I will give you an update.

Original comment by oliver.m...@gmail.com on 24 Jun 2012 at 6:56

[GoogleCodeExporter]

Ok, it seems that the problem is a little bit more complicated. Let us say an 
"all off" event is executed and we have 10 active lights. Openhab will switch 
off all lights one after another. Each light will raise the group event. If the 
connection type is streaming or websocket you receive many events until all 
lights are switched off. If we use long-polling the situation is similar but 
some messages will be lost during the reconnect, so it is possible that the 
last received message still contains some active lights. 
In the previous REST API version we have a small sleep interval with the hope 
that all group events are processed when the thread is resumed. But in this 
case we have also no guarantee.
Maybe someone can give me a hint how this is solved in other bundles. 
To solve the current issue I suggest the following:
1. I will prepare a version without the double broadcast protection -> it`s 
still possible that some long polling messages will be lost.
2. I will work on a BroadcasterCache. The BroadcasterCache is an Atmosphere 
feature but we have to cutomize it for our use case. -> no messages will be 
lost but we still have unneccessary broadcasts
3. We have to work on a solution for the group events

Original comment by oliver.m...@gmail.com on 25 Jun 2012 at 11:06

[GoogleCodeExporter]

I am not sure I can follow all details. Why does the work on the streaming 
support impact the long-polling? We deliberately chose the solution of a delay 
for the long-polling to solve the group event problem (which is not a nice 
solution, but the best we were able to come up with).
So first prio should be to get the long-polling in a working state again - so 
that the existing functionality works as before. Is there any quick way to 
achieve this?

Original comment by kai.openhab on 2 Jul 2012 at 7:40

[GoogleCodeExporter]

Hi Kai,
yes, I will add the long-polling delay again tomorrow.

Original comment by oliver.m...@gmail.com on 2 Jul 2012 at 7:49

[GoogleCodeExporter]

Finished, but I will test all use cases again before I commit the sources.

Original comment by oliver.m...@gmail.com on 4 Jul 2012 at 8:44

[GoogleCodeExporter]

I've comitted a few minutes ago. Long polling respectively non streaming 
connections will be delayed (300ms) again. But as some tests shows me this 
delay is not a guarantee that you will receive the correct result. The problem 
is the same as before: A group event like "All Off" will raise many item 
events, because each item in the group has its own listener and all these 
listeners will broadcast to the client, but if we use long-polling it is 
possible that we lost some messages during reestablishing the connection. To 
reduce this I added the 300ms delay again. With this delay openHab has the 
chance to process the group event and afterwards openHab will broadcast to the 
clients (the API will filter double posts). 
This will work in most cases but not in all.

. An example:

- 6 lights = on
- "All Off" triggered
- 300 ms delay 
- 6 lights = off
- sent to client "All lights [0]"

-----------------------

- 33 lights = on
- "All Off" triggered
- 300 ms delay 
- 28 lights = off
- sent to client "All lights [5]"
     -> delay to short
- long-polling disconnect
- 33 lights = off
- long-polling connect
     -> message lost  "All lights [0]"

As you can see this is not an 100% solution, but it should be the same as 
before.
Currently I'm working on a BroadcasterCache implementation for openHAB to solve 
this problem.

Original comment by oliver.m...@gmail.com on 6 Jul 2012 at 11:15

[GoogleCodeExporter]

Thanks Oliver, I have pulled your changeset.
@Victor: Could you please verify that HABdroid works again as before?
Cheers, Kai.

Original comment by kai.openhab on 7 Jul 2012 at 12:52

[GoogleCodeExporter]

Ok, will check it a little bit later in the evening today and report.
The only clear solution for long polling, I believe, can be creating some sort 
of queues where events are pushed to, and long-polling client polls events from 
this queues then...

Original comment by belovic...@gmail.com on 8 Jul 2012 at 1:48

[GoogleCodeExporter]

Now it works as it used to! Thanks!

Original comment by belovic...@gmail.com on 9 Jul 2012 at 5:42

[GoogleCodeExporter]

Original comment by kai.openhab on 9 Jul 2012 at 5:45