HTTP protocol adaptor - Githubissues

sophokles73 commented 8 years ago

It would be great to have a protocol adapter for receiving telemetry data from devices using plain HTTP. The adaptor could simply take the payload from incoming requests and forward it to Hono's dispatcher using the client component.

This adaptor would be aprticularly useful for testing and demonstration pruposes but could be evolved into a fully functional REST adaptor including security, scalability, fail-over etc.

hekonsek commented 8 years ago

Hi Kai,

I will be happy to work on this.

However I believe that the adapter should have an option to choose between using connector client or "native" AMQP client. For some scenarios we would like to avoid passing messages to the EventDispatcher and rely only on a raw AMQP infrastructure.

By "native AMQP usage" I mean a scenario like this:

HTTP POST URL: http://example.com/foo
POSTED CONTENT: {"bar": "baz"}
HTTP HEADERS: H1=XXX

...should be translated to the following using AMQP 1.0 client...

AMQP 1.0 PATH: /foo
AMQP BODY: {"bar": "baz"}
AMQP HEADERS: H1=XXX

While for non-native mode we should just invoke connector client.

What do you think, Kai?

sophokles73 commented 8 years ago

I think it's great that you want to start to work on that :-)

I also think that we should ignore the (existing) connector client (hono-client module) altogether and just focus on the AMQP 1.0 interface. I have discussed this with Dejan on the mailing list to some extent and we seem to agree in that point.

So, from my point of view you should start with supporting the native AMQP 1.0 mode. However, we will need to start working on the server side (in Hono) then as well in order for the HTTP protocol adapter having something to connect to (apart from existing AMQP 1.0 based brokers).

Sounds good?

hekonsek commented 8 years ago

Definitely makes sense. I will work on this then.

sophokles73 commented 8 years ago

Cool :+1:

hekonsek commented 8 years ago

I have committed my vision of REST (aka HTTP) protocol adapter.

The adapter basically converts incoming HTTP requests/response into request/reply AMQP call.

The body POSTed via HTTP request is wrapped into AMQP body. HTTP headers are mapped into AMQP headers. HTTP URI is matched into AMQP address path.

In the current implementation I treat posted HTTP body as a bulk of binary data and I just wrap it into AMQP body. This is fine for some cases, however we should also implement non-binary mapping mode (which I would call "AMQP-native" mapping). AMQP-native mapping should parse incoming POSTed body (like posted JSON data), parse it and map it into AMQP data types (like primitive types, properties or maps). It should be en up to the user of the adapter to configure it you the mapping mode of choice.

I used Apache Camel library to perform the actual implementation of bridging between HTTP and AMQP. It is the most effecting way of implementing this logic of logic I'm aware of.

What do you think?

sophokles73 commented 8 years ago

@hekonsek,

I have just shortly scanned over the code but I am impressed with its compactness. This is a great start, I think :-)

BTW Have you checked whether the dependencies you introduced are ok with being used in an Eclipse project?

sophokles73 commented 8 years ago

@hekonsek,

I have been playing around with the HTTP adapter and tried to use it in conjunction with the Qpid Dispatch Router. However, this does not work and I suspect this is because we are currently implying request/response semantics on the message exchange, i.e. a client POSTing to the adapter expects to get back a result in the response body. This is reflected in the set up of the Camel routes which seem use InOut exchange pattern.

I think, however, that the main use case for the HTTP adapter is to upload telemetry data which would better be reflected by a PUT on a resource without any particular response to be expected apart from an HTTP OK indicating that Hono has actually received the data.

This would then also imply ExchangePattern.InOnly semantics for the Camel routes, I guess.

What do you think?

ppatierno commented 8 years ago

@sophokles73 I didn't try the adapter yet but I have a different point on the second part ...

The HTTP adapter should be fully functional for all IoT patterns so not only telemetry but command/control as well. (as side note, we haven't to forget the other two main IoT pattern like notification and inquiry). Consider that in the command/control pattern using HTTP, the device has to way to get a command from the service :

opening a long live HTTP connection (see long polling) : in that case the device executes a request and wait "indefinitely" for a response when a new command will be availble from a service. Of course, the response could be empty if no command is available for a timeout (the device should re-open the connection to get commands);
execute a polling : the device should poll Hono with HTTP request and get the command (or nothing) in the related reply;

It should be handle by the HTTP adapter. If we decide to use HTTP only for telemetry purpose, how we can control devices which are able to speak HTTP only ?

sophokles73 commented 8 years ago

I see your point. However, I think we should keep in mind our priorities here. Personally, I do not see that we will have many devices being connected via HTTP in real world scenarios (at least not at Bosch). And even if so, I doubt that these will support sophisticated mechanisms like long polling (Comet) or Server Side Events.

My main intent for the HTTP adapter was to have an easy way to pump data into Hono from the device side, in particular in test scenarios. The beauty being the pervasive support for a simple HTTP post/put in simple clients and browser plugins.

Having said that, I would like to focus on the telemetry direction first and get that supported and intergrated with Hono before we think about sophisticated push mechanisms via HTTP which will probably never be used by any meaningful amount of devices anyways.

ppatierno commented 8 years ago

It could be strange but a lot of IoT platforms (from big and little/medium companies) consider HTTP as a first citizen. A lot of companies who sells devices use HTTP for connection because it's well known, it's text-based, it's simpler to debug and so on. Of course, I don't agree with these companies because binary protocols like MQTT and AMQP are better for a lot of reasons but who is approaching to the IoT space finds HTTP as the simpler solution to connect. Consider for example how many mobile developers are migrating to IoT today and they know very well HTTP, REST architecture and related patterns. Most of developers (not me!) are lazy to study new stuff :-)

Speaking about polling and long polling it doesn't mean using complex stuff like Comet or Server Side Events. I come from the embedded development school and a simple TCP socket (with HTTP on top of it) is your friend :-)

IMHO supporting HTTP only for telemetry side could be a big lack in the Hono solution. Due to its nature, HTTP is a request/reply protocol.

Anyhow, we have to push with AMQP as main protocol for connection.

sophokles73 commented 8 years ago

We can always add support for the command & control direction later. In any case, I want to support telemetry right from the start and this should not require the usage of a full fledged message broker managing temporary reply queues for each device individually from my point of view...

hekonsek commented 8 years ago

Sorry for late answers. I'm travelling this week.

As far as Comet-like connection is concerned. I would create a separated adapter for that purpose. REST connector should operate on small messages and short connections in my opinion. However separated comet-like component could be useful as well and we should think about it.

While we would like to keep the request/reply semantic of the HTTP adapter for reasons mentioned by Paolo, I believe that we should also support scenario mentioned by Kai.

Maybe we can use header-based convention? For example if HONO_NOREPLY HTTP header is set to true, then we don't wait for the response from the AMQP channel? What do you think?

ppatierno commented 8 years ago

I don't know Comet so much but it seems to be a technology based on JavaScript so for web application and for enabling push to client from server side. Speaking about devices we have not to consider web/browser based devices but embedded devices where, if only HTTP protocol is supported, developers have an HTTP/REST client to use (if they are lucky but in a lot of scenarios they have a simple TCP socket and need to build HTTP on top of it from scratch).

I agree that for the telemetry pattern the HTTP response can be empty because only the response line is enough (like a 200 OK) to confirm the client that message was acquired by the server.

For command/control, why we need another adapter to support Comet ? I can imagine that Comet on client side opens an HTTP connection and keeps this connection open to allow server the push action (in our case to send command). IMHO, It means that a simple HTTP adapter could be enough for that. Of course in that scenario, the response won't be empty but will be filled with information about command to the device (it could be empty after a timeout).

Having two adapters means having two different endpoints to connect (on different ports I suppose). Having a single adapter means having a single endpoint; of course two connections (one for telemetry and one for command/control) but on different paths (ex. /telemetry/[device_id], /commands/[device_id]).

Waiting for a response could be related to the path not using an additional header.

Of course above are only opinions .... we are just discussing :-)

sophokles73 commented 8 years ago

@hekonsek,

Maybe we can use header-based convention? For example if HONO_NOREPLY HTTP header is set to true, then we don't wait for the response from the AMQP channel? What do you think?

I think that the REST adapter will always need to wait for the exchange (for the HTTP request at hand) with the AMQP 1.0 container to be settled. Only then it can return HTTP OK to the (REST) client, right?

BTW I was wondering what the https://github.com/rhiot/rhiot/issues/535 issue is supposed to mean. Can you explain a little what's behind this?

hekonsek commented 8 years ago

Hi Kai,

Yes, basically we will be using inOnly when HONO_NOREPLY = true and inOut otherwise.

Oh, and I'm OK with favoring Hamrest over Google Truth. :)

rhiot/rhiot#535 basically means that I'd like to remove REST support from Rhiot and use Hono protocol adapter when the latter is ready to be used.

sophokles73 commented 8 years ago

That's good to hear :-) In general we should be conservative with introducing new dependencies not the least because in order for us to use them we need to go through the Eclipse third party contribution process which you (and all other committers) should make yourself familiar with :-)

Introducing dependencies is usually not a big deal if we use a component (and version) that has already been introduced by any other Eclipse project. We can then simply piggy-back the corresponding existing IPZilla CQ (contribution questionaire) that was originally used to approve the component. However, if we want to use a component that has not been used before we need to create a new CQ and have the component's source code going throught the Eclipse IP process.

In any case we can only create a release of our project if we have approved CQs for all our dependencies. So, mind this the next time you add about 25 dependecies (incl. transitive deps) by using some convenience code in a simple test case ;-)

hekonsek commented 8 years ago

Noted. :) I will also issue CQs for all the necessary dependencies I included.

sophokles73 commented 8 years ago

Dominik and I have also created some CQs already so you might want to first search for existing CQs in IPZilla (you can restrict the search to the iot top level project and then iot.hono) before issuing new ones.

hekonsek commented 8 years ago

Thank you Kai. Can you take a look at [1] and let me what you think?

[1] https://github.com/eclipse/hono/commit/368ecaae165df07e834127e9e29ecd026ff6deb4

hekonsek commented 8 years ago

@ppatierno Working with long-running connections is much more complicated than simple request/reply client usage. That's why I would prefer not to mix these approaches in a single connector. In particular it is not so clear for me how to map message sent over the same connection to AMQP message. That would mix up the path/headers/body mapping convention we use in a current implementation of the adapter.

So I would opt for having a dedicated adapter for long running connections.

I propose to return to the topic after we are done with this simple request/reply adapter.

hekonsek commented 8 years ago

@sophokles73 Kai, where is a kind of central search page for CQs? This [1] doesn't seem to find everything.

[1] https://dev.eclipse.org/ipzilla/buglist.cgi?quicksearch=camel-core

sophokles73 commented 8 years ago

One of the weaknesses of IPZilla is that it does not support search by maven coordinates. Try searching for plain camel and you should be able to find multiple entries for Camel Core as well ...

hekonsek commented 8 years ago

Same here [1] :( .

@kartben Benjamin, How can I search for existing CQs? For example I'm interested in all CQs related to "camel".

[1] https://dev.eclipse.org/ipzilla/buglist.cgi?quicksearch=camel

sophokles73 commented 8 years ago

Go to https://dev.eclipse.org/ipzilla/query.cgi

and enter camel ...

hekonsek commented 8 years ago

OK, it works. Many thanks, Kai. :)

hekonsek commented 8 years ago

Action points to complete the first version of an adapter:

create main class
create Docker image
CQs (including jackson-databind [1])
add documentation

[1] https://dev.eclipse.org/ipzilla/show_bug.cgi?id=11094

sophokles73 commented 8 years ago

Excellent :-)

I think creating docker images would also require:

setting up a Hono repository on Docker Hub
have Eclipse webmasters install Docker Engine on the HIPP instance (if we want to be able to build there as well as on Travis)

hekonsek commented 8 years ago

DockerHub organization has been created - #14. I will add Travis CI Docker automation soon. :)

hekonsek commented 8 years ago

Added Docker image configuration.

Action points to complete the first version of an adapter:

create main class (using Spring Boot bootstrap)
add documentation
CQs:
- jackson-databind [1]
- jackson-annotations [2]
- jackson-core [3]

[1] https://dev.eclipse.org/ipzilla/show_bug.cgi?id=11094 [2] https://dev.eclipse.org/ipzilla/show_bug.cgi?id=11151 [3] https://dev.eclipse.org/ipzilla/show_bug.cgi?id=11152

sophokles73 commented 8 years ago

Hi @hekonsek,

Dominik and I have been working all week on implementing the Telemetry upload in Hono server using the Dispatch Router. Seems to work quite well so far.

It would be cool if we could try to connect the REST protocol adapter to Hono server now and see whether we can send messages downstream via REST ...

hekonsek commented 8 years ago

I will work on this as soon as I'm done with SASL authentication in Vert.x server.

sophokles73 commented 8 years ago

We have a Vert.x based REST adapter in module hono-adapter-rest-vertx. The original Camel based hono-adapter-rest will be removed if nobody wants to keep working on it.

eclipse-hono / hono

HTTP protocol adaptor #3