Create microservice framework and samples

IBMStreams / streamsx.microservice

Provide documentation, utilities, examples for Streams microservices

Apache License 2.0

0 stars 3 forks source link

Create microservice framework and samples #5

Open chanskw opened 6 years ago

chanskw commented 6 years ago

Initial work to create sample implementation of Streams microservices

chanskw commented 6 years ago

@ddebrunner @cancilla Please see initial contributions and let me know if you have any comments.

ddebrunner commented 6 years ago

One comment is I think the publishing as JSON using gson should be in topology as a standard feature, rather than separately in this toolkit.

chanskw commented 6 years ago

completely agree... I think we did this before your json support was available.

ddebrunner commented 6 years ago

I couldn't see an overview of what is being achieved with the AbstractService concept, but maybe I missed it in the 64 files being modified. I think that's probably a good idea but a few thoughts jump out at me:

How do I run these without gradle, e.g. should we have either:
- model of a executable jar
- a runner that can take a class name (like Beam & the Python streamsx.runner)
We may want to create the microservice abstraction at a higher level than one that depends on topology, e.g. what if my microservice is a Beam app, do I have to use an different approach?
What is a service, ie. we need a model of what we are trying to achieve with this code.

chanskw commented 6 years ago

Did you see the description from here? https://github.com/IBMStreams/streamsx.microservice/tree/develop/java/com.ibm.streamsx.microservices.common

chanskw commented 6 years ago

@cancilla and I have been questioning about the same things about a microservice runner. It feels like we are missing that high-level infrastructure to define - connect and launch microservices.

We picked gradle because it was easiest for us. But I think we need to have this standardization. It will be important to have this for integration of microservices with other tooling that allow users to reuse microservices... and allow them to create pipeline using microservices.

ddebrunner commented 6 years ago

Having a Java service be an SPL toolkit seems strange to me, it's not an SPL toolkit and will likely fail if spl-make-toolkit is used on it.

ddebrunner commented 6 years ago

All services publish data in JSON format, to maximize service interoperability with different languages.

I don't think this is the reason for using JSON, SPL/String tuples have the same level of interoperability as JSON (and eventually all will except language specific types).

I think the reason for using JSON is to allow schema evolution.

I don't think we should say all microservices must use JSON, it's a choice between flexibility and performance that the application must make.

ddebrunner commented 6 years ago

Thus the versioning in a topic name only applies when publishing as JSON.

ddebrunner commented 6 years ago

I'm not sure about a service.properties, or at least we need to think why we are adding a new file format when we have an existing one, the job config overlay. I think of a microservice as a pair of {topology/main composite, job config overlay}.

We have one concrete use case where a customer could use a JCO to support features that were not available through the topology apis, thus any microservice representation must support JCO.

ddebrunner commented 6 years ago

I do think it's good to have this as an initial proposal, to provide something concrete to drive discussion.

ddebrunner commented 6 years ago

I guess you could add dependencies into the {topology/main composite, job config overlay} definition , but currently all dependencies (I think) can be specified by the topology/main composite.

chanskw commented 6 years ago

@ddebrunner can you show me some details about job config overlays? I have never worked with them... but want to take a look.

ddebrunner commented 6 years ago

Basically a JSON document used at job submission to define runtime behaviour including job name, fusing, number of target pes, host tag remapping, .....

Topology always submits with 4.2 or the service the job with a JCO containing any runtime configuration. Even when producing a bundle a JCO is also produced.

https://www.ibm.com/support/knowledgecenter/en/SSCRJU_4.2.1/com.ibm.streams.admin.doc/doc/job_configuration_overlays.html