Polyglot application with docker

[paulux84]

Hi all, one of the main advantages of microservices is the possibility to implement the microservices with different technologies. In SCDF doc I read that I must use Spring Cloud Stream based application. With a simple test i can see that SCDF use "SPRING_APPLICATION_JSON" environment variable to configure the docker (see example below):

"SPRING_APPLICATION_JSON":"{
      "endpoints.jmx.unique-names":"true",
      "spring.cloud.dataflow.stream.app.label":"scriptable-transform",
      "endpoints.shutdown.enabled":"true",
      "spring.cloud.stream.instanceIndex":"0",
      "scriptable-transformer.language":"python",
      "scriptable-transformer.script":"import os\\\\nimport json\\\\npayload = json.dumps(os.environ)",     "spring.cloud.stream.metrics.properties":"spring.application.name,spring.application.index,spring.cloud.application.*,spring.cloud.dataflow.*",
      "spring.application.index":"0",
      "spring.cloud.dataflow.stream.name":"py",
      "instance.index":"0",
      "spring.metrics.export.triggers.application.includes":"integration**",
      "spring.cloud.stream.metrics.key":"py.scriptable-transform.${spring.cloud.application.guid}",
      "spring.cloud.stream.bindings.input.group":"py",
      "spring.cloud.stream.bindings.output.producer.requiredGroups":"py",
      "server.port":"38109",
      "spring.cloud.stream.bindings.output.destination":"py.scriptable-transform",
      "spring.cloud.stream.kafka.binder.zkNodes":"zookeeper:2181",
      "spring.cloud.dataflow.stream.app.type":"processor",
      "spring.cloud.application.guid":"38109",
      "spring.cloud.stream.bindings.input.destination":"py.time",
      "spring.cloud.stream.kafka.binder.brokers":"kafka:9092",
      "spring.jmx.default-domain":"py.scriptable-transform",
      "spring.cloud.application.group":"py"
   }"

So...I think...if I would add a non spring application inside SCDF i must "expose" this configuration from my non spring app; but...what about "property whitelisting". I obviously can set non whitelisted properties manually, but what aboud set withelisted properties with gui properties editor form for non spring docker?

Thanks

[markpollack]

Hi, I'm exploring this area now (python + kafka) and haven't gotten much past SPRING_APPLICATION_JSON in practice. At the time when we introduced the companion jar artifact for metadata, docker registries didn't support arbitrary metadata in the image that well, but now it seems that one can store arbitrary text as part of the manifest and hopefully also not have to download the entire image to get the manifest. We can look into that more via the use of labels in the manifest, e.g.

{
    "schemaVersion": 2,
    "target": {
        "mediaType": "application/vnd.docker.container.image.v1+json",
        "size": 7023,
        "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7",
        "labels": {
            "createdAt": "2015-09-16T16:29:07.952493971-07:00",
            "whitelist-properites": "JSON GOES HERE"
         }
    },

[markpollack]

BTW, what are you experimenting with more specifically?

[paulux84]

Hi, nice to meet you. In this moment I'm not experimenting nothing special, I'm simply conducting an analysis with the aim of highlighting strengths and weaknesses for use in a big microservices system. Obviously I would highlight also the effort needed for reduce the gap between what I think we must have and what I can do now with SCDF. I think that SCDF have big potential, but I also think that he needs to mature a little bit more in direction of decoupling specific microservice implementation and SCDF implementation (Spring)... Moreover the documentation about this specific problem lack (to see the environment needed by SCDF to set microservice I've created a python producer that serialize env as json for a simple console log sink). My next step is to try whitelabeling as you suggested.

[sabbyanandan]

Hi, @paulux84. We have experimented native Python workloads in the form of Stream, Task, and Composed Task. All of those experiments were presented as recipes in the Microsite, too.

See: https://dataflow.spring.io/docs/recipes/polyglot/

Please have a look at them and let us know if you have any thoughts on the usability and generally any other feedback.