kube-HPC / hkube

🐟 High Performance Computing over Kubernetes - Core Repo 🎣
http://hkube.io
MIT License
307 stars 20 forks source link

streaming pipeline with datasource #1149

Closed tamir321 closed 3 years ago

tamir321 commented 3 years ago

HKube micro-service api-server

Describe the bug when trying to save datasource in a streaming pipeline you are getting the "@" sign is not allowed in "stream" pipeline, please use the "streaming.flows" property instead

Expected behavior try to save the folwing pipeline


{

    "name": "stream-dataSource1",
    "description": "",
    "kind":"stream",
    "nodes": [
        {
            "nodeName": "A",
            "kind": "dataSource",
            "dataSource": {
                "name": "new1-test"
            }

        },
        {
            "nodeName": "dataSourceNode",
            "algorithmName": "data-source",
            "stateType":"stateful",
            "input": [
                "@A"
            ]
        },
         {
            "nodeName": "get-all",
            "stateType":"stateful",
            "algorithmName": "get-and-sen-all"

        }
    ],
    "flowInput": {},
    "triggers": {
        "cron": {
            "pattern": "0 * * * *",
            "enabled": false
        },
        "pipelines": []
    },
    "options": {
        "batchTolerance": 100,
        "concurrentPipelines": {
            "amount": 10,
            "rejectOnFailure": true
        },
        "progressVerbosityLevel": "info",
        "ttl": 3600
    },
    "priority": 3,
    "streaming":{
        "master":"dataSourceNode >> get-all"
    },
    "experimentName": "main"
}
tamir321 commented 3 years ago

tested on systemVersion: "v2.0.128", fullSystemVersion: "v2.0.128-1618153188549",

tamir321 commented 3 years ago

tested on systemVersion: "v2.0.128", fullSystemVersion: "v2.0.128-1618153188549",

tamir321 commented 3 years ago

tested on systemVersion: "v2.0.128", fullSystemVersion: "v2.0.128-1618153188549",