scramjetorg / transform-hub

Scramjet Transform Hub (STH) is a runtime supervisor that can run data processing programs called Sequences and manage local resources on any Linux server, Docker on small edge servers, and even large-scale Kubernetes clusters in the cloud or datacenters. It connects to Scramjet Spaces in Scramjet Cloud Platform.
GNU Affero General Public License v3.0
67 stars 8 forks source link

Add option to provide file with arguments to start/deploy sequence #971

Closed patuwwy closed 1 year ago

patuwwy commented 1 year ago

What?

Why? All parameters user can provide in command line can be supplied with config file.

Usage: seq-startup-config.json:

{
    "outputTopic": "test-topic",
    "args": ["arg1", "arg2"],
    "limits": {
        "memory": 64
    },
    "appConfig": { "a": 1 }
}

Test with args-config-test Sequence https://github.com/scramjetorg/reference-apps/blob/main/js/args-config-test/index.ts

si seq <start|deploy> --startup-config seq-startup-config.json
si seq deploy packages/args-config-test.tar.gz --startup-config seq-startup-config.json
Request ok: http://127.0.0.1:8000/api/v1/sequence status: 200 OK
{"_id":"0e55aae1-25a5-4f0d-82ac-1b7192a4c788","host":{"apiBase":"http://127.0.0.1:8000/api/v1"},"sequenceURL":"sequence/0e55aae1-25a5-4f0d-82ac-1b7192a4c788"}
Request ok: http://127.0.0.1:8000/api/v1/sequence/0e55aae1-25a5-4f0d-82ac-1b7192a4c788/start status: 200 OK
{"host":{"apiBase":"http://127.0.0.1:8000/api/v1"},"_id":"db06246c-996b-49f7-9f0a-be79c01413f2","instanceURL":"instance/db06246c-996b-49f7-9f0a-be79c01413f2"}
Done in 5.92s.
$ si topic get test-topic
Request ok: http://127.0.0.1:8000/api/v1/topic/test-topic status: 200 OK
config: {"a":1}
string, "arg1"
string, "arg2"

params from file can be overwritten with cli params, ie:

si seq deploy packages/args-config-test.tar.gz --startup-config seq-startup-config.json --output-topic other-topic
Request ok: http://127.0.0.1:8000/api/v1/sequence status: 200 OK
{"_id":"8c72da12-a69c-4fbc-aeb1-2d7fad84e140","host":{"apiBase":"http://127.0.0.1:8000/api/v1"},"sequenceURL":"sequence/8c72da12-a69c-4fbc-aeb1-2d7fad84e140"}
Request ok: http://127.0.0.1:8000/api/v1/sequence/8c72da12-a69c-4fbc-aeb1-2d7fad84e140/start status: 200 OK
{"host":{"apiBase":"http://127.0.0.1:8000/api/v1"},"_id":"e4c4ea28-0ab8-4760-a22e-d2d98fc7c4d4","instanceURL":"instance/e4c4ea28-0ab8-4760-a22e-d2d98fc7c4d4"}
Done in 6.04s.
$ si topic get other-topic
Request ok: http://127.0.0.1:8000/api/v1/topic/other-topic status: 200 OK
config: {"a":1}
string, "arg1"
string, "arg2"

Review checks:

These aspects need to be checked by the reviewer: