earthcube / scheduler

Scheduling approaches related to gleaner tooling
Apache License 2.0
0 stars 0 forks source link

Create a (gleaner|nabu)(mode) docker service so we can use docker configs #45

Closed valentinedwv closed 11 months ago

valentinedwv commented 11 months ago

https://docker-py.readthedocs.io/en/stable/services.html

The only way we are going to get configs to work is to create services. volume persisting is not working.

https://docs.docker.com/engine/swarm/configs/#simple-example-get-started-with-configs

not sure we needed it, but configs can be templated https://docs.docker.com/engine/swarm/configs/#example-use-a-templated-config

valentinedwv commented 11 months ago

This needs to be translated to docker.py...

But here is the Postman testing using a POST to the Docker REST API via Portainer. (need to add an X-API-Key header) https://portainer. :443/api/endpoints/2/docker/services/create

This creates a "postman" service that runs once as defined by

  ```
  "Mode": {"ReplicatedJob": {
  "MaxConcurrent": 1,
  "TotalCompletions": 1
  }},
  ```

Once this is created, you can find the container by:

https://portainer. /api/endpoints/2/docker/containers/json?filters={"label":["com.docker.swarm.service.name=postman"]}&all=true

Using a docker config (in this case):

{
                "ConfigName": "gleaner-test2",
                "ConfigID":"mw7ghwth73v7de9xh21y3smhh",
                "File": {
                    "Name":"/configs/gleanerconfig.yaml",
                    "UID": "0",
                    "GID": "0",
                    "Mode": 444     
                }
            }    

You cannot just just the ConfigName you need to use the ConfigID So a step to lookup the ConfigID from the docker configs API is needed https://portainer. /api/endpoints/2/docker/configs?filters={"name":{"gleaner-test2": true}}

{
    "Name": "postman",
"Mode": {"ReplicatedJob": {
"MaxConcurrent": 1,
"TotalCompletions": 1
}},
    "TaskTemplate": {
    "ContainerSpec": {
        "Env": [
            "MINIO_ADDRESS=oss. ",
            "MINIO_PORT=443",
            "MINIO_USE_SSL=true",
            "MINIO_SECRET_KEY=",
            "MINIO_ACCESS_KEY=",
            "MINIO_BUCKET=",
            "SPARQL_ENDPOINT=https://graph. /blazegraph",
            "GLEANER_HEADLESS_ENDPOINT=http://headless:9222",
            "GLEANER_HEADLESS_NETWORK=headless_gleanerio"
        ],

        "Args":[
             "--cfg",
            "/configs/gleanerconfig.yaml",
            "-source",
            "iris",
            "--rude"
        ],
        "Image": "nsfearthcube/gleaner:dev_ec",
        "Labels": {
            "com.example.vendor": "Acme",
            "com.example.license": "GPL",
            "com.example.version": "1.0"
        },
        "Configs": [
            {
                "ConfigName": "gleaner-test2",
                "ConfigID":"mw7ghwth73v7de9xh21y3smhh",
                "File": {
                    "Name":"/configs/gleanerconfig.yaml",
                    "UID": "0",
                    "GID": "0",
                    "Mode": 444     
                }
            }  ,
       {
                "ConfigName": "nabu-test",
                "ConfigID":"zcy8tl4a7doht300v3h5dlymg",
                "File": {
                    "Name":"/configs/nabuconfig.yaml",
                    "UID": "0",
                    "GID": "0",
                    "Mode": 444     
                }
            }   
        ],
        "Dir": "/gleaner",
        "Networks": [
        {
            "Target": "headless_gleanerio"
        }],
        "RestartPolicy": {
            "Condition": "none",
            "Delay": 10000000000,
            "MaxAttempts": 10,
            "Window": 0
            }
    }
    },  
    "Networks": [
        {
            "Target": "headless_gleanerio"
        }]
}