snowplow / dataflow-runner

Run templatable playbooks of Hadoop/Spark/et al jobs on Amazon EMR
http://snowplowanalytics.com
19 stars 8 forks source link

Add tags to playbook format #27

Closed alexanderdean closed 7 years ago

alexanderdean commented 7 years ago

A top level property, typed as a map[string]string:


      ...
      "secretAccessKey": "env"
    },
    "tags": {
      "client": "Acme Inc",
      "environment": "staging"
    },
},
alexanderdean commented 7 years ago

Shout if you have any questions about this one @BenFradet !

BenFradet commented 7 years ago

should be straightforward enough

BenFradet commented 7 years ago

Just so we're on the same page, there is nothing on the side of EMR that would reflect this (i.e. you can't add tags to your jobflow steps).

BenFradet commented 7 years ago

Also tags in the cluster config are in the format which EMR expects:

    "tags": [
      {
        "key": "client",
        "value": "com.engineering"
      },
      {
        "key": "job",
        "value": "main"
      }
    ]

should we keep the same format?

alexanderdean commented 7 years ago

Yep let's keep the same format, and agree - the tags in the playbook don't make it into the cluster...