task ordering is not well defined

[danielnorberg]

digdag tasks are defined in a key-value map and digdag executes tasks in the literal order of the text in the yaml file. This can be fragile as key-value maps in YAML do not have a well defined semantic order.

http://yaml.org/spec/1.2/spec.html#id2765608

Some examples of how this can become painful:

• Using a YAML editor etc to edit a workflow file might cause tasks to become reordered and break the workflow.
• Parsing and serializing the workflow using a YAML library etc is not safe and will break workflows.
• Generating a workflow becomes harder, as it is not possible to represent a digdag workflow as e.g. a map/hash/dictionary in some programming language and serialize it to YAML. Instead non-insignificant amounts of custom logic might need to be written to represent and serialize tasks in the correct order.

Suggested solution alternatives:

1. Explicitly define digdag workflow files to not be YAML and change suffix to e.g. .digdag or .dd etc. We can still say that syntax is YAML and people can configure editors to do YAML syntax highlighting etc of these files.
2. Change digdag workflow structure to have a well defined task order. E.g.:

tasks:
- name: task1
  sh>: echo first task
- name: task2
  td>: queries/second_task.sql
  tasks: 
    - name: subtask1
      sh>: echo sub
    - name: subtask2
      sh>: echo tasks

Some benefits of this approach are:

• We can keep YAML syntax, semantics and .yml suffix.
• digdag workflows can be safely and easily represented as commonly available map/hash/dictionary datastructures in java/ruby/python/json/etc.

[danielnorberg]

Resolving this per https://github.com/treasure-data/digdag/issues/70 i.e. solution alternative 1.