[CI][Dev] Make tasks.yml easier to maintain

[asfimport]

I think dev/tasks/tasks.yml has reached its limits by using jinja2 templated yml.

We should think about a better way to define crossbow jobs while:

• keeping it readable
• in a dialect which is natively supported by editors

• while supporting tasks parametrization

  Just one idea is to use python files containing python objects, e.g.:

  
  Task(
  name="wheel-macos-big-sur-cp38-arm64",
  ci="github",
  template="python-wheels/github.osx.arm64.yml",
  params=dict(
    arch="arm64",
    arrow_simd_level="DEFAULT",
    python_version="3.8",
    macos_deployment_target="11.0"
  ),
  artifacts=[
    "pyarrow-{no_rc_version}-cp38-cp38-macosx_11_0_arm64.whl"
  ]
  )

  where Task would be the crossbow task class (which could be refactored to use pydantic or another alternative for less boilerplate). Of course porting to the tasks definitions to plain python could make the situation even worse by accessing too many scripting utilities. We could try a dynamic config language which sits between yaml and python like HCL.

  @kou what syntax would you be comfortable to work with? Do you have any alternatives we could use?

  cc @amol- @raulcd @assignUser

Reporter: Krisztian Szucs / @kszucs

_{Note: This issue was originally created as ARROW-16589. Please see the migration documentation for further details.}

[asfimport]

Kouhei Sutou / @kou: I'm OK with Python. I prefer external DSL such as HCL to internal DSL such as Python because external DSL sometimes lacks dynamic functionalities.

I want to add the following to requirements:

• Easy to write syntax (e.g.: less code, less magic and so on)

[asfimport]

Antoine Pitrou / @pitrou: My experience writing configuration using Python (Buildbot configuration files) is that it encourages contributors to write more and more dynamic functionality that becomes hard to maintain.

[asfimport]

Jacob Wujciak / @assignUser: One aspect I would like to add as a requirement for any alternative is the ability to provide documentation, both in-line and outside of the code e.g. generated doc files or CLI help pages describing e.g. the purpose/use case of each task with examples, possible pre-requisites, caveats etc.

Because at the moment we only have docs describing the overall usage of archery/crossbow but it is not very apparent to contributors which builds they should run locally/trigger on PRs and there is no place this information can be found (docker-compse/tasks.yml only provide clues for the most part).