redesign variable interface

[bcoca]

Proposal: redesign vars

Author: Brian Coca <@bcoca> IRC: bcoca

Date: 2018/31/31

• Status: New
• Proposal type: core design
• Targeted Release: future release
• Estimated time to implement: weeks

Motivation

Simplify variables for both users and developers

Problems

• Currently 'variable precedence' is very confusing and error prone.
• We need more flexibility and predictable interactions
• Unclear scopes makes it hard to know if a variable is available or not

Solution proposal

Redesign the variable system

• Scopes
  • Global: currently ‘stats’ and extra vars, but add something more general (via vars plugins?)
  • Play: (playbook=>play=>block/role=>task subscopes): mainly ‘vars:’, directly accesible
  • Host: mainly from inventory and vars plugins Subsections:
    • module facts
    • local: module facts labeled from ‘local’
    • registered vars: the results from a task
  • Defaults? Special scope definable at any of others but lowest priority
  • Builtins, these are things ansible sets itself, aka 'magic vars',
  • config: accessible as it is now via config lookup, deprecate populating 'ansible_user' and similar as those are currently confusing and unreliable.

Accessible naked and/or via specific dicts:

• Access the limited objects Global[varname], Play[varname], Host[varname]), Defaults[varname], Ansible[varname]`
• Using naked 'varname' should give you the one with most precedence.

• Or short name versions? G.varname, P.varname, H.varname/H.facts.varname D.varname, A.varname

  • Precedence is simple:

    • Buiiltin > Host > Play > Global > Defaults, and in each, internally based on ‘last defined’ (keep extra vars as 'side override channel'?)
    • vars: sub-scope is to ‘current object and contained objects’ (current already mostly works this way)

  • Make role vars private by default (both import_role/include role have the switch, but diff defaults):

    • Add facility for roles to to have exlusively private vars ( /private or /local)?
    • Add role namespace? can be a switch, rolename[varname]/Play['Roles'][name][varname]

  • Remove set_fact in favor of: - set: varname= value= conflict(resolution)=overwrite|merge|aggregate type=str|int|etc mode=lazy|static scope is always 'host'

  • Add ability to create 'static/resolved/non lazy' vars vars: varname: !static '{{ template to be evaluated once vars section is read, cannot contain host scope vars}}'

  • Deprecate/remove vars_files and include_vars, in favor of lookups, which cannot create top level variables, but always need to be assigned to one (acts as namespace)

  • Deprecate/remove vars_prompt and ‘pause prompt hack’.  prompt action/lookup?

  Testing (optional)

  much simpler than current as precedence and scopes are clear and simple.

Documentation (optional)

Also much simpler than current, hopefully the new system is much more intuitive

[barlik]

This is such an amazing proposal.

Variable scoping is IMHO the weakest point of Ansible. I hate the fact that I can't set a local variable inside a role (E.g. using register on a task or using set_fact module) and I agree that variable precedence rules are error prone and confusing for everybody.

Make role vars private by default: Add facility for roles to ‘import vars to global or play or 1st container’?

For this, I would prefer having an option to return a variable from a role:

For example:

- include_role:
    name: myrole
  vars:
    myrole_variable1: var1
  register: myrole_out

- set: varname=play_var value=myrole_out.private_var1  

[alikins]

https://github.com/ansible/ansible/pull/28477 is a pr that implemented tracking of where variables "came from" and displaying that info at higher verbosity levels for troubleshooting. It's not an implementation of the proposal here, but it was rooted in the frustration that the variable scoping and precedence can cause.

The two main aspects of it were:

1. Specifying a _'scopename' when using combinevars() to update variable manager. Additional info about the scope to provide extra context via the 'scopeinfo'. For ex, for a variable that comes from defaults/main.yml, the _'scopename' is 'play_rolesdefaults', and the 'scopeinfo' is the dict {'role_name': 'myrole', 'role_path': '/etc/ansible/roles/myrole'}

2. A data structure to keep track of where the variables come from and the scope_name and scope_info associated with them. The tracking data knows all the scopes that provided a value for a variable and what the the final result was. For an implementation more tightly integrated into variable_manager, this could be done in variable_manager itself.

That branch is intended as a way to debug and troubleshoot the existing very complicated variable precedence rules. A simpler and more straightforward variable precedence scheme would be better in general, but it could still be useful to support introspection of the origins of the final set of variables.

[eugene-bright]

Relates to ansible/ansible#6189.