Capability to NOT group by all defined tags

[Sneedes]

Using the oracle.oci.oci inventory plugin. Perhaps it's my usage/implementation, but the default Group by defined tags in the code is undesired.

For example...if I comment out lines 959-971 in oci/plugins/inventory/oci.py, I get my desired functionality. I have OCI compute instances that have many namespace tags that I don't necessarily want to see nor group on within my Ansible inventory.

Is there any way functionality can be added so that my inventory will NOT group on EVERY defined tags? Such as a default/boolean switch to skip this Group by defined tags code block?

            # Group by defined tags
            if hasattr(instance, "defined_tags") and instance.defined_tags:
                for namespace in instance.defined_tags:
                    for key in instance.defined_tags[namespace]:
                        defined_tag_group_name = self.sanitize(
                            namespace
                            + "#"
                            + key
                            + "="
                            + instance.defined_tags[namespace][key]
                        )
                        common_groups.add(defined_tag_group_name)

For example...the default Oracle tags such as CreatedOn have no value for me in an Ansible inventory.

|--@Oracle_Tags#CreatedOn=2021-07-20T23_19_25.865Z:

Additionally, my team has some inventory/processes/workflow items that dislike seeing the "#" character in the group name. Don't know the exact details, but stating the "#" character in the group name is breaking some stuff. Removing the automatic groups of each and every key pair from every tag namespace should remediate this for us.

Please advise what additional information I can provide to assist.

[Sneedes]

all.oci.yml

plugin: oracle.oci.oci

# Oracle Cloud Infrastructure US Government Cloud with FedRAMP Authorization Regions
regions:
  - us-langley-1
  - us-luke-1

# Set the display name to FQDN, based upon short name + domainName tag
hostname_format_preferences:
  - display_name|lower+'.'+defined_tags.Operations.domainName|lower
  - display_name|lower+'.'+'this.host.needs.a.domainName.tag'

# Create additional groups
keyed_groups:
  - key: defined_tags.Operations.applicationName
    prefix: applicationName
    separator: '_'
  - key: defined_tags.Operations.env
    prefix: env
    separator: '_'

compose:
  ansible_host: display_name+'.'+defined_tags.Operations.domainName
  ansible_ssh_host: private_ip
  cloud: '"oci" if id'

# Only include compute instances tagged with osManagedBy for Ansible
include_host_filters:
  - defined_tags.Operations.osManagedBy=='ansible'

[litewhatever]

I am also hitting this issue. Those unnecessary groups are creating distortion.

[sagar2938]

@Sneedes , thanks for all the details. By default, we are creating groups based on the defined tags in inventory plugin. You are also creating your own custom groups using keyed_groups. Your pain point here is that you are seeing a lot more other groups along with the groups which you want. You want a way to disable creation of the default groups so that it generates only the groups which you created ?

Can you provide the error or what kind of issue you are facing with # in the group name? Currently you cannot disable the creation of groups but you can use other options like keyed_groups(which you are already doing)to create your custom group . we will look in to this issue with # in the group name.

[sagar2938]

@Sneedes @litewhatever , I understand your pain point is you are seeing a lot of unnecessary groups. You want a way to disable any of the default groups creation?

[litewhatever]

@Sneedes @litewhatever , I understand your pain point is you are seeing a lot of unnecessary groups. You want a way to disable any of the default groups creation?

Yes.

[Sneedes]

@Sneedes @litewhatever , I understand your pain point is you are seeing a lot of unnecessary groups. You want a way to disable any of the default groups creation?

Yes!

[litewhatever]

@sagar2938 it also seems that these group names are invalid from ansible perspective:

% ansible-playbook -i inventory/poc.oci.yml site.yml -vvvv
ansible-playbook 2.10.12
  config file = /some/path/ansible/ansible.cfg
  configured module search path = ['/home/john/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/john/.pyenv/versions/3.9.4/lib/python3.9/site-packages/ansible
  executable location = /home/john/.pyenv/versions/3.9.4/bin/ansible-playbook
  python version = 3.9.4 (default, Apr  8 2021, 13:23:50) [Clang 12.0.0 (clang-1200.0.32.29)]
Using /some/path/ansible/ansible.cfg as config file
setting up inventory plugins
Loading collection oracle.oci from /some/path/ansible/collections/ansible_collections/oracle/oci
Not replacing invalid character(s) "{'='}" in group name (tag_environment=poc)
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
Not replacing invalid character(s) "{'-', '.', '=', '#'}" in group name (Oracle-Tags#CreatedOn=2021-08-04T06_14_47.900Z)
Not replacing invalid character(s) "{'='}" in group name (tag_role=bastion)
Not replacing invalid character(s) "{'='}" in group name (tag_service=oke)
Not replacing invalid character(s) "{'-', '.', '=', '#'}" in group name (Oracle-Tags#CreatedBy=oracleidentitycloudservice_john_gmail.com)
Not replacing invalid character(s) "{'-'}" in group name (BTew_ME-JEDDAH-1-AD-1)
Not replacing invalid character(s) "{'-'}" in group name (region_me-jeddah-1)
Not replacing invalid character(s) "{'-', '=', '#'}" in group name (Oracle-Tags#CreatedBy=oke)
Not replacing invalid character(s) "{'-', '.', '=', '#'}" in group name (Oracle-Tags#CreatedOn=2021-08-04T06_26_06.841Z)
Not replacing invalid character(s) "{'-', '.', '=', '#'}" in group name (Oracle-Tags#CreatedOn=2021-08-04T06_26_08.254Z)
Not replacing invalid character(s) "{'-', '.', '=', '#'}" in group name (Oracle-Tags#CreatedOn=2021-08-04T06_26_09.517Z)

From curiosity, why do you need them?

[sagar2938]

@litewhatever , thanks for the logs. we will review it internally about using these special characters in the group names

[Sneedes]

bump...any updates on this?

[slmjy]

I have create a MR for this. Plz take a look.

[Sneedes]

Could use a little hand-holding on how to use/test your MR (merge request?)?

If I'm interpreting your code correctly... tags_to_group_by is the new variable I would use. I would populate this list with the complete list of tags I would want to group by? I'd still use this in conjunction with keyed_groups?

[slmjy]

@Sneedes You can use it in conjunction with keyed_groups. tags_to_group_by effectively means - out of all tags found for this instance, group only by these, ignore rest of the tags. If you think another name will be better, or description is not sufficient - please let me know

[sagar2938]

@Sneedes , thanks for the PR. We are actually adding an option default_groups in the next release(2.38.0). Below is the description of this option.

OCI Inventory plugin creates some groups by default based on these properties ["availability_domain", "compartment_name", "region", "freeform_tags", "defined_tags"]. If you don't want OCI inventory plugin to create these default groups, you can use default_groups option to configure which of these default groups should be created. This option takes a list of properties of inventory hosts based on which the groups will be created. The supported properties are

• "availability_domain"
  • "compartment_name"
  • "region"
  • "freeform_tags"
  • "defined_tags" if empty list is passed to this option, none of the default groups are created.

If you want to include or exclude hosts based on the tag, you can use the option groups. groups option takes group name and a jinja expression. all the hosts for which the jinja expression evaluates to truthy value will be included in the group. eg: groups:

: "'' in freeform_tags" : "'' in defined_tags."

[slmjy]

Hi @sagar2938,

It's my PR, so I will reply to your comment. It seems like this default_group option does not support the use-cases in this issue, that I cover in PR:

1. do not create groups for a specific freeform or defined tag
2. create groups for several specified freeform or defined tags and do not create groups for other tags

Feel free to correct me if I'm wrong

[sagar2938]

@slmjy , default_group option disables/enables creation of groups based on freeform/defined tags but it doesnt support for specific tags. these scenarios which you covered in your PR can be achieved through the groups option. you can pass the jinja expressions to the groups option and achieve these scenarios. if you want to create groups based on specific tags, you can use groups option.

eg: groups:

: "'' in freeform_tags" : "'' in defined_tags."

[slmjy]

@sagar2938 Hm.... So I can exclude the tag from inventory using groups option? Let's say - do not create a group for a freeform tag "hostname". Or say - only those 3 tags, no groups for the rest of the tags? I will test it tomorrow in this case, but I had a different impression based on the code...

[sagar2938]

@slmjy , currently in the inventory plugin result, we create groups based on ["availability_domain", "compartment_name", "region", "freeform_tags", "defined_tags"] by default. you will not be able to disable these groups using groups option. You can create your own groups using groups option. you can also check the option keyed_groups as well to create the groups.

If you want to exclude hosts with certain tags, you can check the option exclude_host_filters. this option takes jinja expression and excludes the hosts for which the expression is evaluated to truthy value.

exclude_host_filters:

• "'' in freeform_tags"

[sagar2938]

@slmjy , we have added the option default_groups in the recent release v2.38.0. You can install the latest version using the command ansible-galaxy collection install oracle.oci --force. Please try it out and let us know if it suffices your requirement.

[slmjy]

Hi @sagar2938, Thanks for the new useful feature!

Unfortunately I cannot cover target scenarios of this PR with default_groups, may be I'm missing smth, please advise.

Scenario 1: I don't see a way to not create a group for just one tag. Exclude host filter excludes hosts - that's a totally different thing. Scenario 2: Partial success. I can disable let's say defined_tags creation and create only needed tags with default_groups, but names are not backwards compatible.

# not grouping by any tag by default
default_groups:
  - "region"

# Manually group by a certain defined - not backwards compatible
# old name:    ops#hostname=nspos-fra-zabbix-proxy-1
# new name:    ops_hostname_nspos_fra_zabbix_proxy_1
keyed_groups:
  - key: defined_tags.ops.hostname
    prefix: ops#hostname=

[slmjy]

Overall I think my pull-request is still valid. So I propose I rebase it on your latest version.

[sagar2938]

@slmjy , thanks alot for the details. We are considering extending the option default_groups to configure the include and exclude for each default group. each group can have sub options include, exclude. include and exclude take list of jinja expression as input and these expressions are evaluated on the respective properties of a host. Below is the structure of the default groups. This should give complete control for users on default groups. exclude option will take precedence over include when both options are evaluated to true for a host. Please let me know your thoughts about this.

default_groups:
   region:
     include:
        - <jinja_expression>
     exclude:
        - <jinja_expression>
   availability_domain:
      include:
         - <jinja_expression>
      exclude:
         - <jinja_expression>
   compartment_name:
     include:
        - <jinja_expression>
     exclude:
        - <jinja_expression>
   defined_tags:
      include:
         - <jinja_expression>
         - namespace=="mynamespace" and namespace.key =="mykey"
         - namespace=="mynamespace2" and namespace.key =="mykey2"
      exclude:
        - <jinja_expression>
        - namespace=="mynamespace3" and namespace.key =="mykey3"
    freeform_tags:
      include:
        - <jinja_expression>
        - freeformtag1
      exclude:
        - <jinja_expression>
        - freeform_tag3

[sagar2938]

@slmjy , Please let me know if you have any inputs for the above mentioned design

[nikhiban]

Hi @slmjy , we have added the option "default_groups_preferences" in the recent release v4.9.1 to support suboptions as mentioned above by @sagar2938 . You can install the latest version using the command "ansible-galaxy collection install oracle.oci --force". Please try it out and let us know.

[manojmeda]

@slmjy We are closing this issue since we provided a way to include/exclude default groups. Please feel free to re-open/create a new ticket if you need anything else.