Attribute error during the checkout

[ogajduse]

Reproducer

$ broker --log-level debug checkout --workflow deploy-sat-jenkins --template satellite-6.10 --host_type satellite
Log level changed to [debug]
[D 210726 13:59:25 broker:86] Broker instantiated with kwargs={'workflow': 'deploy-sat-jenkins', 'template': 'satellite-6.10', 'host_type': 'satellite'}
[I 210726 13:59:25 broker:130] Using provider AnsibleTower to checkout
[D 210726 13:59:25 ansible_tower:77] AnsibleTower instantiated with kwargs={'workflow': 'deploy-sat-jenkins', 'template': 'satellite-6.10', 'host_type': 'satellite'}
[I 210726 13:59:26 ansible_tower:102] Using token authentication
[I 210726 13:59:27 ansible_tower:423] No inventory specified, Ansible Tower will use a default.
[D 210726 13:59:27 ansible_tower:424] Launching workflow: https://infra-ansible-tower-01.ourdomain.com/api/v2/workflow_job_templates/56/
    payload={'extra_vars': "{'workflow': 'deploy-sat-jenkins', 'template': 'satellite-6.10', 'host_type': 'satellite'}"}
[I 210726 13:59:28 ansible_tower:433] Waiting for job: 
    API: https://infra-ansible-tower-01.ourdomain.com/api/v2/workflow_jobs/825035/
    UI: https://infra-ansible-tower-01.ourdomain.com/#/workflows/825035
[D 210726 14:21:52 broker:111] {
        "id": 825035,
        "type": "workflow_job",
        "url": "/api/v2/workflow_jobs/825035/",
        "related": {
            "created_by": "/api/v2/users/13/",
            "modified_by": "/api/v2/users/13/",
            "unified_job_template": "/api/v2/workflow_job_templates/56/",
            "workflow_job_template": "/api/v2/workflow_job_templates/56/",
            "notifications": "/api/v2/workflow_jobs/825035/notifications/",
            "workflow_nodes": "/api/v2/workflow_jobs/825035/workflow_nodes/",
            "labels": "/api/v2/workflow_jobs/825035/labels/",
            "activity_stream": "/api/v2/workflow_jobs/825035/activity_stream/",
            "relaunch": "/api/v2/workflow_jobs/825035/relaunch/",
            "cancel": "/api/v2/workflow_jobs/825035/cancel/"
        },
        "summary_fields": {
            "organization": {
                "id": 2,
                "name": "Satellite",
                "description": ""
            },
            "inventory": {
                "id": 50,
                "name": "satlab-rhv-02-inventory",
                "description": "RHV host(s) for QE contributor and CI pipeline use, maintained by SatQE SatLab team. QE has full control of network for PXE and local network creation.",
                "has_active_failures": true,
                "total_hosts": 180,
                "hosts_with_active_failures": 7,
                "total_groups": 10,
                "has_inventory_sources": true,
                "total_inventory_sources": 2,
                "inventory_sources_with_failures": 0,
                "organization_id": 2,
                "kind": ""
            },
            "workflow_job_template": {
                "id": 56,
                "name": "deploy-sat-jenkins",
                "description": "Workflow to deploy sat-jenkins with separated postdeploy"
            },
            "unified_job_template": {
                "id": 56,
                "name": "deploy-sat-jenkins",
                "description": "Workflow to deploy sat-jenkins with separated postdeploy",
                "unified_job_type": "workflow_job"
            },
            "created_by": {
                "id": 13,
                "username": "ogajduse",
                "first_name": "Ondrej",
                "last_name": "Gajdusek"
            },
            "modified_by": {
                "id": 13,
                "username": "ogajduse",
                "first_name": "Ondrej",
                "last_name": "Gajdusek"
            },
            "user_capabilities": {
                "delete": true,
                "start": true
            },
            "labels": {
                "count": 0,
                "results": []
            }
        },
        "created": "2021-07-26T11:59:27.760765Z",
        "modified": "2021-07-26T11:59:28.268990Z",
        "name": "deploy-sat-jenkins",
        "description": "Workflow to deploy sat-jenkins with separated postdeploy",
        "unified_job_template": 56,
        "launch_type": "manual",
        "status": "successful",
        "failed": false,
        "started": "2021-07-26T11:59:28.264982Z",
        "finished": "2021-07-26T12:21:52.082022Z",
        "canceled_on": null,
        "elapsed": 1343.817,
        "job_args": "",
        "job_cwd": "",
        "job_env": {},
        "job_explanation": "",
        "result_traceback": "",
        "workflow_job_template": 56,
        "extra_vars": "{\"host_type\": \"satellite\", \"deploy_scenario\": \"sat-jenkins\", \"deploy_snap_version\": \"\", \"deploy_sat_version\": \"\", \"deploy_template_name\": \"\", \"deploy_rhel_version\": \"7\", \"workflow\": \"deploy-sat-jenkins\", \"template\": \"satellite-6.10\"}",
        "allow_simultaneous": true,
        "job_template": null,
        "is_sliced_job": false,
        "inventory": 50,
        "limit": null,
        "scm_branch": null,
        "webhook_service": "",
        "webhook_credential": null,
        "webhook_guid": ""
    }
[D 210726 14:21:52 ansible_tower:184] Attempting to merge: deploy-sat-jenkins
[D 210726 14:21:53 ansible_tower:206] {
        "id": 184129,
        "type": "workflow_job_node",
        "url": "/api/v2/workflow_job_nodes/184129/",
        "related": {
            "credentials": "/api/v2/workflow_job_nodes/184129/credentials/",
            "success_nodes": "/api/v2/workflow_job_nodes/184129/success_nodes/",
            "failure_nodes": "/api/v2/workflow_job_nodes/184129/failure_nodes/",
            "always_nodes": "/api/v2/workflow_job_nodes/184129/always_nodes/",
            "unified_job_template": "/api/v2/job_templates/21/",
            "job": "/api/v2/jobs/825107/",
            "workflow_job": "/api/v2/workflow_jobs/825035/"
        },
        "summary_fields": {
            "job": {
                "id": 825107,
                "name": "satlab-tower-deploy-set-stats-for-jenkins-automation-rhvm-02-wf",
                "description": "Set Stats output for Jenkins and Automation to use at the end of VM Deployment",
                "status": "successful",
                "failed": false,
                "elapsed": 11.148,
                "type": "job"
            },
            "workflow_job": {
                "id": 825035,
                "name": "deploy-sat-jenkins",
                "description": "Workflow to deploy sat-jenkins with separated postdeploy"
            },
            "unified_job_template": {
                "id": 21,
                "name": "satlab-tower-deploy-set-stats-for-jenkins-automation-rhvm-02-wf",
                "description": "Set Stats output for Jenkins and Automation to use at the end of VM Deployment",
                "unified_job_type": "job"
            }
        },
        "created": "2021-07-26T11:59:27.820927Z",
        "modified": "2021-07-26T12:21:26.065916Z",
        "extra_data": {},
        "inventory": null,
        "scm_branch": null,
        "job_type": null,
        "job_tags": null,
        "skip_tags": null,
        "limit": null,
        "diff_mode": null,
        "verbosity": null,
        "job": 825107,
        "workflow_job": 825035,
        "unified_job_template": 21,
        "success_nodes": [],
        "failure_nodes": [],
        "always_nodes": [],
        "all_parents_must_converge": false,
        "do_not_run": false,
        "identifier": "3e7293dd-ce45-4132-94f7-6cd2c08576d1"
    }
[D 210726 14:21:53 ansible_tower:184] Attempting to merge: satlab-tower-deploy-set-stats-for-jenkins-automation-rhvm-02-wf
[D 210726 14:21:53 ansible_tower:188] Found artifacts: {'deploy_snap_version': '10.0', 'name': 'ogajduse-sat-jenkins-6.10.0-10.0-4d17b8f4', 'host_type': 'satellite', 'os_distribution_version': '7.9', 'fqdn': 'dhcp-3-43.ourdomain.com', 'deploy_rhel_version': '7.9', 'template': 'satellite-6.10', 'tower_inventory': 'satlab-rhv-02-inventory', 'deploy_sat_version': '6.10.0', 'os_distribution': 'RedHat', 'reported_devices': {'nics': ['lo', 'eth0']}}
[D 210726 14:21:53 ansible_tower:371] {'deploy_snap_version': '10.0', 'name': 'ogajduse-sat-jenkins-6.10.0-10.0-4d17b8f4', 'host_type': 'satellite', 'os_distribution_version': '7.9', 'fqdn': 'dhcp-3-43.ourdomain.com', 'deploy_rhel_version': '7.9', 'template': 'satellite-6.10', 'tower_inventory': 'satlab-rhv-02-inventory', 'deploy_sat_version': '6.10.0', 'os_distribution': 'RedHat', 'reported_devices_nics': ['lo', 'eth0']}
[D 210726 14:21:53 ansible_tower:382] hostname: dhcp-3-43.ourdomain.com, name: ogajduse-sat-jenkins-6.10.0-10.0-4d17b8f4, host type: host
[D 210726 14:21:53 broker:135] host=<broker.hosts.Host object at 0x7fa5e8cd46d0>
[I 210726 14:21:53 broker:138] Host: dhcp-3-43.ourdomain.com
[I 210726 14:21:53 broker:130] Using provider AnsibleTower to checkout
[D 210726 14:21:53 ansible_tower:77] AnsibleTower instantiated with kwargs={'workflow': 'deploy-sat-jenkins', 'template': 'satellite-6.10', 'host_type': 'satellite'}
[I 210726 14:21:53 ansible_tower:102] Using token authentication
[E 210726 14:21:54 exceptions:10] getattr(): attribute name must be string
    Traceback (most recent call last):
      File "/home/ogajduse/.virtualenvs/broker/lib/python3.9/site-packages/broker/commands.py", line 18, in __call__
        return self.main(*args, **kwargs)
      File "/home/ogajduse/.virtualenvs/broker/lib/python3.9/site-packages/click/core.py", line 782, in main
        rv = self.invoke(ctx)
      File "/home/ogajduse/.virtualenvs/broker/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/home/ogajduse/.virtualenvs/broker/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/home/ogajduse/.virtualenvs/broker/lib/python3.9/site-packages/click/core.py", line 610, in invoke
        return callback(*args, **kwargs)
      File "/home/ogajduse/.virtualenvs/broker/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func
        return f(get_current_context(), *args, **kwargs)
      File "/home/ogajduse/.virtualenvs/broker/lib/python3.9/site-packages/broker/commands.py", line 168, in checkout
        broker_inst.checkout()
      File "/home/ogajduse/.virtualenvs/broker/lib/python3.9/site-packages/broker/broker.py", line 146, in checkout
        hosts = self._checkout()
      File "/home/ogajduse/.virtualenvs/broker/lib/python3.9/site-packages/broker/broker.py", line 57, in mp_split
        return self.func(instance, *args, **kwargs)
      File "/home/ogajduse/.virtualenvs/broker/lib/python3.9/site-packages/broker/broker.py", line 132, in _checkout
        host = self._act(provider, method, checkout=True)
      File "/home/ogajduse/.virtualenvs/broker/lib/python3.9/site-packages/broker/broker.py", line 110, in _act
        result = getattr(provider_inst, method)(**self._kwargs)
    TypeError: getattr(): attribute name must be string
[E 210726 14:21:54 exceptions:12] BrokerError: getattr(): attribute name must be string

[ogajduse]

My broker_settings.yaml:

    # Broker settings
    debug: True
    inventory_file: "inventory.yaml"
    # Host Settings
    host_username: "root"
    host_password: "redhat"

    # Provider settings
    AnsibleTower:
      instances:
        - rhv:
            base_url: "https://infra-ansible-tower-01.infra.ourdomain.com/"
            # username: "ogajduse"
            # password: ""
            token: mytoken
            default: True
        - osp:
            base_url: "https://infra-ansible-tower-01.infra.ourdomain.com/"
            # username: "ogajduse"
            # password: ""
            token: mytoken
            inventory: "satlab-osp-01-inventory"
        - testing:
            base_url: "https://dhcp-2-80.vms.ourdomain.com/"
            #username: ""
            #password: ""
            inventory: "satlab-rhv-02-inventory"
            username: ogajduse
            password: ""
            #default: True
      release_workflow: "remove-vm"
      extend_workflow: "extend-vm"
      workflow_timeout: 3600
      #results_limit: 50

[ogajduse]

my venv:

awxkit==17.1.0
broker @ file:///home/ogajduse/qa_repos/satelliteqe-github/broker
certifi==2020.12.5
chardet==4.0.0
click==7.1.2
dynaconf==3.1.4
idna==2.10
logzero==1.7.0
PyYAML==5.4.1
requests==2.25.1
ssh2-python==0.26.0
urllib3==1.26.4

[jarovo]

I found out that the _act method gets called first with proivder AnsibleTower and method execute. Then it is called with same provider and method None

(Pdb) ll
121         def _checkout(self):
122             """checkout one or more VMs
123     
124             :return: List of Host objects
125             """
126             hosts = []
127             if not self._provider_actions:
128                 raise self.BrokerError("Could not determine an appropriate provider")
129             for action in self._provider_actions.keys():
130                 provider, method = PROVIDER_ACTIONS[action]
131                 logger.info(f"Using provider {provider.__name__} to checkout")
132                 try:
133  ->                 host = self._act(provider, method, checkout=True)
134                 except exceptions.ProviderError:
135                     host = None
136                 logger.debug(f"host={host}")
137                 if host:
138                     hosts.append(host)
139                     logger.info(f"{host.__class__.__name__}: {host.hostname}")
140             return hosts
(Pdb) PROVIDER_ACTIONS
{'workflow': (<class 'broker.providers.ansible_tower.AnsibleTower'>, 'execute'), 'job_template': (<class 'broker.providers.ansible_tower.AnsibleTower'>, 'execute'), 'template': (<class 'broker.providers.ansible_tower.AnsibleTower'>, None), 'test_action': (<class 'broker.providers.test_provider.TestProvider'>, 'test_action'), 'inventory': (<class 'broker.providers.ansible_tower.AnsibleTower'>, None)}
(Pdb) action
'template'
(Pdb) provider
<class 'broker.providers.ansible_tower.AnsibleTower'>
(Pdb) print(method)
None

[JacobCallahan]

@jaryn what is the value of self._provider_actions in the reproducer?

[jarovo]

A reproducer that doesn't take so long time is broker --log-level debug checkout --template deploy-sat-jenkins

[JacobCallahan]

A reproducer that doesn't take so long time is broker --log-level debug checkout --template deploy-sat-jenkins

templates are not valid checkout actions for AnsibleTower, which is why they aren't listed under the help output for checkout

[ogajduse]

I did not realized that I reported duplicate of the issue that I reported earlier. As I stated in #136, We agreed with @rdrazny that --template is a valid option that can be passed to a workflow. Do we know what is causing broker to fail with such error?