foreman_hostgroup Unable to add activation keys if they don't already exist

[chsnell]

SUMMARY

I'm not sure if this is an issue with the foreman API or with this module. When creating a brand new hostgroup with foreman_hostgroup, or editing an existing one, if the kt_activation_keys parameter does not already exist, you will receive an Internal Server Error. Everything works fine, can even edit the parameter, if it does exist.

ISSUE TYPE

• Bug Report

ANSIBLE VERSION

ansible 2.8.4
  config file = /home/itcxs02/dev/rci-ansible/ansible.cfg
  configured module search path = [u'/home/itcxs02/dev/rci-ansible/library', u'/home/itcxs02/dev/rci-ansible/foreman-ansible-modules/plugins/modules', u'/usr/lib/python2.7/site-packages/ansible/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, May 31 2018, 09:41:32) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

KATELLO/FOREMAN VERSION

tfm-rubygem-katello-3.13.0-1.el7.noarch
foreman-1.23.0-1.el7.noarch

APYPIE VERSION

Version: 0.1.0

STEPS TO REPRODUCE

This is a minimal task entry from a role that should work, provided you have all the variables filled out with valid values.

- name: Manage DEV-CENT7 host group in Foreman
  foreman_hostgroup:
    username: 'admin'
    password: "{{ admin_pw[env] }}"
    server_url: "https://{{ inventory_hostname }}"
    organization: "{{ foreman_org }}"
    locations:
      - 'foobar'
    name: 'DEV-CENT7'
    parent: 'DEV'
    content_view: 'CentOS 7.7 Composite'
    operatingsystem: 'CentOS 7.7'
    media: 'CentOS-7'
    parameters:
      - name: 'kt_activation_keys'
        value: 'centos7_dev'

EXPECTED RESULTS

I expect the kt_activation_keys parameter to be created if it does not exist.

ACTUAL RESULTS

TASK [ri-foreman : Manage DEV-CENT7 host group in Foreman] ******************************************************
task path: roles/ri-foreman/tasks/configure/host_groups.yml:21
<myforeman> ESTABLISH SSH CONNECTION FOR USER: None
<myforeman> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=60 -o ControlPath=/myhome/.ansible/cp/13d71cf2e3 myforeman '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<myforeman> (0, '/myhome\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017\r\ndebug1: Reading configuration data /myhome/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 10681\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<myforeman> ESTABLISH SSH CONNECTION FOR USER: None
<myforeman> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=60 -o ControlPath=/myhome/.ansible/cp/13d71cf2e3 myforeman '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /myhome/.ansible/tmp/ansible-tmp-1572643095.24-150043195022494 `" && echo ansible-tmp-1572643095.24-150043195022494="` echo /myhome/.ansible/tmp/ansible-tmp-1572643095.24-150043195022494 `" ) && sleep 0'"'"''
<myforeman> (0, 'ansible-tmp-1572643095.24-150043195022494=/myhome/.ansible/tmp/ansible-tmp-1572643095.24-150043195022494\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017\r\ndebug1: Reading configuration data /myhome/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 10681\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file foreman-ansible-modules/plugins/modules/foreman_hostgroup.py
<myforeman> PUT /myhome/.ansible/tmp/ansible-local-10645_bZR6g/tmpxJbhd9 TO /myhome/.ansible/tmp/ansible-tmp-1572643095.24-150043195022494/AnsiballZ_foreman_hostgroup.py
<myforeman> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=60 -o ControlPath=/myhome/.ansible/cp/13d71cf2e3 '[myforeman]'
<myforeman> (0, 'sftp> put /myhome/.ansible/tmp/ansible-local-10645_bZR6g/tmpxJbhd9 /myhome/.ansible/tmp/ansible-tmp-1572643095.24-150043195022494/AnsiballZ_foreman_hostgroup.py\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017\r\ndebug1: Reading configuration data /myhome/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 10681\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug3: Sent message fd 6 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /myhome size 0\r\ndebug3: Looking up /myhome/.ansible/tmp/ansible-local-10645_bZR6g/tmpxJbhd9\r\ndebug3: Sent message fd 6 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/myhome/.ansible/tmp/ansible-tmp-1572643095.24-150043195022494/AnsiballZ_foreman_hostgroup.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:15186\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 15186 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<myforeman> ESTABLISH SSH CONNECTION FOR USER: None
<myforeman> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=60 -o ControlPath=/myhome/.ansible/cp/13d71cf2e3 myforeman '/bin/sh -c '"'"'chmod u+x /myhome/.ansible/tmp/ansible-tmp-1572643095.24-150043195022494/ /myhome/.ansible/tmp/ansible-tmp-1572643095.24-150043195022494/AnsiballZ_foreman_hostgroup.py && sleep 0'"'"''
<myforeman> (0, '', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017\r\ndebug1: Reading configuration data /myhome/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 10681\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<myforeman> ESTABLISH SSH CONNECTION FOR USER: None
<myforeman> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=60 -o ControlPath=/myhome/.ansible/cp/13d71cf2e3 -tt myforeman '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=iuwhyjhpuizsqtcghqhnpknvaofcgwkw] password:" -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-iuwhyjhpuizsqtcghqhnpknvaofcgwkw ; /usr/bin/python2.7 /myhome/.ansible/tmp/ansible-tmp-1572643095.24-150043195022494/AnsiballZ_foreman_hostgroup.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<myforeman> (1, '\r\n\r\n{"msg": "Error while performing create on parameters: 500 Server Error: Internal Server Error", "failed": true, "exception": "WARNING: The below traceback may *not* be related to the actual failure.\\n  File \\"/tmp/ansible_foreman_hostgroup_payload_iIhEVT/ansible_foreman_hostgroup_payload.zip/ansible/module_utils/foreman_helper.py\\", line 513, in resource_action\\n    result = self.foremanapi.resource(resource).call(action, resource_payload, options=options, data=data, files=files)\\n  File \\"/usr/lib/python2.7/site-packages/apypie/resource.py\\", line 30, in call\\n    return self.api.call(self.name, action, params, headers, options, data, files)\\n  File \\"/usr/lib/python2.7/site-packages/apypie/api.py\\", line 158, in call\\n    return self._call_action(action, params, headers, data, files)\\n  File \\"/usr/lib/python2.7/site-packages/apypie/api.py\\", line 167, in _call_action\\n    headers, data, files)\\n  File \\"/usr/lib/python2.7/site-packages/apypie/api.py\\", line 193, in http_call\\n    request.raise_for_status()\\n  File \\"/usr/lib/python2.7/site-packages/requests/models.py\\", line 834, in raise_for_status\\n    raise HTTPError(http_error_msg, response=self)\\n", "error": {"message": "Internal Server Error: the server was unable to finish the request. This may be caused by unavailability of some required service, incorrect API call or a server-side bug. There may be more information in the server\'s logs."}, "invocation": {"module_args": {"domain": null, "subnet6": null, "locations": ["MyLocation"], "content_source": null, "operatingsystem": "CentOS-7.7 7.7", "subnet": null, "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "realm": null, "parameters": [{"value": "centos7_dev", "name": "kt_activation_keys", "parameter_type": "string"}], "media": "CentOS-7", "environment": null, "state": "present", "puppet_ca_proxy": null, "ptable": "myptable", "username": "admin", "root_pass": null, "medium": "CentOS-7", "config_groups": null, "description": null, "parent": "DEV", "compute_profile": null, "server_url": "https://myforeman", "lifecycle_environment": null, "puppet_proxy": null, "openscap_proxy": null, "organizations": null, "name": "DEV-CENT7", "content_view": "CentOS 7.7 Composite", "compute_resource": null, "updated_name": null, "architecture": null, "organization": "MyOrg", "pxe_loader": null, "validate_certs": true}}}\r\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017\r\ndebug1: Reading configuration data /myhome/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 10681\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\nShared connection to myforeman closed.\r\n')
<myforeman> Failed to connect to the host via ssh: OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /myhome/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 10681
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 1
Shared connection to myforeman closed.
<myforeman> ESTABLISH SSH CONNECTION FOR USER: None
<myforeman> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=60 -o ControlPath=/myhome/.ansible/cp/13d71cf2e3 myforeman '/bin/sh -c '"'"'rm -f -r /myhome/.ansible/tmp/ansible-tmp-1572643095.24-150043195022494/ > /dev/null 2>&1 && sleep 0'"'"''
<myforeman> (0, '', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017\r\ndebug1: Reading configuration data /myhome/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 10681\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
The full traceback is:
WARNING: The below traceback may *not* be related to the actual failure.
  File "/tmp/ansible_foreman_hostgroup_payload_iIhEVT/ansible_foreman_hostgroup_payload.zip/ansible/module_utils/foreman_helper.py", line 513, in resource_action
    result = self.foremanapi.resource(resource).call(action, resource_payload, options=options, data=data, files=files)
  File "/usr/lib/python2.7/site-packages/apypie/resource.py", line 30, in call
    return self.api.call(self.name, action, params, headers, options, data, files)
  File "/usr/lib/python2.7/site-packages/apypie/api.py", line 158, in call
    return self._call_action(action, params, headers, data, files)
  File "/usr/lib/python2.7/site-packages/apypie/api.py", line 167, in _call_action
    headers, data, files)
  File "/usr/lib/python2.7/site-packages/apypie/api.py", line 193, in http_call
    request.raise_for_status()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 834, in raise_for_status
    raise HTTPError(http_error_msg, response=self)

fatal: [myforeman]: FAILED! => {
    "changed": false,
    "error": {
        "message": "Internal Server Error: the server was unable to finish the request. This may be caused by unavailability of some required service, incorrect API call or a server-side bug. There may be more information in the server's logs."
    },
    "invocation": {
        "module_args": {
            "architecture": null,
            "compute_profile": null,
            "compute_resource": null,
            "config_groups": null,
            "content_source": null,
            "content_view": "CentOS 7.7 Composite",
            "description": null,
            "domain": null,
            "environment": null,
            "lifecycle_environment": null,
            "locations": [
                "MyLocation"
            ],
            "media": "CentOS-7",
            "medium": "CentOS-7",
            "name": "DEV-CENT7",
            "openscap_proxy": null,
            "operatingsystem": "CentOS-7.7 7.7",
            "organization": "MyOrg",
            "organizations": null,
            "parameters": [
                {
                    "name": "kt_activation_keys",
                    "parameter_type": "string",
                    "value": "centos7_dev"
                }
            ],
            "parent": "DEV",
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "ptable": "myptable",
            "puppet_ca_proxy": null,
            "puppet_proxy": null,
            "pxe_loader": null,
            "realm": null,
            "root_pass": null,
            "server_url": "https://myforeman",
            "state": "present",
            "subnet": null,
            "subnet6": null,
            "updated_name": null,
            "username": "admin",
            "validate_certs": true
        }
    }
}

MSG:

Error while performing create on parameters: 500 Server Error: Internal Server Error

[evgeni]

This should work. Can you please try with a different parameter name, just in case there is some special handling in Katello and post the error in production.log?

[chsnell]

Confirmed it fails with a generic parameter called 'foo' as well.

I think this is the relevant error section from production.log: 2019-11-04T20:13:50 [I|app|64390e6c] Started POST "/api/hostgroups/2/parameters" for 10.10.15.10 at 2019-11-04 20:13:50 -0500 2019-11-04T20:13:50 [I|app|64390e6c] Processing by Api::V2::ParametersController#create as JSON 2019-11-04T20:13:50 [I|app|64390e6c] Parameters: {"parameter"=>{"parameter_type"=>"string", "name"=>"foo", "value"=>"[FILTERED]"}, "apiv"=>"v2", "hostgroup_id"=>"2"} 2019-11-04T20:13:50 [W|app|64390e6c] Action failed ActiveModel::UnknownAttributeError: unknown attribute 'organization_id' for GroupParameter.

[evgeni]

Ooh, that error is interesting.

@mdellweg did you ever see something like that before?

[evgeni]

Ah! Can you apply the patch from https://github.com/theforeman/foreman/pull/7122 on your foreman and retry?

[evgeni]

It seems to be a bug in 1.23, to be fixed in 1.23.1: https://projects.theforeman.org/issues/27740

[mdellweg]

No, i have never seen this. But i see, that it's a 500. So yes i agree, that it must be a server error.

[chsnell]

Confirmed. That one line addition to app/controllers/api/base_controller.rb allows this to work now. Thanks much.

[evgeni]

Great, closing then as it's not our bug :)