search

theforeman / foreman_puppet

https://theforeman.org/plugins/foreman_puppet/
GNU General Public License v3.0
7 stars 25 forks source link

Puppet Class Importer fails to update parameters: undefined method `override' for nil:NilClass #388

Open SebbiUltimate opened 8 months ago

SebbiUltimate commented 8 months ago

When I try to import the puppet environments, I get the error "undefined method `override' for nil:NilClass". Beginning of Backtrace:

2024-03-02T00:21:03 [I|app|92f6450f] Backtrace for 'Failed to calculate obsolete and new' error (NoMethodError): undefined method override' for nil:NilClass 92f6450f | /usr/share/gems/gems/foreman_puppet-5.0.0/app/services/foreman_puppet/puppet_class_importer.rb:277:in block in update_parameter' 92f6450f | /usr/share/gems/gems/foreman_puppet-5.0.0/app/services/foreman_puppet/puppet_class_importer.rb:275:in `each' 92f6450f | /usr/share/gems/gems/foreman_puppet-5.0.0/app/services/foreman_puppet/puppet_class_importer.rb:275:in update_parameter' 92f6450f | /usr/share/gems/gems/foreman_puppet-5.0.0/app/services/foreman_puppet/puppet_class_importer.rb:270:in block in update_classes_in_foreman'

It seems that the parameters cannot be found in the lookup_keys table even though they exist.

When I replace the line key = db_class.class_params.find_by key: param_name in puppet_class_importer.rb L275 with key = db_class.class_params.where("key like ?", param_name) it works.

nadjaheitmann commented 8 months ago

Hi @SebbiUltimate, I cannot reproduce it, can you please describe more detailed the configuration of your system and how you tried to import the environment?

SebbiUltimate commented 8 months ago

I think it happens when the default values of puppet class parameters change. So the parameter already exists in the lookup_keys table but has another default value. Then this function update_parameter gets called but doesn't find the record in the table. After changing this line, it works.

stejskalleos commented 8 months ago

@SebbiUltimate Can you please share the exact steps to reproduce with example classes you import? Without it, it's hard to debug the issue.

SebbiUltimate commented 8 months ago

I couldn't reproduce it using a new puppet sample class.

But when I update classes like keystone::ldap, it get the error.

I added more logging:

 def update_parameter(db_class, changed_params)
      changed_params['updated'].each do |param_name, value|
        key = db_class.class_params.find_by key: param_name
        db_class.class_params.map do |dbcp|
                Foreman::Logging.logger('app').info "dbcp.key=#{dbcp.key}!"
        end
        Foreman::Logging.logger('app').info "update_parameter: key not found! class=#{db_class.name}! param_name=#{param_name}!" if key.nil?
        next unless key.override == false
        key.default_value = value
        key.key_type = nil
        key.validator_type = nil
        key.save!(context: :importer)
      end
    end

As you can see in the output, the parameter group_filter ist existing, but find_by cannot find it.

2024-03-09T18:29:06 [I|app|6bc7d633] Processing by ForemanPuppet::PuppetclassesController#obsolete_and_new as HTML 2024-03-09T18:29:06 [I|app|6bc7d633] Parameters: {"authenticity_token"=>"...", "changed"=>{"updated"=>{"production"=>"{\"apache::mod::passenger\":{\"obsolete\":[\"rails_autodetect\",\"rails_auto_detect\",\"rack_auto_detect\"]},\"keystone::ldap\":{\"updated\":{\"group_filter\":\"${::os_service_default}\",\"user_additional_attribute_mapping\":\"${::os_service_default}\",\"user_enabled_emulation_dn\":\"${::os_service_default}\",\"user_enabled_emulation\":\"${::os_service_default}\",\"user_pass_attribute\":\"${::os_service_default}\",\"user_default_project_id_attribute\":\"${::os_service_default}\",\"user_attribute_ignore\":\"${::os_service_default}\",\"user_enabled_invert\":\"${::os_service_default}\",\"user_enabled_default\":\"${::os_service_default}\",\"user_enabled_mask\":\"${::os_service_default}\",\"user_enabled_attribute\":\"${::os_service_default}\",\"user_mail_attribute\":\"${::os_service_default}\",\"user_description_attribute\":\"${::os_service_default}\",\"user_id_attribute\":\"${::os_service_default}\",\"user_filter\":\"${::os_service_default}\",\"page_size\":\"${::os_service_default}\",\"query_scope\":\"${::os_service_default}\"}},\"keystone::policy\":{\"updated\":{\"policy_path\":\"/etc/keystone/policy.yaml\"}},\"keystone::wsgi::apache\":{\"updated\":{\"ssl\":\"undef\"}},\"magnum::clients::glance\":{\"updated\":{\"api_version\":\"${::os_service_default}\"}},\"magnum::clients::heat\":{\"updated\":{\"api_version\":\"${::os_service_default}\"}},\"magnum::clients::nova\":{\"updated\":{\"api_version\":\"${::os_service_default}\"}},\"magnum::policy\":{\"updated\":{\"policy_path\":\"/etc/magnum/policy.yaml\"}},\"murano\":{\"updated\":{\"identity_uri\":\"undef\",\"memcached_servers\":\"undef\",\"project_domain_name\":\"undef\",\"user_domain_name\":\"undef\",\"www_authenticate_uri\":\"undef\",\"admin_tenant_name\":\"undef\",\"admin_user\":\"undef\"}},\"murano::db_cfapi\":{\"updated\":{\"database_idle_timeout\":\"undef\"}},\"murano::policy\":{\"updated\":{\"policy_path\":\"/etc/murano/policy.yaml\"}},\"neutron\":{\"updated\":{\"control_exchange\":\"${::os_service_default}\",\"allow_overlapping_ips\":\"undef\"}},\"neutron::agents::l3\":{\"updated\":{\"gateway_external_network_id\":\"undef\"}},\"neutron::agents::ml2::networking_baremetal\":{\"updated\":{\"max_retries\":\"undef\",\"retry_interval\":\"undef\",\"ironic_url\":\"undef\",\"auth_strategy\":\"undef\"}},\"neutron::agents::ovn_metadata\":{\"updated\":{\"ovn_remote_probe_interval\":\"undef\"}},\"neutron::designate\":{\"updated\":{\"project_id\":\"undef\"}},\"neutron::keystone::auth\":{\"updated\":{\"service_description\":\"OpenStack Networking Service\"}},\"neutron::plugins::ml2::mellanox::mlnx_sdn_assist\":{\"updated\":{\"sdn_password\":\"undef\",\"sdn_username\":\"undef\"}},\"neutron::plugins::ml2::ovn\":{\"updated\":{\"vif_type\":\"undef\",\"ovn_l3_mode\":\"undef\"}},\"neutron::policy\":{\"updated\":{\"policy_path\":\"/etc/neutron/policy.yaml\"}},\"neutron::quota\":{\"updated\":{\"quota_vip\":\"undef\",\"quota_packet_filter\":\"undef\",\"quota_network_gateway\":\"undef\"}},\"neutron::server\":{\"updated\":{\"validate\":\"undef\"}},\"neutron::server::placement\":{\"updated\":{\"username\":\"placement\"}},\"neutron::wsgi::apache\":{\"updated\":{\"ssl\":\"undef\"}},\"nova\":{\"updated\":{\"cross_az_attach\":\"undef\",\"upgrade_level_console\":\"undef\",\"cinder_catalog_info\":\"undef\",\"os_region_name\":\"undef\",\"block_device_allocate_retries_interval\":\"undef\",\"block_device_allocate_retries\":\"undef\"}},\"nova::api\":{\"updated\":{\"install_cinder_client\":\"undef\",\"instance_name_template\":\"${::os_service_default}\",\"validation_options\":\"undef\",\"validate\":\"undef\",\"ratelimits_factory\":\"undef\"}},\"nova::cinder\":{\"updated\":{\"user_domain_name\":\"undef\",\"username\":\"undef\",\"project_domain_name\":\"undef\",\"project_name\":\"undef\",\"region_name\":\"undef\",\"auth_url\":\"undef\",\"auth_type\":\"undef\"}},\"nova::compute\":{\"updated\":{\"keymgr_backend\":\"undef\",\"barbican_api_version\":\"undef\",\"barbican_endpoint\":\"undef\",\"barbican_auth_endpoint\":\"undef\",\"heal_instance_info_cache_interval\":\"${::os_service_default}\",\"reserved_host_memory\":\"${::os_service_default}\",\"force_raw_images\":\"${::os_service_default}\"}},\"nova::compute::libvirt\":{\"updated\":{\"tls_priority\":\"${::os_service_default}\",\"log_filters\":\"${::os_service_default}\",\"log_outputs\":\"${::os_service_default}\"}},\"nova::compute::libvirt::virtlogd\":{\"updated\":{\"max_backups\":\"${::os_service_default}\",\"max_size\":\"${::os_service_default}\",\"admin_max_clients\":\"${::os_service_default}\",\"max_clients\":\"${::os_service_default}\",\"log_outputs\":\"${::os_service_default}\",\"log_filters\":\"${::os_service_default}\",\"log_level\":\"${::os_service_default}\"}},\"nova::compute::vmware\":{\"updated\":{\"use_linked_clone\":\"${::os_service_default}\",\"task_poll_interval\":\"${::os_service_default}\",\"maximum_objects\":\"${::os_service_default}\",\"api_retry_count\":\"${::os_service_default}\"}},\"nova::cron::archive_deleted_rows\":{\"updated\":{\"user\":\"${::nova::params::user}\"}},\"nova::cron::purge_shadow_tables\":{\"updated\":{\"user\":\"${::nova::params::user}\"}},\"nova::keystone::auth\":{\"updated\":{\"service_description\":\"OpenStack Compute Service\"}},\"nova::metadata::novajoin::policy\":{\"updated\":{\"policy_path\":\"/etc/novajoin/policy.yaml\"}},\"nova::migration::libvirt\":{\"updated\":{\"crl_file\":\"${::os_service_default}\",\"listen_address\":\"${::os_service_default}\"}},\"nova::policy\":{\"updated\":{\"policy_path\":\"/etc/nova/policy.yaml\"}},\"nova::quota\":{\"updated\":{\"max_age\":\"undef\",\"until_refresh\":\"undef\",\"reservation_expire\":\"undef\",\"server_group_members\":\"${::os_service_default}\",\"server_groups\":\"${::os_service_default}\",\"key_pairs\":\"${::os_service_default}\",\"security_group_rules\":\"undef\",\"security_groups\":\"undef\",\"injected_file_path_length\":\"${::os_service_default}\",\"injected_file_content_bytes\":\"${::os_service_default}\",\"injected_files\":\"${::os_service_default}\",\"metadata_items\":\"${::os_service_default}\",\"fixed_ips\":\"undef\",\"floating_ips\":\"undef\",\"ram\":\"${::os_service_default}\",\"cores\":\"${::os_service_default}\",\"instances\":\"${::os_service_default}\"}},\"nova::scheduler::filter\":{\"updated\":{\"max_instances_per_host\":\"${::os_service_default}\",\"max_io_ops_per_host\":\"${::os_service_default}\",\"scheduler_host_subset_size\":\"${::os_service_default}\"}},\"nova::wsgi::apache_api\":{\"updated\":{\"ssl\":\"undef\"}},\"nova::wsgi::apache_metadata\":{\"updated\":{\"ssl\":\"undef\"}},\"openstack_extras::repo::redhat::redhat\":{\"updated\":{\"manage_priorities\":\"undef\",\"release\":\"${openstack_extras::repo::redhat::params::release}\"}},\"placement::api\":{\"updated\":{\"port\":\"undef\",\"host\":\"undef\"}},\"placement::keystone::auth\":{\"updated\":{\"internal_url\":\"http://127.0.0.1:8778\",\"admin_url\":\"http://127.0.0.1:8778\",\"public_url\":\"http://127.0.0.1:8778\",\"service_description\":\"OpenStack Placement Service\"}},\"placement::policy\":{\"updated\":{\"policy_path\":\"/etc/placement/policy.yaml\"}},\"placement::wsgi::apache\":{\"updated\":{\"ensure_package\":\"undef\",\"threads\":1,\"workers\":\"${::os_workers}\",\"ssl\":\"undef\",\"path\":\"/\",\"api_port\":8778}},\"sssd\":{\"obsolete\":[\"config_file\"]},\"swift::keystone::auth\":{\"updated\":{\"service_description_s3\":\"OpenStack S3 Service\",\"service_description\":\"OpenStack Object-Store Service\",\"password\":\"undef\"}},\"swift::objectexpirer\":{\"updated\":{\"pipeline\":[\"catch_errors\",\"proxy-logging\",\"cache\",\"proxy-server\"]}},\"swift::proxy\":{\"updated\":{\"pipeline\":[\"catch_errors\",\"gatekeeper\",\"healthcheck\",\"proxy-logging\",\"cache\",\"listing_formats\",\"tempauth\",\"copy\",\"proxy-logging\",\"proxy-server\"]}},\"swift::proxy::authtoken\":{\"updated\":{\"password\":\"undef\",\"auth_plugin\":\"undef\",\"signing_dir\":\"undef\"}},\"swift::proxy::bulk\":{\"updated\":{\"yield_frequency\":\"10\"}},\"swift::proxy::ceilometer\":{\"updated\":{\"password\":\"undef\"}},\"swift::proxy::dlo\":{\"updated\":{\"max_get_time\":\"${::os_service_default}\",\"rate_limit_segments_per_sec\":\"${::os_service_default}\",\"rate_limit_after_segment\":\"${::os_service_default}\"}},\"swift::proxy::s3api\":{\"updated\":{\"max_upload_part_num\":\"${::os_service_default}\",\"ensure\":\"undef\"}},\"swift::proxy::slo\":{\"updated\":{\"max_get_time\":\"${::os_service_default}\",\"rate_limit_segments_per_sec\":\"${::os_service_default}\",\"rate_limit_after_segment\":\"${::os_service_default}\",\"min_segment_size\":\"undef\",\"max_manifest_size\":\"${::os_service_default}\",\"max_manifest_segments\":\"${::os_service_default}\"}},\"swift::proxy::tempurl\":{\"updated\":{\"outgoing_allow_headers\":\"${::os_service_default}\",\"outgoing_remove_headers\":\"${::os_service_default}\",\"incoming_allow_headers\":\"${::os_service_default}\",\"incoming_remove_headers\":\"${::os_service_default}\",\"methods\":\"${::os_service_default}\"}},\"swift::storage::all\":{\"updated\":{\"allow_versions\":\"undef\",\"account_port\":6002,\"container_port\":6001,\"object_port\":6000}},\"vswitch::ovs\":{\"updated\":{\"dkms_ensure\":\"undef\"}}}"}}, "commit"=>"Update"} 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=assignment_driver! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=auth_pool_connection_lifetime! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=auth_pool_size! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=chase_referrals! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=credential_driver! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_additional_attribute_mapping! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_attribute_ignore! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_desc_attribute! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_filter! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_id_attribute! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_member_attribute! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_members_are_ids! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_name_attribute! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_objectclass! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_tree_dn! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=identity_driver! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=manage_packages! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=package_ensure! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=page_size! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=password! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=pool_connection_lifetime! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=pool_connection_timeout! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=pool_retry_delay! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=pool_retry_max! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=pool_size! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=query_scope! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=suffix! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=tls_cacertdir! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=tls_cacertfile! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=tls_req_cert! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=url! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=use_auth_pool! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=use_pool! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_additional_attribute_mapping! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_attribute_ignore! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_default_project_id_attribute! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_description_attribute! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_enabled_attribute! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_enabled_default! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_enabled_emulation! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_enabled_emulation_dn! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_enabled_invert! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_enabled_mask! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_filter! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_id_attribute! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_mail_attribute! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_name_attribute! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_objectclass! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_pass_attribute! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_tree_dn! 2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=use_tls! 2024-03-09T18:29:06 [I|app|6bc7d633] update_parameter: key not found! class=keystone::ldap! param_name=group_filter! 2024-03-09T18:29:06 [W|app|6bc7d633] Failed to calculate obsolete and new 2024-03-09T18:29:06 [I|app|6bc7d633] Backtrace for 'Failed to calculate obsolete and new' error (NoMethodError): undefined method override' for nil:NilClass 6bc7d633 | /usr/share/gems/gems/foreman_puppet-6.2.0/app/services/foreman_puppet/puppet_class_importer.rb:280:inblock in update_parameter' 6bc7d633 | /usr/share/gems/gems/foreman_puppet-6.2.0/app/services/foreman_puppet/puppet_class_importer.rb:274:in each' 6bc7d633 | /usr/share/gems/gems/foreman_puppet-6.2.0/app/services/foreman_puppet/puppet_class_importer.rb:274:inupdate_parameter' .....

It is also in the database:

foreman=> SELECT p.name,l.key,e.id,p.id,l.id FROM environment_classes AS e JOIN puppetclasses AS p ON e.puppetclass_id=p.id JOIN lookup_keys AS l ON e.puppetclass_lookup_key_id=l.id WHERE p.name LIKE 'keystone::ldap' AND l.key LIKE 'group_filter'; name | key | id | id | id ----------------+--------------+------+-----+------ keystone::ldap | group_filter | 4714 | 539 | 4123 (1 row)