search

clemlabprojects / ambari

Fork of Apache Ambari maintained by Clemlab Company
https://www.clemlab.com
Apache License 2.0
39 stars 15 forks source link

Wrong Hadoop Conf Dir after Upgrade #71

Open Arni82 opened 3 months ago

Arni82 commented 3 months ago

I have installed a cluster with version 1.2.2.0-46 and updated it to 1.2.2.0-105. The packages are installed correctly, but the cluster can no longer be started after the update because the wrong hadoop conf dir is being used:

Using hadoop conf dir: /usr/odp/1.2.2.0-46/hadoop/conf

How can I change the directory?

`2024-03-12 15:46:32,657 - Stack Feature Version Info: Cluster Stack=1.2, Command Stack=None, Command Version=1.2.2.0-105 -> 1.2.2.0-105 2024-03-12 15:46:32,659 - Using hadoop conf dir: /usr/odp/1.2.2.0-46/hadoop/conf 2024-03-12 15:46:32,660 - Skipping param: datanode_max_locked_memory, due to Configuration parameter 'dfs.datanode.max.locked.memory' was not found in configurations dictionary! 2024-03-12 15:46:32,660 - Skipping param: dfs_ha_namenode_ids, due to Configuration parameter 'dfs.ha.namenodes' was not found in configurations dictionary! 2024-03-12 15:46:32,660 - Skipping param: falcon_user, due to Configuration parameter 'falcon-env' was not found in configurations dictionary! 2024-03-12 15:46:32,660 - Skipping param: gmetad_user, due to Configuration parameter 'ganglia-env' was not found in configurations dictionary! 2024-03-12 15:46:32,660 - Skipping param: gmond_user, due to Configuration parameter 'ganglia-env' was not found in configurations dictionary! 2024-03-12 15:46:32,660 - Skipping param: ranger_group, due to Configuration parameter 'ranger-env' was not found in configurations dictionary! 2024-03-12 15:46:32,660 - Skipping param: ranger_user, due to Configuration parameter 'ranger-env' was not found in configurations dictionary! 2024-03-12 15:46:32,660 - Skipping param: repo_info, due to Configuration parameter 'repoInfo' was not found in configurations dictionary! 2024-03-12 15:46:32,660 - Skipping param: zeppelin_group, due to Configuration parameter 'zeppelin-env' was not found in configurations dictionary! 2024-03-12 15:46:32,660 - Skipping param: zeppelin_user, due to Configuration parameter 'zeppelin-env' was not found in configurations dictionary! 2024-03-12 15:46:32,660 - Group['hdfs'] {} 2024-03-12 15:46:32,660 - Group['hadoop'] {} 2024-03-12 15:46:32,661 - Group['users'] {} 2024-03-12 15:46:32,661 - User['hive'] {'uid': None, 'gid': 'hadoop', 'groups': ['hadoop'], 'fetch_nonlocal_groups': True} 2024-03-12 15:46:32,662 - User['yarn-ats'] {'uid': None, 'gid': 'hadoop', 'groups': ['hadoop'], 'fetch_nonlocal_groups': True} 2024-03-12 15:46:32,662 - User['infra-solr'] {'uid': None, 'gid': 'hadoop', 'groups': ['hadoop'], 'fetch_nonlocal_groups': True} 2024-03-12 15:46:32,663 - User['zookeeper'] {'uid': None, 'gid': 'hadoop', 'groups': ['hadoop'], 'fetch_nonlocal_groups': True} 2024-03-12 15:46:32,663 - User['ams'] {'uid': None, 'gid': 'hadoop', 'groups': ['hadoop'], 'fetch_nonlocal_groups': True} 2024-03-12 15:46:32,664 - User['oozie'] {'uid': None, 'gid': 'hadoop', 'groups': ['hadoop', 'users'], 'fetch_nonlocal_groups': True} 2024-03-12 15:46:32,664 - User['tez'] {'uid': None, 'gid': 'hadoop', 'groups': ['hadoop', 'users'], 'fetch_nonlocal_groups': True} 2024-03-12 15:46:32,665 - User['ambari-qa'] {'uid': None, 'gid': 'hadoop', 'groups': ['hadoop', 'users'], 'fetch_nonlocal_groups': True} 2024-03-12 15:46:32,665 - User['kafka'] {'uid': None, 'gid': 'hadoop', 'groups': ['hadoop'], 'fetch_nonlocal_groups': True} 2024-03-12 15:46:32,666 - User['hdfs'] {'uid': None, 'gid': 'hadoop', 'groups': ['hdfs', 'hadoop'], 'fetch_nonlocal_groups': True} 2024-03-12 15:46:32,666 - User['yarn'] {'uid': None, 'gid': 'hadoop', 'groups': ['hadoop'], 'fetch_nonlocal_groups': True} 2024-03-12 15:46:32,667 - User['mapred'] {'uid': None, 'gid': 'hadoop', 'groups': ['hadoop'], 'fetch_nonlocal_groups': True} 2024-03-12 15:46:32,667 - User['hbase'] {'uid': None, 'gid': 'hadoop', 'groups': ['hadoop'], 'fetch_nonlocal_groups': True} 2024-03-12 15:46:32,668 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0o555} 2024-03-12 15:46:32,668 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2024-03-12 15:46:32,675 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if 2024-03-12 15:46:32,675 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'mode': 0o775, 'create_parents': True, 'cd_access': 'a'} 2024-03-12 15:46:32,676 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0o555} 2024-03-12 15:46:32,676 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0o555} 2024-03-12 15:46:32,676 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {} 2024-03-12 15:46:32,683 - call returned (0, '1016') 2024-03-12 15:46:32,684 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1016'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'} 2024-03-12 15:46:32,689 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1016'] due to not_if 2024-03-12 15:46:32,689 - Group['hdfs'] {} 2024-03-12 15:46:32,690 - User['hdfs'] {'groups': ['hdfs', 'hadoop', 'hdfs'], 'fetch_nonlocal_groups': True} 2024-03-12 15:46:32,691 - FS Type: HDFS 2024-03-12 15:46:32,691 - Directory['/etc/hadoop'] {'mode': 0o755} 2024-03-12 15:46:32,698 - File['/usr/odp/1.2.2.0-46/hadoop/conf/hadoop-env.sh'] {'owner': 'hdfs', 'group': 'hadoop', 'content': InlineTemplate(...)} 2024-03-12 15:46:32,698 - Writing File['/usr/odp/1.2.2.0-46/hadoop/conf/hadoop-env.sh'] because contents don't match 2024-03-12 15:46:32,702 - Changing owner for /usr/odp/1.2.2.0-46/hadoop/conf/hadoop-env.sh from 0 to hdfs 2024-03-12 15:46:32,702 - Changing group for /usr/odp/1.2.2.0-46/hadoop/conf/hadoop-env.sh from 0 to hadoop 2024-03-12 15:46:32,702 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0o1777} 2024-03-12 15:46:32,726 - Skipping param: gmetad_user, due to Configuration parameter 'ganglia-env' was not found in configurations dictionary! 2024-03-12 15:46:32,726 - Skipping param: gmond_user, due to Configuration parameter 'ganglia-env' was not found in configurations dictionary! 2024-03-12 15:46:32,727 - Skipping param: repo_info, due to Configuration parameter 'repoInfo' was not found in configurations dictionary! 2024-03-12 15:46:32,727 - Repository['ODP-1.2-repo-1'] {'action': ['prepare'], 'base_url': 'https://clemlabs.s3.eu-west-3.amazonaws.com/ubuntu22/odp-release/1.2.2.0-105/ODP', 'mirror_list': None, 'repo_file_name': 'ambari-odp-1', 'repo_template': '{{package_type}} {{base_url}} {{components}}', 'components': ['ODP', 'main']} 2024-03-12 15:46:32,729 - Repository['ODP-UTILS-1.2.2.0-repo-1'] {'action': ['prepare'], 'base_url': 'https://clemlabs.s3.eu-west-3.amazonaws.com/ubuntu22/odp-utils/1.2.2.0/ODP-UTILS', 'mirror_list': None, 'repo_file_name': 'ambari-odp-1', 'repo_template': '{{package_type}} {{base_url}} {{components}}', 'components': ['ODP-UTILS', 'main']} 2024-03-12 15:46:32,730 - Repository[None] {'action': ['create']} 2024-03-12 15:46:32,730 - File['/tmp/tmpl6dt0me8'] {'content': b'deb https://clemlabs.s3.eu-west-3.amazonaws.com/ubuntu22/odp-release/1.2.2.0-105/ODP ODP main\ndeb https://clemlabs.s3.eu-west-3.amazonaws.com/ubuntu22/odp-utils/1.2.2.0/ODP-UTILS ODP-UTILS main'} 2024-03-12 15:46:32,730 - Writing File['/tmp/tmpl6dt0me8'] because contents don't match 2024-03-12 15:46:32,734 - File['/tmp/tmpxg5ewqla'] {'content': StaticFile('/etc/apt/sources.list.d/ambari-odp-1.list')} 2024-03-12 15:46:32,735 - Writing File['/tmp/tmpxg5ewqla'] because contents don't match 2024-03-12 15:46:32,739 - Package['unzip'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2024-03-12 15:46:32,754 - Skipping installation of existing package unzip 2024-03-12 15:46:32,754 - Package['curl'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2024-03-12 15:46:32,765 - Skipping installation of existing package curl 2024-03-12 15:46:32,767 - The repository with version 1.2.2.0-46 for this command has been marked as resolved. It will be used to report the version of the component which was installed 2024-03-12 15:46:32,839 - Reporting component version failed Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 365, in execute self.save_component_version_to_structured_out(self.command_name) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 222, in save_component_version_to_structured_out stack_select_package_name = stack_select.get_package_name() File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py", line 117, in get_package_name package = get_packages(PACKAGE_SCOPE_STACK_SELECT, service_name, component_name) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py", line 231, in get_packages supported_packages = get_supported_packages() File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py", line 155, in get_supported_packages raise Fail("Unable to query for supported packages using {0}".format(stack_selector_path)) resource_management.core.exceptions.Fail: Unable to query for supported packages using /usr/bin/odp-select 2024-03-12 15:46:33,055 - Using hadoop conf dir: /usr/odp/1.2.2.0-46/hadoop/conf 2024-03-12 15:46:33,057 - Stack Feature Version Info: Cluster Stack=1.2, Command Stack=None, Command Version=1.2.2.0-105 -> 1.2.2.0-105 2024-03-12 15:46:33,064 - Using hadoop conf dir: /usr/odp/1.2.2.0-46/hadoop/conf 2024-03-12 15:46:33,070 - Skipping param: dfs_ha_namenode_ids, due to Configuration parameter 'dfs.ha.namenodes' was not found in configurations dictionary! 2024-03-12 15:46:33,070 - Skipping param: falcon_user, due to Configuration parameter 'falcon-env' was not found in configurations dictionary! 2024-03-12 15:46:33,070 - Skipping param: ha_zookeeper_quorum, due to Configuration parameter 'ha.zookeeper.quorum' was not found in configurations dictionary! 2024-03-12 15:46:33,070 - Skipping param: hdfs_user_keytab, due to Configuration parameter 'hdfs_user_keytab' was not found in configurations dictionary! 2024-03-12 15:46:33,070 - Skipping param: smokeuser_principal, due to Configuration parameter 'smokeuser_principal_name' was not found in configurations dictionary! 2024-03-12 15:46:33,070 - Skipping param: webhcat_user, due to Configuration parameter 'webhcat_user' was not found in configurations dictionary! 2024-03-12 15:46:33,071 - Package['hadoop-1-2-2-0-46-client'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2024-03-12 15:46:33,086 - Installing package hadoop-1-2-2-0-46-client ('/usr/bin/apt-get -o Dpkg::Options::=--force-confdef --allow-unauthenticated --assume-yes install hadoop-1-2-2-0-46-client') 2024-03-12 15:47:05,538 - The repository with version 1.2.2.0-46 for this command has been marked as resolved. It will be used to report the version of the component which was installed 2024-03-12 15:47:05,611 - Reporting component version failed Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 350, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/ODP/1.0/services/HDFS/package/scripts/hdfs_client.py", line 37, in install self.install_packages(env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 849, in install_packages Package(name, File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 164, in init self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 163, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 127, in run_action provider_action() File "/usr/lib/ambari-agent/lib/resource_management/core/providers/packaging.py", line 30, in action_install self._pkg_manager.install_package(package_name, self.__create_context()) File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/apt_manager.py", line 35, in wrapper return function_to_decorate(self, name, *args[2:], **kwargs) File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/apt_manager.py", line 282, in install_package shell.repository_manager_executor(cmd, self.properties, context, env=self.properties.install_cmd_env) File "/usr/lib/ambari-agent/lib/ambari_commons/shell.py", line 756, in repository_manager_executor raise RuntimeError(message) RuntimeError: Failed to execute command '/usr/bin/apt-get -o Dpkg::Options::=--force-confdef --allow-unauthenticated --assume-yes install hadoop-1-2-2-0-46-client', exited with code '100', message: 'E: Für Paket »hadoop-1-2-2-0-46-client« existiert kein Installationskandidat. '

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 365, in execute self.save_component_version_to_structured_out(self.command_name) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 222, in save_component_version_to_structured_out stack_select_package_name = stack_select.get_package_name() File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py", line 117, in get_package_name package = get_packages(PACKAGE_SCOPE_STACK_SELECT, service_name, component_name) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py", line 231, in get_packages supported_packages = get_supported_packages() File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py", line 155, in get_supported_packages raise Fail("Unable to query for supported packages using {0}".format(stack_selector_path)) resource_management.core.exceptions.Fail: Unable to query for supported packages using /usr/bin/odp-select

Command failed after 1 tries`

lucasbak commented 3 months ago

Hi @Arni82

How to did you upgrade the cluster ? through Ambari or just through package manager ?

Arni82 commented 3 months ago

through Ambari. When I installed the cluster, the -46 version was suggested to me by Ambari. However, the -105 version was already available at https://www.opensourcedataplatform.com/docs/1.2.2.0/getting-ready/source-repository/odp-stack-repository/, so I created a new version and uploaded the VDF file for the 105 version. The -46 version was never installed.

lucasbak commented 3 months ago

@Arni82 I don't understand as you wrote I have installed a cluster with version 1.2.2.0-46 and updated it to 1.2.2.0-105

Best regards

Arni82 commented 3 months ago

Hi @lucasbak

I think I may have found the problem. After updating ambari to 2.7.9.0.0-78, ambari-server only starts with python3:

ambari-server start

Using python /usr/bin/python3

ambari-python-wrap /usr/bin/odp-select versions

File "/usr/bin/odp-select", line 372 os.mkdir(current, 0755) ^ SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers

odp-select versions

1.2.2.0-105 1.2.2.0-134

Is there Python3 support for Ambari 2.7.9.0.0-78?

Thank you

lucasbak commented 3 months ago

Hi @Arni82

There is support indeed. You may have a bad version of odp-select can you execute yum list installed | grep odp ?

Best regards

lucasbak commented 3 months ago

which os do you use ?

Arni82 commented 3 months ago

ubuntu 22.04

apt list --installed | grep odp

odp-select-1-2-2-0-105/now 0.0.3.1.2.1.0-105 all [Installiert,lokal] odp-select/opensource.clemlab.com,now 0.0.3.1.2.2.0-134 all [installiert]

lucasbak commented 3 months ago

@Arni82

It should not use python3 but python2 However with have ODP 1.2.2.0 python3 repositories

deb https://clemlabs.s3.eu-west-3.amazonaws.com/ubuntu22-python3/odp-release/1.2.2.0-134/ODP ODP main
deb https://clemlabs.s3.eu-west-3.amazonaws.com/ubuntu22-python3/odp-utils/1.2.2.0/ODP-UTILS ODP-UTILS main