search

abajwa-hw / ambari-nifi-service

Demo Ambari service to deploy/manage NiFi on HDP - Deprecated
75 stars 60 forks source link

'ascii' codec can't decode byte 0xe2 in position 191: ordinal not in range(128 #5

Closed oluies closed 8 years ago

oluies commented 8 years ago

Env: Ubuntu 14.04.1 HDP 2.4

I got this error File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 163, in run Logger.info("Skipping failure of %s due to ignore_failures. Failure reason: %s" % (resource, str(ex))) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 191: ordinal not in range(128)

Lang settings 

$ sudo -u nifi sh
sh-4.3$ echo $LANG
en_US.UTF-8
xadmin@c3gw:~$ sudo -u root sh
sh-4.3# echo $LANG
en_US.UTF-8
sh-4.3# python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> s = '(\xef\xbd\xa1\xef\xbd\xa5\xcf\x89\xef\xbd\xa5\xef\xbd\xa1)\xef\xbe\x89'
>>> s1 = s.decode('utf-8')
>>> print s1
(｡･ω･｡)ﾉ

Full logs: 


Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.4/services/NIFI/package/scripts/master.py", line 201, in <module>
    Master().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.4/services/NIFI/package/scripts/master.py", line 109, in install
    self.configure(env, True)
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.4/services/NIFI/package/scripts/master.py", line 145, in configure
    Execute(format("cd {params.conf_dir}; mv flow.xml.gz flow_$(date +%d-%m-%Y).xml.gz ;"), user=params.nifi_user, ignore_failures=True)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in **init**
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 163, in run
    Logger.info("Skipping failure of %s due to ignore_failures. Failure reason: %s" % (resource, str(ex)))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 191: ordinal not in range(128)
stdout:   /var/lib/ambari-agent/data/output-2802.txt

2016-03-21 17:50:08,974 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.4.0.0-169
2016-03-21 17:50:08,974 - Checking if need to create versioned conf dir /etc/hadoop/2.4.0.0-169/0
2016-03-21 17:50:08,975 - call['conf-select create-conf-dir --package hadoop --stack-version 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-03-21 17:50:09,003 - call returned (1, '/etc/hadoop/2.4.0.0-169/0 exist already', '')
2016-03-21 17:50:09,003 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.4.0.0-169 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-03-21 17:50:09,031 - checked_call returned (0, '/usr/hdp/2.4.0.0-169/hadoop/conf -> /etc/hadoop/2.4.0.0-169/0')
2016-03-21 17:50:09,031 - Ensuring that hadoop has the correct symlink structure
2016-03-21 17:50:09,031 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-03-21 17:50:09,033 - Group['spark'] {}
2016-03-21 17:50:09,034 - Group['zeppelin'] {}
2016-03-21 17:50:09,034 - Group['hadoop'] {}
2016-03-21 17:50:09,035 - Group['nifi'] {}
2016-03-21 17:50:09,035 - Group['users'] {}
2016-03-21 17:50:09,035 - Group['knox'] {}
2016-03-21 17:50:09,035 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2016-03-21 17:50:09,036 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2016-03-21 17:50:09,037 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2016-03-21 17:50:09,037 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2016-03-21 17:50:09,038 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2016-03-21 17:50:09,039 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2016-03-21 17:50:09,039 - User['falcon'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2016-03-21 17:50:09,040 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2016-03-21 17:50:09,041 - User['zeppelin'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2016-03-21 17:50:09,041 - User['nifi'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2016-03-21 17:50:09,042 - User['accumulo'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2016-03-21 17:50:09,043 - User['mahout'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2016-03-21 17:50:09,043 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2016-03-21 17:50:09,044 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2016-03-21 17:50:09,044 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2016-03-21 17:50:09,045 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2016-03-21 17:50:09,046 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2016-03-21 17:50:09,046 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2016-03-21 17:50:09,047 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2016-03-21 17:50:09,048 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2016-03-21 17:50:09,048 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2016-03-21 17:50:09,049 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2016-03-21 17:50:09,050 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2016-03-21 17:50:09,050 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-03-21 17:50:09,054 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2016-03-21 17:50:09,059 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2016-03-21 17:50:09,059 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'recursive': True, 'mode': 0775, 'cd_access': 'a'}
2016-03-21 17:50:09,060 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-03-21 17:50:09,061 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2016-03-21 17:50:09,066 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2016-03-21 17:50:09,067 - Group['hdfs'] {}
2016-03-21 17:50:09,067 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': [u'hadoop', u'hdfs']}
2016-03-21 17:50:09,068 - Directory['/etc/hadoop'] {'mode': 0755}
2016-03-21 17:50:09,083 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2016-03-21 17:50:09,083 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0777}
2016-03-21 17:50:09,097 - Repository['HDP-2.4'] {'base_url': 'http://public-repo-1.hortonworks.com/HDP/ubuntu14/2.x/updates/2.4.0.0/', 'action': ['create'], 'components': [u'HDP', 'main'], 'repo_template': '{{package_type}} {{base_url}} {{components}}', 'repo_file_name': 'HDP', 'mirror_list': None}
2016-03-21 17:50:09,106 - File['/tmp/tmpnWfyts'] {'content': 'deb http://public-repo-1.hortonworks.com/HDP/ubuntu14/2.x/updates/2.4.0.0/ HDP main'}
2016-03-21 17:50:09,106 - Writing File['/tmp/tmpnWfyts'] because contents don't match
2016-03-21 17:50:09,107 - File['/tmp/tmpdRtnXR'] {'content': StaticFile('/etc/apt/sources.list.d/HDP.list')}
2016-03-21 17:50:09,107 - Writing File['/tmp/tmpdRtnXR'] because contents don't match
2016-03-21 17:50:09,108 - Repository['HDP-UTILS-1.1.0.20'] {'base_url': 'http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/ubuntu14', 'action': ['create'], 'components': [u'HDP-UTILS', 'main'], 'repo_template': '{{package_type}} {{base_url}} {{components}}', 'repo_file_name': 'HDP-UTILS', 'mirror_list': None}
2016-03-21 17:50:09,110 - File['/tmp/tmpfGdQL8'] {'content': 'deb http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/ubuntu14 HDP-UTILS main'}
2016-03-21 17:50:09,110 - Writing File['/tmp/tmpfGdQL8'] because contents don't match
2016-03-21 17:50:09,110 - File['/tmp/tmpIqqJht'] {'content': StaticFile('/etc/apt/sources.list.d/HDP-UTILS.list')}
2016-03-21 17:50:09,110 - Writing File['/tmp/tmpIqqJht'] because contents don't match
2016-03-21 17:50:09,111 - Package['unzip'] {}
2016-03-21 17:50:09,146 - Skipping installation of existing package unzip
2016-03-21 17:50:09,146 - Package['curl'] {}
2016-03-21 17:50:09,187 - Skipping installation of existing package curl
2016-03-21 17:50:09,187 - Package['hdp-select'] {}
2016-03-21 17:50:09,241 - Skipping installation of existing package hdp-select
2016-03-21 17:50:09,442 - Execute['cp /etc/sudoers /etc/sudoers.bak'] {}
2016-03-21 17:50:09,449 - Execute['echo "nifi    ALL=(ALL)       NOPASSWD: ALL" >> /etc/sudoers'] {}
2016-03-21 17:50:09,452 - Execute['echo Creating /var/log/nifi /var/run/nifi'] {}
2016-03-21 17:50:09,456 - Directory['/var/run/nifi'] {'owner': 'nifi', 'group': 'nifi', 'recursive': True}
2016-03-21 17:50:09,456 - Creating directory Directory['/var/run/nifi'] since it doesn't exist.
2016-03-21 17:50:09,456 - Changing owner for /var/run/nifi from 0 to nifi
2016-03-21 17:50:09,456 - Changing group for /var/run/nifi from 0 to nifi
2016-03-21 17:50:09,457 - Directory['/var/log/nifi'] {'owner': 'nifi', 'group': 'nifi', 'recursive': True}
2016-03-21 17:50:09,457 - Execute['touch /var/log/nifi/nifi-setup.log'] {'user': 'nifi'}
2016-03-21 17:50:09,474 - Execute['rm -rf /opt/nifi-0.5.1.1.1.2.0-32'] {'ignore_failures': True}
2016-03-21 17:50:09,478 - Execute['mkdir -p /opt/nifi-0.5.1.1.1.2.0-32'] {}
2016-03-21 17:50:09,482 - Execute['chown -R nifi:nifi /opt/nifi-0.5.1.1.1.2.0-32'] {}
2016-03-21 17:50:09,486 - Execute['echo Compiling nifi from source'] {}
2016-03-21 17:50:09,490 - Execute['cd /opt; git clone https://git-wip-us.apache.org/repos/asf/nifi.git /opt/nifi-0.5.1.1.1.2.0-32 >> /var/log/nifi/nifi-setup.log'] {}
2016-03-21 17:50:37,284 - Execute['chown -R nifi:nifi /opt/nifi-0.5.1.1.1.2.0-32'] {}
2016-03-21 17:50:37,355 - Execute['cd /opt/nifi-0.5.1.1.1.2.0-32; mvn -T C2.0 clean install -DskipTests >> /var/log/nifi/nifi-setup.log'] {'user': 'nifi'}
2016-03-21 17:58:23,924 - File['/opt/nifi-0.5.1.1.1.2.0-32/nifi-assembly/target/nifi-0.6.0-SNAPSHOT-bin/nifi-0.6.0-SNAPSHOT/conf/nifi.properties'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi'}
2016-03-21 17:58:23,924 - Writing File['/opt/nifi-0.5.1.1.1.2.0-32/nifi-assembly/target/nifi-0.6.0-SNAPSHOT-bin/nifi-0.6.0-SNAPSHOT/conf/nifi.properties'] because contents don't match
2016-03-21 17:58:23,925 - Changing group for /opt/nifi-0.5.1.1.1.2.0-32/nifi-assembly/target/nifi-0.6.0-SNAPSHOT-bin/nifi-0.6.0-SNAPSHOT/conf/nifi.properties from 1003 to nifi
2016-03-21 17:58:23,925 - Execute['echo "First time setup so generating flow.xml.gz" >> /var/log/nifi/nifi-setup.log'] {}
2016-03-21 17:58:23,930 - File['/opt/nifi-0.5.1.1.1.2.0-32/nifi-assembly/target/nifi-0.6.0-SNAPSHOT-bin/nifi-0.6.0-SNAPSHOT/conf/flow.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi'}
2016-03-21 17:58:23,931 - Writing File['/opt/nifi-0.5.1.1.1.2.0-32/nifi-assembly/target/nifi-0.6.0-SNAPSHOT-bin/nifi-0.6.0-SNAPSHOT/conf/flow.xml'] because it doesn't exist
2016-03-21 17:58:23,931 - Changing owner for /opt/nifi-0.5.1.1.1.2.0-32/nifi-assembly/target/nifi-0.6.0-SNAPSHOT-bin/nifi-0.6.0-SNAPSHOT/conf/flow.xml from 0 to nifi
2016-03-21 17:58:23,931 - Changing group for /opt/nifi-0.5.1.1.1.2.0-32/nifi-assembly/target/nifi-0.6.0-SNAPSHOT-bin/nifi-0.6.0-SNAPSHOT/conf/flow.xml from 0 to nifi
2016-03-21 17:58:23,931 - Execute['cd /opt/nifi-0.5.1.1.1.2.0-32/nifi-assembly/target/nifi-0.6.0-SNAPSHOT-bin/nifi-0.6.0-SNAPSHOT/conf; mv flow.xml.gz flow_$(date +%d-%m-%Y).xml.gz ;'] {'ignore_failures': True, 'user': 'nifi'}
oluies commented 8 years ago

Tried upgrading all python stuff/packages to no avail: 

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.4/services/NIFI/package/scripts/master.py", line 201, in <module>
    Master().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.4/services/NIFI/package/scripts/master.py", line 109, in install
    self.configure(env, True)
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.4/services/NIFI/package/scripts/master.py", line 145, in configure
    Execute(format("cd {params.conf_dir}; mv flow.xml.gz flow_$(date +%d-%m-%Y).xml.gz ;"), user=params.nifi_user, ignore_failures=True)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 163, in run
    Logger.info("Skipping failure of %s due to ignore_failures. Failure reason: %s" % (resource, str(ex)))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 191: ordinal not in range(128)
oluies commented 8 years ago

Environment does not contain any strangeness

>>> print b
{'LANG': 'en_US.UTF-8', 'TERM': 'screen', 'AMBARI_CONF_DIR': '/etc/ambari-server/conf:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'PYTHONPATH': '/usr/lib/python2.6/site-packages:/usr/lib/python2.6/site-packages/ambari_commons:/usr/lib/python2.6/site-packages::/usr/lib/python2.6/site-packages', 'OLDPWD': '/', 'SHLVL': '5', 'PWD': '/var/lib/ambari-agent/tmp', 'PYTHON': '/usr/bin/python2', 'PATH': '/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/ambari-agent', 'AMBARI_PASSPHRASE': 'DEV', '_': '/usr/bin/nohup'}
>>> s1 = b.decode('utf-8')
>>> print s1
{'LANG': 'en_US.UTF-8', 'TERM': 'screen', 'AMBARI_CONF_DIR': '/etc/ambari-server/conf:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'PYTHONPATH': '/usr/lib/python2.6/site-packages:/usr/lib/python2.6/site-packages/ambari_commons:/usr/lib/python2.6/site-packages::/usr/lib/python2.6/site-packages', 'OLDPWD': '/', 'SHLVL': '5', 'PWD': '/var/lib/ambari-agent/tmp', 'PYTHON': '/usr/bin/python2', 'PATH': '/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/ambari-agent', 'AMBARI_PASSPHRASE': 'DEV', '_': '/usr/bin/nohup'}
>>>
abajwa-hw commented 8 years ago

On Ubuntu we needed to make below additions to master.py in the Zeppelin service.

encoding=utf8

reload(sys) sys.setdefaultencoding

Might be the same issue here on the Nifi service. Can you try and let me know if that fixes the issue?

See this commit for more details

https://github.com/hortonworks-gallery/ambari-zeppelin-service/commits/f86483318e54c5f4136d939d5c48e9e1341cdf0f/package/scripts/master.py

On Mar 22, 2016, at 7:49 AM, ?rjan Lundberg notifications@github.com<mailto:notifications@github.com> wrote:

Environment does not contain any strangeness

print b {'LANG': 'en_US.UTF-8', 'TERM': 'screen', 'AMBARI_CONFDIR': '/etc/ambari-server/conf:/usr/sbin:/sbin:/usr/lib/ambari-server/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'PYTHONPATH': '/usr/lib/python2.6/site-packages:/usr/lib/python2.6/site-packages/ambaricommons:/usr/lib/python2.6/site-packages::/usr/lib/python2.6/site-packages', 'OLDPWD': '/', 'SHLVL': '5', 'PWD': '/var/lib/ambari-agent/tmp', 'PYTHON': '/usr/bin/python2', 'PATH': '/usr/sbin:/sbin:/usr/lib/ambari-server/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/ambari-agent', 'AMBARIPASSPHRASE': 'DEV', '': '/usr/bin/nohup'} s1 = b.decode('utf-8') print s1 {'LANG': 'en_US.UTF-8', 'TERM': 'screen', 'AMBARI_CONFDIR': '/etc/ambari-server/conf:/usr/sbin:/sbin:/usr/lib/ambari-server/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'PYTHONPATH': '/usr/lib/python2.6/site-packages:/usr/lib/python2.6/site-packages/ambaricommons:/usr/lib/python2.6/site-packages::/usr/lib/python2.6/site-packages', 'OLDPWD': '/', 'SHLVL': '5', 'PWD': '/var/lib/ambari-agent/tmp', 'PYTHON': '/usr/bin/python2', 'PATH': '/usr/sbin:/sbin:/usr/lib/ambari-server/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/ambari-agent', 'AMBARIPASSPHRASE': 'DEV', '': '/usr/bin/nohup'}

You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHubhttps://github.com/abajwa-hw/ambari-nifi-service/issues/5#issuecomment-199849813

oluies commented 8 years ago

Thanks now install and start works