always failing to apply blueprint

[huanganqing]

Sorry i am just curious what is the correct result of apply_blueprint.sh ? I have retried the install from scratch (snapshot before apply and retry) for about 20 times with different combination but never get success msg from it. And what's minimal services combination to successful applying blueprint ? I have tried remove solr, solr+hive or solr+hst+hive+spark but still can't pass the apply phase. Need help or a bit tips here. Appreciate any help.

[alexandruanghel]

Hi, will need to give more information, like your group_vars/all file and the errors you encounter. A minimal service is this: https://github.com/hortonworks/ansible-hortonworks/blob/master/playbooks/roles/ambari-blueprint/templates/blueprint_hdfs_only.j2#L27

Although the default blueprint config gives a small cluster with a reasonable amount of services: https://github.com/hortonworks/ansible-hortonworks/blob/master/playbooks/group_vars/all#L161

Also, make sure your Jinja2 version is up to date (pip install ansible Jinja2 --upgrade) if you have the same issue as https://github.com/hortonworks/ansible-hortonworks/issues/29 https://github.com/hortonworks/ansible-hortonworks/issues/38 and https://github.com/hortonworks/ansible-hortonworks/issues/43.

[huanganqing]

Thanks. The minimal set if HDFS works, but fail to add service like SPARK(Hive, Tez, YARN) from panel after minimal HDFS installation. It would be a lot help if CI/CI is applied for every change or release.

[alexandruanghel]

Hi, not sure I understand, the default blueprint has all those basic services, spark, hive, tez, yarn and it works fine: https://github.com/hortonworks/ansible-hortonworks/blob/master/playbooks/group_vars/all#L161

And all the blueprint from the examples work fine and have been tested.

So please give more information, looks like something is wrong in your cluster...

[huanganqing]

hdp_version: '3.0.1.0' database: 'embedded'

this is the log which fail to install:

2018-10-11 02:43:16,145 - Skipping installation of existing package curl 2018-10-11 02:43:16,146 - Package['hdp-select'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2018-10-11 02:43:16,215 - Skipping installation of existing package hdp-select 2018-10-11 02:43:16,278 - call[('ambari-python-wrap', u'/usr/bin/hdp-select', 'versions')] {} 2018-10-11 02:43:16,307 - call returned (1, 'Traceback (most recent call last):\n File "/usr/bin/hdp-select", line 456, in \n printVersions()\n File "/usr/bin/hdp-select", line 295, in printVersions\n for f in os.listdir(root):\nOSError: [Errno 2] No such file or directory: \'/usr/hdp\'') 2018-10-11 02:43:16,427 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2018-10-11 02:43:16,428 - Stack Feature Version Info: Cluster Stack=3.0, Command Stack=None, Command Version=None -> 3.0 2018-10-11 02:43:16,442 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2018-10-11 02:43:16,451 - Package['hadoop-3-0-1-0-187-client'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2018-10-11 02:43:16,521 - Installing package hadoop-3-0-1-0-187-client ('/usr/bin/apt-get -o Dpkg::Options::=--force-confdef --allow-unauthenticated --assume-yes install hadoop-3-0-1-0-187-client')

and i try to install from console: apt-get -o Dpkg::Options::=--force-confdef --allow-unauthenticated --assume-yes install hadoop-3-0-1-0-187-client, i got:

E: Unable to locate package hadoop-3-0-1-0-187-client

I found the /etc/apt/sources.list.d/ambari.list is absent. After I add the following into it and apt-update. Now I can install manually by apt-get install.

deb http://public-repo-1.hortonworks.com/ambari/ubuntu16/2.x/updates/2.7.0.0 Ambari main deb http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/ubuntu16 HDP-UTILS main deb http://public-repo-1.hortonworks.com/HDP/ubuntu16/3.x/updates/3.0.1.0 HDP main

Why should people need to setup /etc/apt/sources.list.d/ambari.list manually instead of doing it by ansible ?

[alexandruanghel]

Hi, are your two comments related? Because they are confusing. You mentioned you managed to install a cluster, but then received an error about the hadoop-3-0-1-0-187-client package? This package is one of the first ones to be installed even on a minimal cluster so that's strange because it would have been installed when you did the minimal cluster.

About repositories, they are being setup automatically, you don't have to do anything manually. So something might have happened on your environment (I've seen before repo files being removed automatically).

Ansible will setup this file:

cat /etc/apt/sources.list.d/ambari.list 
deb http://public-repo-1.hortonworks.com/ambari/ubuntu16/2.x/updates/2.7.0.0 Ambari main

And Ambari will setup this file:

cat /etc/apt/sources.list.d/ambari-hdp-1.list 
deb http://public-repo-1.hortonworks.com/HDP/ubuntu16/3.x/updates/3.0.1.0 HDP main
deb http://public-repo-1.hortonworks.com/HDP-GPL/ubuntu16/3.x/updates/3.0.1.0 HDP-GPL main
deb http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/ubuntu16 HDP-UTILS main

This works fine on my Ubuntu16 / Ambari 2.7.0 / HDP 3.0.1 test. You should also use Ambari 2.7.1!

So at this moment I'm blaming something in your environment causing your issues or you are not following the process. Please follow correctly, step by step, the installation instructions.

[alexandruanghel]

As I haven't heard any more information on this, I'll close this issue.