After Ambari turns on kerberos authentication, yarn/hdfs web ui access error 401

[SGITLOGIN]

@lucasbak Hello, Under the premise that kerberos authentication must be enabled, is there any other way to prevent the web ui from being accessed through kerberos authentication？

[lucasbak]

Hi @SGITLOGIN,

If you want to protect your webui you can enable Kerebros on the webui or proxify your webui using Apache Knox

[SGITLOGIN]

@lucasbak My purpose is not to enable security authentication for the web UI and allow everyone to directly access it, so I want to ask if it can be achieved?

[SGITLOGIN]

@lucasbak Using anonymous access to the web ui as described in the following document will fail when starting HDFS and YARN.

https://blog.csdn.net/qq_33684569/article/details/116919941?utm_medium=distribute.pc_relevant.none-task-blog-2~default~BlogCommendFromBaidu~default-6.control&depth_1-utm_source=distribute.pc_relevant.none-task-blog-2~default~BlogCommendFromBaidu~default-6.control

[SGITLOGIN]

@lucasbak Hello, I installed Knox here, but the page can only access https://ali-odp-test-01.huan.tv:8443/gateway/default/yarn/cluster. Clicking Applications cannot access https://ali-odp-test-01.huan.tv:8443/gateway/default/yarn/cluster/apps page.

[lucasbak]

@SGITLOGIN ,

its because of your acl in yarn-site put *.

[lucasbak]

@lucasbak Using anonymous access to the web ui as described in the following document will fail when starting HDFS and YARN.

https://blog.csdn.net/qq_33684569/article/details/116919941?utm_medium=distribute.pc_relevant.none-task-blog-2~default~BlogCommendFromBaidu~default-6.control&depth_1-utm_source=distribute.pc_relevant.none-task-blog-2~default~BlogCommendFromBaidu~default-6.control

Indeed, it's because Kerberos is stil enable on the cluster and as a consequence, it trie to communicate with NN using kerberos. It may be an improvement to do on Ambari side.

[SGITLOGIN]

@lucasbak Which configuration item of my Yarn is configured incorrectly? Please correct it

@SGITLOGIN ,

its because of your acl in yarn-site put *.

[SGITLOGIN]

@lucasbak Hello，At present, we are preparing to launch ODP in the prod environment. other problems have been solved. Now there is only the problem of failed access to the web ui page. Can you provide solutions and steps as soon as possible? Thank you

[lucasbak]

@SGITLOGIN

You can set:

• yarn.admin.acl: yarn,knox or yarn.admin.acl: *

You can also set yarn.acl.enable to false if you want free access to all yarn webuis

[SGITLOGIN]

@lucasbak I have modified the configuration and restarted the cluster, but the same problem is still reported when accessing the webui page.

yarn.admin.acl=* yarn.acl.enable=false

[SGITLOGIN]

@lucasbak Is there any other way?

[fb-clemlabb]

Dear SGITLOGIN,

First and foremost, I would like to thank you for your interest in [Your Company Name] and our Open Source Data Platform (ODP) solution. We are thrilled to see the enthusiasm for our product and are convinced of the significant value it can add to your business.

We understand your need for assistance in setting up the ODP cluster and are fully committed to supporting our clients in their success. Our ODP support offering is specifically designed to meet these needs, providing personalized guidance and expert advice to ensure optimal implementation of our solution.

Subscribing to our ODP support offer will give you access to our dedicated team of experts who can assist you with configuration, deployment, and maintenance of your ODP cluster, while offering quick and effective responses to all your technical inquiries. This will not only allow you to maximize the performance of your platform but also ensure the security and sustainability of your data infrastructure.

We are confident that this approach is crucial for leveraging the full potential of our ODP solution and ensuring the success of your project. I would be delighted to discuss your needs in more detail and how our support offering can benefit you. Please do not hesitate to contact me to arrange a meeting or for any further information you may require.

Thank you for your understanding, and I remain at your complete disposal for any questions or clarifications.

Kind regards, Clemlab Team Support odp@clemlab.com

[SGITLOGIN]

@fb-clemlabb OK, we will carefully consider your plan here

[SGITLOGIN]

@lucasbak @fb-clemlabb Hello，Are there any other solutions to the above web ui problem？ Our purpose is to access the web UI without Kerberos authentication. Simple account and password or anonymous login are acceptable.

[SGITLOGIN]

@lucasbak I reinstalled the cluster today (this time I tested it on the premise that the Kerberos task was not enabled in the ODP cluster) and found that before turning on the high availability of HDFS and YARN, I could use knox to access yarn's web ui, but after turning on the high availability of HDFS and YARN When I try to access Yarn's web UI later, I get a 403 No Permission error.

HDFS namenode：ali-odp-test-01，ali-odp-test-02 YARN resourcemanager：ali-odp-test-01，ali-odp-test-02 Knox Gateway：ali-odp-test-01

Login account：admin

I also tried modifying the yarn.admin.acl=* and yarn.acl.enable=false parameter values, but still reported the same error.

[lucasbak]

This error comes from Knox acls try to modify also topology default acls

[lucasbak]

@SGITLOGIN

Can you check the authorization part of the default topology, also check in core-site proxy knox user

[SGITLOGIN]

@lucasbak The problem that yarn webui cannot be accessed has been solved, but there is a strange phenomenon on yarn webui. Spark type tasks are not displayed on yarn webui. Only the running tasks of the TEZ engine are displayed. However, when running Spark tasks, the top of yarn webui Apps Running also prompts that a task is running. Why is this?

[SGITLOGIN]

@lucasbak Hello, the problem I asked above about "the yarn webui page does not display spark type tasks" is a bug in this version of ODP or should this version of hadoop be like this?

[SGITLOGIN]

@lucasbak The problem has been resolved.

@lucasbak Hello, the problem I asked above about "the yarn webui page does not display spark type tasks" is a bug in this version of ODP or should this version of hadoop be like this?

[lucasbak]

@lucasbak The problem that yarn webui cannot be accessed has been solved, but there is a strange phenomenon on yarn webui. Spark type tasks are not displayed on yarn webui. Only the running tasks of the TEZ engine are displayed. However, when running Spark tasks, the top of yarn webui Apps Running also prompts that a task is running. Why is this?

@SGITLOGIN , there is may be an application running that you do not see because of the acls.

[lucasbak]

@lucasbak Hello, the problem I asked above about "the yarn webui page does not display spark type tasks" is a bug in this version of ODP or should this version of hadoop be like this?

It depends of the tasks, if you are running Hive queries though spark, you will have Hive/Tez taskk type

[SGITLOGIN]

@lucasbak OK, I have solved this problem in ODP version 1.2.2.0-105 and am installing ODP version 1.2.2.0-127.

[lucasbak]

Hi @SGITLOGIN,

Nice to hear !