chenshuai1995 commented 6 months ago

Before asking

[X] I have read the How to ask for help (我已阅读如何寻求帮助指引文档).

Your environment

Linkis version used: 1.4.0
Environment name and version:
- hdp-3.1.5（和Hadoop集群版本无关）
- dolphinscheduler-1.3.9

Describe your questions

Q1. 大家好，我现在在用官网提供好编译后的包DSS1.1.2&Linkis1.4.0，在集成DolphinScheduler的时候遇到了点问题。有没有小伙伴，有这方面的经验，能支持我一下？

目前是安装官网提供的包和文档安装的，版本都是按照文档要求搭建的。 1、其中用到的相关jar包，如dss-dolphinscheduler-token.jar，dolphinscheduler-prod-metrics都是从Maven中央仓库下载的对应1.1.2的版本 2、1.1.1文档中对应旧的微服务，已按照DSS-1096，修改对应的配置了

遇到的问题： DSS集成后，点击顶部DolphinScheduler菜单，提示访问调度接口401（海豚调度的admin账户已经token确定是对的，以及在有效期内）

本地相关的核心配置，如下：

dss_linkis为dss_linkis安装目录

1. 查看一下dss_linkis的目录结构

ll dss_linkis

结果

-rw-r--r-- 1 hadoop hadoop 576756467 Aug 4 2023 apache-linkis-1.4.0-bin.tar.gz drwxr-xr-x 2 hadoop hadoop 100 May 22 15:52 bin drwxr-xr-x 2 hadoop hadoop 36 May 23 18:01 conf drwxr-xr-x 9 hadoop hadoop 147 May 22 15:57 dss drwxr-xr-x 9 hadoop hadoop 147 May 21 16:42 dss_bak drwxr-xr-x 10 hadoop hadoop 98 May 22 15:55 linkis drwxr-xr-x 2 hadoop hadoop 6 May 21 16:37 linkis-back-1716280671 drwxr-xr-x 10 hadoop hadoop 98 May 21 16:40 linkis-back-1716364422 drwxr-xr-x 4 hadoop hadoop 64 May 22 15:55 web -rw-r--r-- 1 hadoop hadoop 642954023 Aug 25 2023 wedatasphere-dss-1.1.2-dist.tar.gz -rw-r--r-- 1 hadoop hadoop 22114764 Aug 4 2023 wedatasphere-dss-web-1.1.2-dist.zip

2. 查看一下dss的目录结构

ll dss_linkis/dss

结果

drwxrwxrwx 2 hadoop hadoop 181 May 22 15:54 bin drwxrwxrwx 2 hadoop hadoop 214 May 22 16:13 conf drwxrwxrwx 11 hadoop hadoop 284 May 23 18:03 dss-appconns drwxrwxrwx 5 hadoop hadoop 66 May 22 18:34 lib -rwxrwxrwx 1 hadoop hadoop 11558 Aug 25 2023 LICENSE drwxr-xr-x 2 root root 151 May 22 15:57 logs drwxr-xr-x 2 root root 55 May 23 18:03 pid -rwxrwxrwx 1 hadoop hadoop 17021 Aug 25 2023 README.md -rwxrwxrwx 1 hadoop hadoop 15562 Aug 25 2023 README-ZH.md drwxrwxrwx 5 hadoop hadoop 125 May 22 15:57 sbin

3. 查看一下dss插件的目录结构

ll dss_linkis/dss/dss-appconns/

结果

drwxrwxrwx 6 hadoop hadoop 78 May 22 15:54 datachecker drwxrwxrwx 5 hadoop hadoop 92 May 23 18:03 dolphinscheduler -rw-r--r-- 1 root root 82340948 May 23 18:03 dolphinscheduler.zip drwxrwxrwx 6 hadoop hadoop 52 Aug 25 2023 eventchecker -rwxrwxrwx 1 hadoop hadoop 166075986 Aug 25 2023 linkis-engineplugin-appconn.zip drwxrwxrwx 5 hadoop hadoop 65 May 22 15:54 schedulis drwxrwxrwx 4 hadoop hadoop 56 May 22 15:57 scriptis -rw-r--r-- 1 root root 75221 May 22 15:57 scriptis.zip drwxrwxrwx 6 hadoop hadoop 78 May 22 15:54 sendemail drwxrwxrwx 5 hadoop hadoop 65 May 22 15:54 sparketl drwxrwxrwx 4 hadoop hadoop 56 May 22 15:57 sso -rw-r--r-- 1 root root 3028 May 22 15:57 sso.zip drwxrwxrwx 4 hadoop hadoop 82 May 22 15:57 workflow -rw-r--r-- 1 root root 22367 May 22 15:57 workflow.zip

4. 查看一下dss的dolphinscheduler插件的目录结构

ll dss_linkis/dss/dss-appconns/dolphinscheduler

结果

-rwxrwxrwx 1 hadoop hadoop 1359 May 23 18:00 appconn.properties drwxrwxrwx 2 hadoop hadoop 48 May 22 15:54 conf drwxrwxrwx 2 hadoop hadoop 43 May 22 16:04 db -rw-r--r-- 1 root root 0 May 23 18:03 index_v000006.index drwxrwxrwx 2 hadoop hadoop 12288 May 22 19:43 lib

5. 查看一下dss的dolphinscheduler插件的配置

cat dss_linkis/dss/dss-appconns/dolphinscheduler/appconn.properties |grep wds

结果

wds.dss.appconn.ds.admin.user=admin

wds.dss.appconn.ds.admin.user=hadoop

wds.dss.appconn.ds.admin.token=0de582f96a3b7eeb9bf3204e607b66d8

wds.dss.appconn.ds.admin.token=4ee0e7e5bbfa53ad8442b36569b5088c wds.dss.appconn.ds.version=1.3.9 wds.dss.appconn.ds.client.home=/opt/app/dss-dolphinscheduler-client

this property, the real url will be http://ip:port/${wds.dss.appconn.ds.url.prefix}/users/create

wds.dss.appconn.ds.url.prefix=

6. 查看一下dss的dss-dolphinscheduler-token包

find dss_linkis -name dss-dolphinscheduler-token*.jar

结果

dss_linkis/dss/lib/dss-server/dss-dolphinscheduler-token-1.1.2.jar

7. 查看一下nginx中dolphinscheduler的配置

cat /etc/nginx/conf.d/dss.conf |grep -A 5 /dolphinscheduler

结果

        location /dolphinscheduler {
        proxy_pass http://xx.xx.xx.xx:12345;#后端dolphinscheduler服务的地址
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection upgrade;
        }

8. 查看调度中心的配置

cat dss_linkis/dss/conf/dss-server.properties |grep schedulerCenter

结果

wds.dss.workflow.schedulerCenter.url=http://xx.xx.xx.xx:12345/dolphinscheduler

因为，官网的安装版本是基于dss1.1.1的。如果有基于dss1.1.2安装成功的，麻烦对比一下，上面的配置是否有问题，或者有遗漏。谢谢！

Eureka service list

Some logs info or acctch file

linkis-xxx.log:


<!--日志文字贴到这里-->
curl 'http://xx.xx.xx.xx:12345/dolphinscheduler/projects/analysis/define-user-count?redirect=http%3A%2F%2Fxx.xx.xx.xx%3A12345%2Fdolphinscheduler&dssurl=http%3A%2F%2Fdev-chens-vnode001%3A9001&cookies=linkis_user_session_ticket_id_v1%3DM7UZXQP9Ld2QWfUv6uxVAGsHMp3ZIHsG2HLiVea4FcQ%3D%3Bdataworkcloud_inner_request%3Dtrue%3BworkspaceName%3DbdapWorkspace%3BworkspaceId%3D224&workspace=bdapWorkspace&appName=dolphinscheduler&dsspath=%2F' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7' \
  -H 'Accept-Language: zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6' \
  -H 'Connection: keep-alive' \
  -H 'Referer: http://xx.xx.xx.xx:8085/' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0' \
  --insecure  

目测，这个接口，header头里缺少了sessionId，DolphinScheduler中应该是基于seeesionId进行免密认证的。

log file:

linkis-xxx.log

github-actions[bot] commented 6 months ago

:blush: Welcome to the Apache Linkis community!!

We are glad that you are contributing by opening this issue.

Please make sure to include all the relevant context. We will be here shortly.

If you are interested in contributing to our website project, please let us know! You can check out our contributing guide on :point_right: How to Participate in Project Contribution.

Community

WeChat Assistant	WeChat Public Account

Mailing Lists

Name	Description	Subscribe	Unsubscribe	Archive
dev@linkis.apache.org	community activity information	subscribe	unsubscribe	archive

peacewong commented 5 months ago

Hello, you can submit PR support

apache / linkis

[Question] 使用官网提供的DSS1.1.2&Linkis1.4.0编译后的包，集成dolphinscheduler1.3.9，出现401免密问题 #5122