search

running-elephant / datart

Datart is a next generation Data Visualization Open Platform
https://running-elephant.github.io/datart-docs/
Apache License 2.0
2k stars 591 forks source link

下载 数据图表 的图片和pdf 出现任务失败 #1340

Closed HUIHUIHUANG2018 closed 2 years ago

HUIHUIHUANG2018 commented 2 years ago

Datart版本号 datart-server-1.0.0-beta.3

错误描述 在数据图表开启下载任务，任务失败 下载为pdf，下载为图片报错 下载为excel 正常

如何重现 重现错误的步骤，例如：

  1. 到...页面
  2. 点击...
  3. 滚动到...
  4. 看到错误

截图

运行环境详情：(可选，取决于错误发生的具体位置)

其他内容 报错日志 Caused by: org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"css selector","selector":"#headlessBrowserRenderSign"} (Session info: headless chrome=101.0.4951.41) For documentation on this error, please visit: https://www.seleniumhq.org/exceptions/no_such_element.html Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03' System info: host: 'centos7', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '3.10.0-1062.1.2.el7.x86_64', java.version: '1.8.0_282' Driver info: org.openqa.selenium.remote.RemoteWebDriver Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 101.0.4951.41, chrome: {chromedriverVersion: 101.0.4951.41 (93c720db8323..., userDataDir: /tmp/.com.google.Chrome.RdmpGa}, goog:chromeOptions: {debuggerAddress: localhost:35114}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, proxy: Proxy(), se:cdp: ws://172.17.0.13:4444/sessi..., se:cdpVersion: 101.0.4951.41, se:vnc: ws://172.17.0.13:4444/sessi..., se:vncEnabled: true, se:vncLocalAddress: ws://172.17.0.13:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true}

scottsut commented 2 years ago

提供一下完整的错误日志吧

HUIHUIHUANG2018 commented 2 years ago

2022-05-16 02:03:17.218 ERROR datart.server.service.impl.DownloadServiceImpl : Download Task execute error org.openqa.selenium.TimeoutException: Expected condition failed: waiting for all conditions to be valid: presence of element located by: By.id: headlessBrowserRenderSign && presence of element located by: By.id: width && presence of element located by: By.id: height (tried for 60 second(s) with 500 milliseconds interval) at org.openqa.selenium.support.ui.WebDriverWait.timeoutException(WebDriverWait.java:95) at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:272) at datart.core.common.WebUtils.screenShot(WebUtils.java:70) at datart.core.common.WebUtils.screenShot2File(WebUtils.java:97) at datart.server.service.impl.AttachmentPdfServiceImpl.getFile(AttachmentPdfServiceImpl.java:65) at datart.server.service.impl.DownloadServiceImpl.lambda$submitDownloadTask$0(DownloadServiceImpl.java:95) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"css selector","selector":"#headlessBrowserRenderSign"} (Session info: headless chrome=101.0.4951.41) For documentation on this error, please visit: https://www.seleniumhq.org/exceptions/no_such_element.html Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03' System info: host: 'centos', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '3.10.0-1062.1.2.el7.x86_64', java.version: '1.8.0_282' Driver info: org.openqa.selenium.remote.RemoteWebDriver Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 101.0.4951.41, chrome: {chromedriverVersion: 101.0.4951.41 (93c720db8323..., userDataDir: /tmp/.com.google.Chrome.jzYng7}, goog:chromeOptions: {debuggerAddress: localhost:34050}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, proxy: Proxy(), se:cdp: ws://172.17.0.13:4444/sessi..., se:cdpVersion: 101.0.4951.41, se:vnc: ws://172.17.0.13:4444/sessi..., se:vncEnabled: true, se:vncLocalAddress: ws://172.17.0.13:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true} Session ID: d984969f977f0c99e94e6b90c746cc84 *** Element info: {Using=id, value=headlessBrowserRenderSign} at sun.reflect.GeneratedConstructorAccessor145.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:187) at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:122) at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49) at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158) at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552) at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:323) at org.openqa.selenium.remote.RemoteWebDriver.findElementById(RemoteWebDriver.java:372) at org.openqa.selenium.By$ById.findElement(By.java:188) at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:315) at org.openqa.selenium.support.ui.ExpectedConditions$6.apply(ExpectedConditions.java:182) at org.openqa.selenium.support.ui.ExpectedConditions$6.apply(ExpectedConditions.java:179) at org.openqa.selenium.support.ui.ExpectedConditions$48.apply(ExpectedConditions.java:1413) at org.openqa.selenium.support.ui.ExpectedConditions$48.apply(ExpectedConditions.java:1409) at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:249) ... 9 common frames omitted

HUIHUIHUANG2018 commented 2 years ago
server version comment
datart 1.0.0-beta.3
mysql 8.0 docker install
redis 7.0 docker install
selenium-chrome Google Chrome 101.0.4951.41 docker install
xiaohuangdou666 commented 2 years ago

需要提供一下前端访问地址、这条日志之前的生成分享链接的日志、以及datart的配置中的datart.address

HUIHUIHUANG2018 commented 2 years ago

我们内部nginx做了端口和ip的映射，datart.com=: datart.address=https://datart.qa.com

前端访问地址(内部做了映射，限制访问) https://datart.qa.com/

分享的连接为 https://datart.qa.com/shareChart/e833952821a543a980fac776ec3624c2?type=NONE

分享连接的log 2022-05-16 09:50:03.616 INFO d.server.service.impl.AttachmentPdfServiceImpl : share url https://datart.qa.com/shareChart/e833952821a543a980fac776ec3624c2?eager=true&type=NONE 2022-05-16 09:50:04.657 INFO org.openqa.selenium.remote.ProtocolHandshake : Detected dialect: W3C 2022-05-16 09:50:35.584 ERROR datart.server.service.impl.DownloadServiceImpl : Download Task execute error org.openqa.selenium.TimeoutException: Expected condition failed: waiting for all conditions to be valid: presence of element located by: By.id: headlessBrowserRenderSign && presence of element located by: By.id: width && presence of element located by: By.id: height (tried for 30 second(s) with 500 milliseconds interval) at org.openqa.selenium.support.ui.WebDriverWait.timeoutException(WebDriverWait.java:95) at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:272) at datart.core.common.WebUtils.screenShot(WebUtils.java:70) at datart.core.common.WebUtils.screenShot2File(WebUtils.java:97) at datart.server.service.impl.AttachmentPdfServiceImpl.getFile(AttachmentPdfServiceImpl.java:65) at datart.server.service.impl.DownloadServiceImpl.lambda$submitDownloadTask$0(DownloadServiceImpl.java:95) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"css selector","selector":"#headlessBrowserRenderSign"} (Session info: headless chrome=101.0.4951.41) For documentation on this error, please visit: https://www.seleniumhq.org/exceptions/no_such_element.html Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03' System info: host: 'centos', ip: '172.16.16.3', os.name: 'Linux', os.arch: 'amd64', os.version: '3.10.0-693.21.1.el7.x86_64', java.version: '1.8.0_201' Driver info: org.openqa.selenium.remote.RemoteWebDriver Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 101.0.4951.41, chrome: {chromedriverVersion: 101.0.4951.41 (93c720db8323..., userDataDir: /tmp/.com.google.Chrome.AjmcID}, goog:chromeOptions: {debuggerAddress: localhost:37351}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, proxy: Proxy(), se:cdp: ws://192.168.0.4:4444/sessi..., se:cdpVersion: 101.0.4951.41, se:vnc: ws://192.168.0.4:4444/sessi..., se:vncEnabled: true, se:vncLocalAddress: ws://192.168.0.4:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true} Session ID: 129619336dce1eb3df67adb76a8efb50 *** Element info: {Using=id, value=headlessBrowserRenderSign} at sun.reflect.GeneratedConstructorAccessor103.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:187) at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:122) at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49) at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158) at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552) at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:323) at org.openqa.selenium.remote.RemoteWebDriver.findElementById(RemoteWebDriver.java:372) at org.openqa.selenium.By$ById.findElement(By.java:188) at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:315) at org.openqa.selenium.support.ui.ExpectedConditions$6.apply(ExpectedConditions.java:182) at org.openqa.selenium.support.ui.ExpectedConditions$6.apply(ExpectedConditions.java:179) at org.openqa.selenium.support.ui.ExpectedConditions$48.apply(ExpectedConditions.java:1413) at org.openqa.selenium.support.ui.ExpectedConditions$48.apply(ExpectedConditions.java:1409) at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:249) ... 9 common frames omitted

HUIHUIHUANG2018 commented 2 years ago

你们的selenium-chrome是拉去最新版本101.0.4951.41 还是以前的版本

HUIHUIHUANG2018 commented 2 years ago

selenium/standalone-chrome 是docker部署，查看logs 没发现有报错， 所以可能还是前端css的问题 docker pull selenium/standalone-chrome docker run -p 4444:4444 -d --name selenium-chrome --shm-size="2g" selenium/standalone-chrome

-- datart.conf的配置地址 datart.webdriver-path=http://127.0.0.1:4444/wd/hub

因为发现datart-1.0.0-beta.3 发布时间是2022-04-22 而最近的selenium/standalone-chrome 是2022-04-27发布的 所以测试了安装旧版本的(100.0)的镜像(2022-04-05号发布) 发现还是一样的报错

xiaohuangdou666 commented 2 years ago

从日志来看是chrome访问分享地址超时，首先确认一下此分享地址能够访问，然后排查一下通过域名访问是否有限制，若有去除限制看看情况

HUIHUIHUANG2018 commented 2 years ago

谢谢排查网络的思路 目前问题已经解决，已经能够下载pdf和图片 在这里说下修改的2个地方

  1. 我上文提到的域名，是内部域名是nginx反向代理的，不是公网注册域名，所以他解析还是服务器上的 127.0.0.1:8080, 所以我修改了datart.conf的配置 
    
# server
server.port=8080
server.address=0.0.0.0

datart config

datart.address=https://datart.qa.com (nginx反向代理加签名的

datart.address=http://<服务器的公网ip>:8080


2. 还开通了自身通过公网ip访问自身服务的防火墙
```shell
>curl cip.cc
><服务器的公网ip>
>iptables -A INPUT -s <服务器的公网ip> -p tcp -m tcp --dport 8080 -j ACCEPT
  1. 还有一点截图的时间好长 
    2022-05-17 07:06:54.865 INFO  d.server.service.impl.AttachmentPdfServiceImpl : share url http://<服务器的公网ip>:8080/shareChart/2c26b6a965f44e7f8eb6c746c22c1b2c?eager=true&type=NONE 
2022-05-17 07:09:47.870 INFO  org.openqa.selenium.remote.ProtocolHandshake : Detected dialect: W3C
2022-05-17 07:10:00.243 INFO  d.server.service.impl.AttachmentPdfServiceImpl : create pdf file complete.

    从07:06:54到07:10:00 约3分钟了

scottsut commented 2 years ago
  1. 还有一点截图的时间好长 从07:06:54到07:10:00 约3分钟了

截图时间约等于查询完成时间，可以从优化查询角度来减少截图时间