search

xiyuan-fengyu / ppspider

web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppeteer的web爬虫框架，提供灵活的任务队列管理调度方案，提供便捷的数据保存方案（nedb/mongodb），提供数据可视化和用户交互的实现方案
MIT License
334 stars 77 forks source link

第一个例子ppspider-webMonitor的web ui 没法正常显示 #19

Closed wokao98 closed 4 years ago

wokao98 commented 4 years ago

hi xiyuan,第一个例子编译完成启动后。http://localhost:9000/ 是空白页。 以下是启动info

D:\nodejs_10.16.0\node.exe C:\IdeaProject\ppspider-webMonitor\lib\App.js 2020-07-31 10:14:34.742 info new PuppeteerWorkerFactory (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\puppeteer\PuppeteerWorkerFactory.ts:12:16) init PuppeteerWorkerFactory ... 2020-07-31 10:14:34.751 info Object. (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\spider\decorators\Launcher.ts:166:16) init db(nedb://C:\IdeaProject\ppspider-webMonitor\lib/workplace/nedb) ... 2020-07-31 10:14:34.755 info nedbs.(anonymous function).Promise (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\common\db\NedbDao.ts:297:24) load nedb collection(job) from C:\IdeaProject\ppspider-webMonitor\lib/workplace/nedb/job.collection 2020-07-31 10:14:34.763 info nedbs.(anonymous function).Promise (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\common\db\NedbDao.ts:297:24) load nedb collection(networkTracing) from C:\IdeaProject\ppspider-webMonitor\lib/workplace/nedb/networkTracing.collection 2020-07-31 10:14:34.867 info Server.TimelineViewer.http.listen (C:\IdeaProject\ppspider-webMonitor\src\timeline-viewer\TimelineViewer.ts:20:20) C:\IdeaProject\ppspider-webMonitor\lib\timeline-viewer/../../src/timeline-viewer/web 2020-07-31 10:14:34.869 info Server.TimelineViewer.http.listen (C:\IdeaProject\ppspider-webMonitor\src\timeline-viewer\TimelineViewer.ts:21:20) The timeline-viewer ui server started successfully 2020-07-31 10:14:35.035 info nedb.loadDatabase.error (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\common\db\NedbDao.ts:314:32) load nedb collection(job) successfully 2020-07-31 10:14:35.091 info nedb.loadDatabase.error (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\common\db\NedbDao.ts:314:32) load nedb collection(networkTracing) successfully 2020-07-31 10:14:35.092 info Object. (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\spider\decorators\Launcher.ts:177:16) init db(nedb://C:\IdeaProject\ppspider-webMonitor\lib/workplace/nedb) successfully 2020-07-31 10:14:35.093 info Object. (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\spider\decorators\Launcher.ts:182:16) init QueueManager ... 2020-07-31 10:14:35.099 info Object. (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\spider\decorators\Launcher.ts:190:16) init QueueManager successfully 2020-07-31 10:14:35.172 info Server.WebServer.http.listen (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\spider\ui\WebServer.ts:71:20) The web ui server started successfully, have a look at http://localhost:9000 2020-07-31 10:14:35.283 info PuppeteerWorkerFactory.browser.puppeteer_1.launch.then.browser (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\puppeteer\PuppeteerWorkerFactory.ts:14:20) init PuppeteerWorkerFactory successfully

xiyuan-fengyu commented 4 years ago

确认一下 node_modules/ppspider/lib/spider/ui/web 文件夹下是否有内容，这个下面是web ui的静态文件

------------------ 原始邮件 ------------------ 发件人: "xiyuan-fengyu/ppspider" <notifications@github.com>; 发送时间: 2020年7月31日(星期五) 上午10:16 收件人: "xiyuan-fengyu/ppspider"<ppspider@noreply.github.com>; 抄送: "Subscribed"<subscribed@noreply.github.com>; 主题: [xiyuan-fengyu/ppspider] 第一个例子ppspider-webMonitor的web ui 没法正常显示 (#19)

hi xiyuan,第一个例子编译完成启动后。http://localhost:9000/ 是空白页。 以下是启动info

D:\nodejs_10.16.0\node.exe C:\IdeaProject\ppspider-webMonitor\lib\App.js 2020-07-31 10:14:34.742 info new PuppeteerWorkerFactory (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\puppeteer\PuppeteerWorkerFactory.ts:12:16) init PuppeteerWorkerFactory ... 2020-07-31 10:14:34.751 info Object. (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\spider\decorators\Launcher.ts:166:16) init db(nedb://C:\IdeaProject\ppspider-webMonitor\lib/workplace/nedb) ... 2020-07-31 10:14:34.755 info nedbs.(anonymous function).Promise (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\common\db\NedbDao.ts:297:24) load nedb collection(job) from C:\IdeaProject\ppspider-webMonitor\lib/workplace/nedb/job.collection 2020-07-31 10:14:34.763 info nedbs.(anonymous function).Promise (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\common\db\NedbDao.ts:297:24) load nedb collection(networkTracing) from C:\IdeaProject\ppspider-webMonitor\lib/workplace/nedb/networkTracing.collection 2020-07-31 10:14:34.867 info Server.TimelineViewer.http.listen (C:\IdeaProject\ppspider-webMonitor\src\timeline-viewer\TimelineViewer.ts:20:20) C:\IdeaProject\ppspider-webMonitor\lib\timeline-viewer/../../src/timeline-viewer/web 2020-07-31 10:14:34.869 info Server.TimelineViewer.http.listen (C:\IdeaProject\ppspider-webMonitor\src\timeline-viewer\TimelineViewer.ts:21:20) The timeline-viewer ui server started successfully 2020-07-31 10:14:35.035 info nedb.loadDatabase.error (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\common\db\NedbDao.ts:314:32) load nedb collection(job) successfully 2020-07-31 10:14:35.091 info nedb.loadDatabase.error (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\common\db\NedbDao.ts:314:32) load nedb collection(networkTracing) successfully 2020-07-31 10:14:35.092 info Object. (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\spider\decorators\Launcher.ts:177:16) init db(nedb://C:\IdeaProject\ppspider-webMonitor\lib/workplace/nedb) successfully 2020-07-31 10:14:35.093 info Object. (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\spider\decorators\Launcher.ts:182:16) init QueueManager ... 2020-07-31 10:14:35.099 info Object. (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\spider\decorators\Launcher.ts:190:16) init QueueManager successfully 2020-07-31 10:14:35.172 info Server.WebServer.http.listen (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\spider\ui\WebServer.ts:71:20) The web ui server started successfully, have a look at http://localhost:9000 2020-07-31 10:14:35.283 info PuppeteerWorkerFactory.browser.puppeteer_1.launch.then.browser (C:\IdeaProject\ppspider-webMonitor\node_modules\ppspider\src\puppeteer\PuppeteerWorkerFactory.ts:14:20) init PuppeteerWorkerFactory successfully

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

wokao98 commented 4 years ago

有的。在quickstart那个例子里 UI是正常的。换这个webMonitor例子就显示不出来了 image

xiyuan-fengyu commented 4 years ago

可能是ppspider的版本问题，换成最新的版本试试

------------------ 原始邮件 ------------------ 发件人: "wokao98"<notifications@github.com>; 发送时间: 2020年7月31日(星期五) 上午10:48 收件人: "xiyuan-fengyu/ppspider"<ppspider@noreply.github.com>; 抄送: "西渊风雨"<847639468@qq.com>; "Comment"<comment@noreply.github.com>; 主题: Re: [xiyuan-fengyu/ppspider] 第一个例子ppspider-webMonitor的web ui 没法正常显示 (#19)

有的。在quickstart那个例子里 UI是正常的。换这个webMonitor例子就显示不出来了

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

xiyuan-fengyu commented 4 years ago

ppspider-webMonitor 这个项目中package.json 里面配置的是2.1.2，不是最新的，最新的是 2.2.4-preview.1595667732337

------------------ 原始邮件 ------------------ 发件人: "wokao98"<notifications@github.com>; 发送时间: 2020年7月31日(星期五) 上午10:56 收件人: "xiyuan-fengyu/ppspider"<ppspider@noreply.github.com>; 抄送: "西渊风雨"<847639468@qq.com>; "Comment"<comment@noreply.github.com>; 主题: Re: [xiyuan-fengyu/ppspider] 第一个例子ppspider-webMonitor的web ui 没法正常显示 (#19)

从npm install过来的依赖是 ppspider@2.1.2，应该是最新的了吧

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

wokao98 commented 4 years ago

我把2.2.4的放到IJ里面auto build 出了些问题。。是不是不能这么编译的呢

`D:\nodejs_10.16.0\node.exe D:\nodejs_10.16.0\node_modules\npm\bin\npm-cli.js run "auto build" --scripts-prepend-node-path=auto [11:03:26 AM] Starting compilation in watch mode...

src/common/bean/Bean.ts:129:40 - error TS2339: Property 'getMetadata' does not exist on type 'typeof Reflect'.

129 const paramTypes = Reflect.getMetadata('design:paramtypes', target, fieldOrMethod);


src/common/bean/Bean.ts:140:39 - error TS2339: Property 'getMetadata' does not exist on type 'typeof Reflect'.

140             const fieldType = Reflect.getMetadata('design:type', target, fieldOrMethod);

src/common/util/RequestUtil.ts:3:33 - error TS2307: Cannot find module 'http-proxy-agent' or its corresponding type declarations.

3 import * as HttpProxyAgent from 'http-proxy-agent';


src/common/util/RequestUtil.ts:5:34 - error TS2307: Cannot find module 'socks-proxy-agent' or its corresponding type declarations.

5 import * as SocksProxyAgent from 'socks-proxy-agent';

src/spider/decorators/FromQueue.ts:19:40 - error TS2339: Property 'getMetadata' does not exist on type 'typeof Reflect'.

19 config["paramtypes"] = Reflect.getMetadata('design:paramtypes', target, key);


src/spider/decorators/OnStart.ts:18:40 - error TS2339: Property 'getMetadata' does not exist on type 'typeof Reflect'.

18         config["paramtypes"] = Reflect.getMetadata('design:paramtypes', target, key);

src/spider/decorators/OnTime.ts:18:40 - error TS2339: Property 'getMetadata' does not exist on type 'typeof Reflect'.

18 config["paramtypes"] = Reflect.getMetadata('design:paramtypes', target, key);


src/spider/manager/JobManager.ts:153:48 - error TS2769: No overload matches this call.
  Overload 1 of 5, '(value: string | number | Date): Date', gave the following error.
    Argument of type 'Number' is not assignable to parameter of type 'string | number | Date'.
      Type 'Number' is not assignable to type 'number'.
        'number' is a primitive, but 'Number' is a wrapper object. Prefer using 'number' when possible.
  Overload 2 of 5, '(vd: VarDate): Date', gave the following error.
    Argument of type 'Number' is not assignable to parameter of type 'VarDate'.
      Property 'VarDate_typekey' is missing in type 'Number' but required in type 'VarDate'.
  Overload 3 of 5, '(value: string | number): Date', gave the following error.
    Argument of type 'Number' is not assignable to parameter of type 'string | number'.
      Type 'Number' is not assignable to type 'number'.
        'number' is a primitive, but 'Number' is a wrapper object. Prefer using 'number' when possible.

153                 return DateUtil.toStr(new Date(value));

../typescript/lib/lib.scripthost.d.ts:318:13 318 private VarDate_typekey: VarDate;

    'VarDate_typekey' is declared here.

src/test/proxy/httpsProxyTest.ts:4:33 - error TS2307: Cannot find module 'http-proxy-agent' or its corresponding type declarations.

4 import * as HttpProxyAgent from 'http-proxy-agent';

src/test/proxy/httpsProxyTest1.ts:4:33 - error TS2307: Cannot find module 'http-proxy-agent' or its corresponding type declarations.

4 import * as HttpProxyAgent from 'http-proxy-agent';


src/test/proxy/MidManServer.ts:5:33 - error TS2307: Cannot find module 'http-proxy-agent' or its corresponding type declarations.

5 import * as HttpProxyAgent from 'http-proxy-agent';

src/test/proxy/MidManServer.ts:15:31 - error TS2345: Argument of type 'string | string[] | ParsedQs | ParsedQs[]' is not assignable to parameter of type 'string'. Type 'string[]' is not assignable to type 'string'.

15 const options = url.parse(req.query.url);


src/test/proxy/MidManServer.ts:17:37 - error TS2345: Argument of type 'string | string[] | ParsedQs | ParsedQs[]' is not assignable to parameter of type 'string'.
  Type 'string[]' is not assignable to type 'string'.

17     options["headers"] = JSON.parse(req.query.headers);

src/test/proxy/MidManServer.ts:54:48 - error TS2345: Argument of type 'string | string[] | ParsedQs | ParsedQs[]' is not assignable to parameter of type 'string | HttpsProxyAgentOptions'. Type 'string[]' is not assignable to type 'string | HttpsProxyAgentOptions'. Type 'string[]' is missing the following properties from type 'HttpsProxyAgentOptions': host, port

54 options["agent"] = new HttpsProxyAgent(proxy);


src/test/proxy/PuppeteerPageProxyTest.ts:6:33 - error TS2307: Cannot find module 'http-proxy-agent' or its corresponding type declarations.

6 import * as HttpProxyAgent from 'http-proxy-agent';

src/test/reflect/ReflectTest.ts:7:35 - error TS2339: Property 'getMetadata' does not exist on type 'typeof Reflect'.

7 const providers = Reflect.getMetadata('design:paramtypes', target, key);



[11:03:33 AM] Found 16 errors. Watching for file changes.

`
xiyuan-fengyu commented 4 years ago

我更新了一下 ppspider-webMonitor 这个项目，拖一下最新代码试试

------------------ 原始邮件 ------------------ 发件人: "wokao98"<notifications@github.com>; 发送时间: 2020年7月31日(星期五) 中午11:10 收件人: "xiyuan-fengyu/ppspider"<ppspider@noreply.github.com>; 抄送: "西渊风雨"<847639468@qq.com>; "Comment"<comment@noreply.github.com>; 主题: Re: [xiyuan-fengyu/ppspider] 第一个例子ppspider-webMonitor的web ui 没法正常显示 (#19)

我把2.2.4的放到IJ里面auto build 出了些问题。。是不是不能这么编译的呢

`D:\nodejs_10.16.0\node.exe D:\nodejs_10.16.0\node_modules\npm\bin\npm-cli.js run "auto build" --scripts-prepend-node-path=auto [11:03:26 AM] Starting compilation in watch mode...

src/common/bean/Bean.ts:129:40 - error TS2339: Property 'getMetadata' does not exist on type 'typeof Reflect'.

129 const paramTypes = Reflect.getMetadata('design:paramtypes', target, fieldOrMethod);


src/common/bean/Bean.ts:140:39 - error TS2339: Property 'getMetadata' does not exist on type 'typeof Reflect'.

140             const fieldType = Reflect.getMetadata('design:type', target, fieldOrMethod);

src/common/util/RequestUtil.ts:3:33 - error TS2307: Cannot find module 'http-proxy-agent' or its corresponding type declarations.

3 import * as HttpProxyAgent from 'http-proxy-agent';


src/common/util/RequestUtil.ts:5:34 - error TS2307: Cannot find module 'socks-proxy-agent' or its corresponding type declarations.

5 import * as SocksProxyAgent from 'socks-proxy-agent';

src/spider/decorators/FromQueue.ts:19:40 - error TS2339: Property 'getMetadata' does not exist on type 'typeof Reflect'.

19 config["paramtypes"] = Reflect.getMetadata('design:paramtypes', target, key);


src/spider/decorators/OnStart.ts:18:40 - error TS2339: Property 'getMetadata' does not exist on type 'typeof Reflect'.

18         config["paramtypes"] = Reflect.getMetadata('design:paramtypes', target, key);

src/spider/decorators/OnTime.ts:18:40 - error TS2339: Property 'getMetadata' does not exist on type 'typeof Reflect'.

18 config["paramtypes"] = Reflect.getMetadata('design:paramtypes', target, key);


src/spider/manager/JobManager.ts:153:48 - error TS2769: No overload matches this call.
 Overload 1 of 5, '(value: string | number | Date): Date', gave the following error.
 Argument of type 'Number' is not assignable to parameter of type 'string | number | Date'.
 Type 'Number' is not assignable to type 'number'.
 'number' is a primitive, but 'Number' is a wrapper object. Prefer using 'number' when possible.
 Overload 2 of 5, '(vd: VarDate): Date', gave the following error.
 Argument of type 'Number' is not assignable to parameter of type 'VarDate'.
 Property 'VarDate_typekey' is missing in type 'Number' but required in type 'VarDate'.
 Overload 3 of 5, '(value: string | number): Date', gave the following error.
 Argument of type 'Number' is not assignable to parameter of type 'string | number'.
 Type 'Number' is not assignable to type 'number'.
 'number' is a primitive, but 'Number' is a wrapper object. Prefer using 'number' when possible.

153                 return DateUtil.toStr(new Date(value));

../typescript/lib/lib.scripthost.d.ts:318:13 318 private VarDate_typekey: VarDate;

 'VarDate_typekey' is declared here.

src/test/proxy/httpsProxyTest.ts:4:33 - error TS2307: Cannot find module 'http-proxy-agent' or its corresponding type declarations.

4 import * as HttpProxyAgent from 'http-proxy-agent';

src/test/proxy/httpsProxyTest1.ts:4:33 - error TS2307: Cannot find module 'http-proxy-agent' or its corresponding type declarations.

4 import * as HttpProxyAgent from 'http-proxy-agent';


src/test/proxy/MidManServer.ts:5:33 - error TS2307: Cannot find module 'http-proxy-agent' or its corresponding type declarations.

5 import * as HttpProxyAgent from 'http-proxy-agent';

src/test/proxy/MidManServer.ts:15:31 - error TS2345: Argument of type 'string | string[] | ParsedQs | ParsedQs[]' is not assignable to parameter of type 'string'. Type 'string[]' is not assignable to type 'string'.

15 const options = url.parse(req.query.url);


src/test/proxy/MidManServer.ts:17:37 - error TS2345: Argument of type 'string | string[] | ParsedQs | ParsedQs[]' is not assignable to parameter of type 'string'.
 Type 'string[]' is not assignable to type 'string'.

17     options["headers"] = JSON.parse(req.query.headers);

src/test/proxy/MidManServer.ts:54:48 - error TS2345: Argument of type 'string | string[] | ParsedQs | ParsedQs[]' is not assignable to parameter of type 'string | HttpsProxyAgentOptions'. Type 'string[]' is not assignable to type 'string | HttpsProxyAgentOptions'. Type 'string[]' is missing the following properties from type 'HttpsProxyAgentOptions': host, port

54 options["agent"] = new HttpsProxyAgent(proxy);


src/test/proxy/PuppeteerPageProxyTest.ts:6:33 - error TS2307: Cannot find module 'http-proxy-agent' or its corresponding type declarations.

6 import * as HttpProxyAgent from 'http-proxy-agent';

src/test/reflect/ReflectTest.ts:7:35 - error TS2339: Property 'getMetadata' does not exist on type 'typeof Reflect'.

7 const providers = Reflect.getMetadata('design:paramtypes', target, key);



[11:03:33 AM] Found 16 errors. Watching for file changes.

`

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
wokao98 commented 4 years ago

UI正常了，谢谢