search

code4craft / webmagic

A scalable web crawler framework for Java.
http://webmagic.io/
Apache License 2.0
11.34k stars 4.18k forks source link

method='null', extras=null, priority=0, headers={}, cookies={}} error java.lang.NullPointerException #656

Closed taroyutao closed 6 years ago

taroyutao commented 6 years ago

17-08-22 19:15:02,914 INFO us.codecraft.webmagic.downloader.HttpClientDownloader(HttpClientDownloader.java:88) ## downloading page success https://www.zhihu.com/question/33452102 17-08-22 19:15:02,928 ERROR us.codecraft.webmagic.Spider(Spider.java:324) ## process request Request{url='https://www.zhihu.com/question/33452102', method='null', extras=null, priority=0, headers={}, cookies={}} error java.lang.NullPointerException at buct.edu.cn.MySpiderTest.process(MySpiderTest.java:36) at us.codecraft.webmagic.Spider.onDownloadSuccess(Spider.java:414) at us.codecraft.webmagic.Spider.processRequest(Spider.java:406) at us.codecraft.webmagic.Spider.access$000(Spider.java:61) at us.codecraft.webmagic.Spider$1.run(Spider.java:320) at us.codecraft.webmagic.thread.CountableThreadPool$1.run(CountableThreadPool.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)

jacky-guo commented 6 years ago

问题在于：在自定义的Pipeline的里面无法自动注入。 解決：.addPipeline(new SaveSqlPipeline(Service)) 在new pipeline的時候把service当做参数传入，service在外面完成注入。