code4craft / webmagic

A scalable web crawler framework for Java.
http://webmagic.io/
Apache License 2.0
11.45k stars 4.18k forks source link

webmagic0.74 RedisScheduler无法使用 #979

Open FXyaru opened 3 years ago

FXyaru commented 3 years ago

springboot2.2.6 ,无法启动,直接报错: java.lang.IllegalAccessError: tried to access method redis.clients.jedis.JedisPool.returnResource(Lredis/clients/jedis/Jedis;)V from class us.codecraft.webmagic.scheduler.RedisScheduler

代码: @Bean public JedisPool redisPoolFactory() { JedisPoolConfig jedisPoolConfig = new JedisPoolConfig(); jedisPoolConfig.setMaxIdle(maxIdle); jedisPoolConfig.setMaxWaitMillis(maxWaitMillis); return new JedisPool(jedisPoolConfig,host,port,timeout,password); }

Spider.create(new CrawlerProcessor()) .addUrl(muLuUrl) // 设置Secheduler` .setScheduler(new RedisScheduler(jedisPool)) .thread(10) // 设置自定义的Pipeline储存数据 .addPipeline(new MybatisPipeline()) .run(); 后发现是需要回调redisPool中的returnResource该方法,3.2.0的版本已弃用,改成了closed方法。 故降低版本到springboot2.2.6,redis版本到2.9.3. 发现项目能启动,但是不爬取页面! 异常信息: 2021-01-05 18:12:38.568 INFO [main] u.c.w.Spider.run(306): Spider e.dangdang.com started! 2021-01-05 18:12:38.569 INFO [main] u.c.w.Spider.run(338): Spider e.dangdang.com closed! 0 pages downloaded.

sutra commented 3 years ago

升级了一下依赖: 0e01550a79883e7df6c0bd8d0b0ab31156a9412a