Closed penndu closed 2 years ago
调用页面为https://dusays.com/fcircle/
yml已经弃用很久了,请按照最新文档部署https://fcircle-doc.js.cool/
yml已经弃用很久了,请按照最新文档部署https://fcircle-doc.js.cool/
Dockerfile如下
FROM python:3.8
MAINTAINER yyyz
COPY . /
### 在这里配置环境变量
### 通用配置
ENV RUN_PER_HOURS=1
#ENV PROXY=""
### leancloud配置
#ENV APPID=""
#ENV APPKEY=""
### mysql配置
ENV MYSQL_USERNAME="USERNAME"
ENV MYSQL_PASSWORD="PASSWORD"
ENV MYSQL_IP="IP"
ENV MYSQL_PORT="3306"
ENV MYSQL_DB="DB"
### mongodb配置
#ENV MONGODB_URI=""
EXPOSE 8000
WORKDIR /
RUN cd ./hexo_circle_of_friends && pip3 install -r requirements.txt -i https://pypi.douban.com/simple/
CMD bash ./docker.sh
settings.py如下:
LINK = [
{
"link": "https://dusays.com/friends/", # 友链页地址1,修改为你的友链页地址
"theme": "volantis"
BLOCK_SITE = [
"https://dusays.com/",
# "https://example.com/",
]
# 过期文章清除(天)
OUTDATE_CLEAN = 600
# 存储方式,可选项:leancloud,mysql,sqlite,mongodb;默认为leancloud
DATABASE = "mysql"
# 部署方式,可选项:github,server,docker;默认为github
DEPLOY_TYPE = "docker"
api.log如下
INFO: Started server process [16]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO: 153.35.52.197:60246 - "GET /all HTTP/1.1" 200 OK
INFO: 153.35.52.191:47341 - "GET /favicon.ico HTTP/1.1" 404 Not Found
INFO: 61.147.73.164:43327 - "GET / HTTP/1.1" 404 Not Found
INFO: 61.147.73.164:49243 - "GET / HTTP/1.1" 404 Not Found
INFO: 61.147.73.164:20469 - "GET / HTTP/1.1" 404 Not Found
WARNING: Invalid HTTP request received.
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 136, in handle_events
event = self.conn.next_event()
File "/usr/local/lib/python3.8/site-packages/h11/_connection.py", line 485, in next_event
exc._reraise_as_remote_protocol_error()
File "/usr/local/lib/python3.8/site-packages/h11/_util.py", line 77, in _reraise_as_remote_protocol_error
raise self
File "/usr/local/lib/python3.8/site-packages/h11/_connection.py", line 467, in next_event
event = self._extract_next_receive_event()
File "/usr/local/lib/python3.8/site-packages/h11/_connection.py", line 409, in _extract_next_receive_event
event = self._reader(self._receive_buffer)
File "/usr/local/lib/python3.8/site-packages/h11/_readers.py", line 81, in maybe_read_from_IDLE_client
raise LocalProtocolError("illegal request line")
h11._util.RemoteProtocolError: illegal request line
INFO: 68.183.234.55:36770 - "GET / HTTP/1.1" 404 Not Found
INFO: 209.17.97.10:35923 - "GET / HTTP/1.1" 404 Not Found
INFO: 83.97.20.34:26630 - "GET / HTTP/1.0" 404 Not Found
INFO: 140.205.80.137:48063 - "GET /all?rule=created&start=0&end=20 HTTP/1.1" 200 OK
INFO: 120.27.74.170:60391 - "GET /all?rule=created&start=0&end=20 HTTP/1.1" 200 OK
INFO: 120.27.74.152:42392 - "GET /all HTTP/1.1" 200 OK
INFO: 120.27.74.159:22585 - "GET /favicon.ico HTTP/1.1" 404 Not Found
INFO: 120.27.74.170:49428 - "GET /all?rule=created&start=0&end=20 HTTP/1.1" 200 OK
crawler.log如下
Initialization complete
----------------------
友链总数 : 0
失联友链数 : 0
共 0 篇文章
最后运行于:2022-03-28 01:46:48
done!
Initialization complete
----------------------
友链总数 : 0
失联友链数 : 0
共 0 篇文章
最后运行于:2022-03-28 07:46:57
done!
Initialization complete
----------------------
友链总数 : 0
失联友链数 : 0
共 0 篇文章
最后运行于:2022-03-28 08:46:58
done!
第一个问题:感谢反馈,是我的疏忽。mysql api的连接问题已经紧急修复,尝试重新clone一下代码再部署
第二个问题:看了你的友链页面,抓不到文章应该是css未匹配。解决方案1:尝试适配友链,参考友链页适配;解决方案2:原理同解决方案1,尝试修改仓库内./hexo-circle-of-friends/get_url.py
下面的get_volantis_url
方法的抓取规则以适配你的前端页面;解决方案3:尝试使用配置项友链,在settings.py中配置即可;
~(第三个问题:github上暴露了你的mysql连接配置,注意安全,及时修改密码)~
请教这种的如何写抓取规则?
<div class='content'>
<ul class="list entry navigation">
<li><a class="flat-box" title="http://dusays.com/" href="http://dusays.com/" active-action="action-http:dusayscom">
<div class='name'>
<img src="https://cdn.dusays.com/avatar.png" class="lazyload" data-srcset="https://cdn.dusays.com/avatar.png" srcset="data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==">
杜老师说
</div>
</a></li>
</ul>
</div>
这样是否可行?
avatar = response.css('a.name img::attr(src)').extract()
link = response.css('a.flat-box a::attr(href)').extract()
name = response.css('a.name a::text').extract()
暂时不考虑对特例做适配。如果采用解决方案2,授人以渔,可以参考CSS 选择器参考手册;
另外补充解决方案4:尝试使用配置项友链的json_api
同样可以达到效果;
暂时不考虑对特例做适配。如果采用解决方案2,授人以渔,可以参考CSS 选择器参考手册; 另外补充解决方案4:尝试使用配置项友链的
json_api
同样可以达到效果;
非常感谢!
已经部署上了而且没有问题的话这个issue就可以close了
用的是Volantis主题,友链使用_data/sites.yml方式管理。