功能与意见反馈，报bug可以另开issue

[DormyMo]

都可以在这里交流，我会及时回复的~ 也欢迎加入QQ群讨论：389688974

[jxltom]

感谢作者分享，简单试用了一下，有几个想法

• Package

  • 添加setup.py和requirements.txt，start.sh放置到Python的scripts文件夹。这是因为既然entrypoint是两个Python文件，并且项目依赖于flask和requests，应该制作成可安装的package，利用pip install https://github.com/DormyMo/SpiderKeeper.git安装，自动安装依赖

  • 在启动时通过传入conf文件或者传入--scrapyd-server=localhost:6800添加scrapyd server

• 运行

  • 利用gunicorn等服务器运行flask。目前UI是用过flask.run()运行，但flask自带的服务器不适合生产环境

  • ScheduleMonitor.py利用线程5秒钟检查一下数据库是否会影响性能

• 其他

  • Http authentication

  • Python3 support

  谢谢

[DormyMo]

@jxltom 很不错的建议 ，谢谢 ，后续版本会改善这些问题。

[jxltom]

@DormyMo 好的。我看到UI是用flask运行的，请问nodejs在项目里发挥什么作用？

对于Flask的话，如果实现Http auth，Flask-BasicAuth插件几行代码就可以实现http basic authentication，可以考虑一下

[bosbyj]

您好，请问SpiderKeeper可以在win下跑起来吗？需要scrapyd？

[PythonYXY]

请教一下spider的优先级是什么意思，具体是怎样实现的？

[DormyMo]

@bosbyj SpiderKeeper 基于scrapyd服务，需要scrapyd，win下可以，不过没有测试过

[DormyMo]

@PythonYXY 现在优先级相当于 运行爬虫的机器数，优先级越高，会再越多的机器上运行同一个爬虫(分布式执行，需要scrapy-redis)

[PythonYXY]

@DormyMo 嗯，现在我就是正在做分布式爬虫。所以如果A的优先级比B高，就相当于A会比B多执行几次“scrapy crawl A”的命令么？

[kxxoling]

试用了下，spiderkeeper 直接提出了一些 warning，可以顺手解决一下：

/usr/local/lib/python2.7/dist-packages/SpiderKeeper/app/__init__.py:9: ExtDeprecationWarning: Importing flask.ext.basicauth is deprecated, use flask_basicauth instead.
  from flask.ext.basicauth import BasicAuth
/usr/local/lib/python2.7/dist-packages/SpiderKeeper/app/__init__.py:10: ExtDeprecationWarning: Importing flask.ext.restful is deprecated, use flask_restful instead.
  from flask.ext.restful import Api
/usr/local/lib/python2.7/dist-packages/SpiderKeeper/app/__init__.py:11: ExtDeprecationWarning: Importing flask.ext.restful_swagger is deprecated, use flask_restful_swagger instead.
  from flask.ext.restful_swagger import swagger
/usr/local/lib/python2.7/dist-packages/SpiderKeeper/app/__init__.py:11: ExtDeprecationWarning: Importing flask.ext.restful_swagger.swagger is deprecated, use flask_restful_swagger.swagger instead.
  from flask.ext.restful_swagger import swagger
/usr/local/lib/python2.7/dist-packages/flask_restful_swagger/swagger.py:14: ExtDeprecationWarning: Importing flask.ext.restful.fields is deprecated, use flask_restful.fields instead.
  from flask.ext.restful import Resource, fields
/usr/local/lib/python2.7/dist-packages/flask_sqlalchemy/__init__.py:839: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future.  Set it to True or False to suppress this warning.
  'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '
/usr/local/lib/python2.7/dist-packages/flask_sqlalchemy/__init__.py:839: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future.  Set it to True or False to suppress this warning.
  'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '
--------------------------------------------------------------------------------

[billsteve]

能不能加上监控简单脚本的功能啊