search

my8100 / logparser

A tool for parsing Scrapy log files periodically and incrementally, extending the HTTP JSON API of Scrapyd.
GNU General Public License v3.0
89 stars 22 forks source link

I run logparser use docker，but get some error #15

Closed luzihang123 closed 5 years ago

luzihang123 commented 5 years ago

Dockfile：

FROM python:3.7
ADD . /code
WORKDIR /code
COPY ./scrapyd.conf /etc/scrapyd/
EXPOSE 6800
RUN echo "Asia/Shanghai" > /etc/timezone && \
    ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN pip3 install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
RUN pip install --upgrade logparser
RUN ["chmod", "777", "entrypoint.sh"]
ENTRYPOINT ["./entrypoint.sh"]

entrypoint.sh

#!/bin/bash
cd ~/code
mkdir logs
logparser -dir /code/logs -t 10 --delete_json_files & scrapyd

Scrapyd.conf

[scrapyd]
eggs_dir    = eggs
logs_dir    = logs
items_dir   =
jobs_to_keep = 5
dbs_dir     = dbs
max_proc    = 0
max_proc_per_cpu = 50
finished_to_keep = 100
poll_interval = 5.0
bind_address = 0.0.0.0
http_port   = 6800
debug       = off
runner      = scrapyd.runner
application = scrapyd.app.application
launcher    = scrapyd.launcher.Launcher
webroot     = scrapyd.website.Root

[services]
schedule.json     = scrapyd.webservice.Schedule
cancel.json       = scrapyd.webservice.Cancel
addversion.json   = scrapyd.webservice.AddVersion
listprojects.json = scrapyd.webservice.ListProjects
listversions.json = scrapyd.webservice.ListVersions
listspiders.json  = scrapyd.webservice.ListSpiders
delproject.json   = scrapyd.webservice.DeleteProject
delversion.json   = scrapyd.webservice.DeleteVersion
listjobs.json     = scrapyd.webservice.ListJobs
daemonstatus.json = scrapyd.webservice.DaemonStatus

docker log：

2019/8/20 下午3:13:23[2019-08-20 15:13:23,199] INFO     in logparser.run: LogParser version: 0.8.2
2019/8/20 下午3:13:23[2019-08-20 15:13:23,200] INFO     in logparser.run: Use 'logparser -h' to get help
2019/8/20 下午3:13:23[2019-08-20 15:13:23,200] INFO     in logparser.run: Main pid: 12
2019/8/20 下午3:13:23[2019-08-20 15:13:23,200] INFO     in logparser.run: Check out the config file below for more advanced settings.
2019/8/20 下午3:13:23
2019/8/20 下午3:13:23****************************************************************************************************
2019/8/20 下午3:13:23Loading settings from /usr/local/lib/python3.7/site-packages/logparser/settings.py
2019/8/20 下午3:13:23****************************************************************************************************
2019/8/20 下午3:13:23
2019/8/20 下午3:13:23[2019-08-20 15:13:23,201] DEBUG    in logparser.run: Reading settings from command line: Namespace(delete_json_files=True, disable_telnet=False, main_pid=0, scrapyd_logs_dir='/code/logs', scrapyd_server='127.0.0.1:6800', sleep='10', verbose=False)
2019/8/20 下午3:13:23[2019-08-20 15:13:23,201] DEBUG    in logparser.run: Checking config
2019/8/20 下午3:13:23[2019-08-20 15:13:23,201] INFO     in logparser.run: SCRAPYD_SERVER: 127.0.0.1:6800
2019/8/20 下午3:13:23[2019-08-20 15:13:23,201] INFO     in logparser.run: SCRAPYD_LOGS_DIR: /code/logs
2019/8/20 下午3:13:23[2019-08-20 15:13:23,201] INFO     in logparser.run: PARSE_ROUND_INTERVAL: 10
2019/8/20 下午3:13:23[2019-08-20 15:13:23,201] INFO     in logparser.run: ENABLE_TELNET: True
2019/8/20 下午3:13:23[2019-08-20 15:13:23,201] INFO     in logparser.run: DELETE_EXISTING_JSON_FILES_AT_STARTUP: True
2019/8/20 下午3:13:23[2019-08-20 15:13:23,201] INFO     in logparser.run: VERBOSE: False
2019/8/20 下午3:13:23
2019/8/20 下午3:13:23****************************************************************************************************
2019/8/20 下午3:13:23Visit stats at: http://127.0.0.1:6800/logs/stats.json
2019/8/20 下午3:13:23****************************************************************************************************
2019/8/20 下午3:13:23
2019/8/20 下午3:13:232019-08-20T15:13:23+0800 [-] Removing stale pidfile /code/twistd.pid
2019/8/20 下午3:13:232019-08-20T15:13:23+0800 [-] Loading /usr/local/lib/python3.7/site-packages/scrapyd/txapp.py...
2019/8/20 下午3:13:232019-08-20T15:13:23+0800 [-] Scrapyd web console available at http://0.0.0.0:6800/
2019/8/20 下午3:13:232019-08-20T15:13:23+0800 [-] Loaded.
2019/8/20 下午3:13:232019-08-20T15:13:23+0800 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 18.9.0 (/usr/local/bin/python 3.7.4) starting up.
2019/8/20 下午3:13:232019-08-20T15:13:23+0800 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.epollreactor.EPollReactor.
2019/8/20 下午3:13:232019-08-20T15:13:23+0800 [-] Site starting on 6800
2019/8/20 下午3:13:232019-08-20T15:13:23+0800 [twisted.web.server.Site#info] Starting factory <twisted.web.server.Site object at 0x7fee8f976190>
2019/8/20 下午3:13:232019-08-20T15:13:23+0800 [Launcher] Scrapyd 1.2.0 started: max_proc=200, runner='scrapyd.runner'
2019/8/20 下午3:13:232019-08-20T15:13:23+0800 [twisted.python.log#info] "192.168.95.55" - - [20/Aug/2019:07:13:23 +0000] "GET /daemonstatus.json HTTP/1.1" 200 89 "-" "python-requests/2.22.0"
2019/8/20 下午3:13:25[2019-08-20 15:13:25,131] INFO     in logparser.utils: Running the latest version: 0.8.2
2019/8/20 下午3:13:25[2019-08-20 15:13:25,134] WARNING  in logparser.logparser: Deleted /code/logs/risk_control_info/comsumer_company_share_change_info_alipay/2019-08-20T06_22_32.json
2019/8/20 下午3:13:25[2019-08-20 15:13:25,134] WARNING  in logparser.logparser: Deleted /code/logs/risk_control_info/job_51/2019-08-20T06_28_21.json
2019/8/20 下午3:13:28[2019-08-20 15:13:28,137] WARNING  in logparser.logparser: New logfile found: /code/logs/risk_control_info/comsumer_company_share_change_info_alipay/2019-08-20T06_22_32.log (5234 bytes)
2019/8/20 下午3:13:28[2019-08-20 15:13:28,137] WARNING  in logparser.logparser: Json file not found: /code/logs/risk_control_info/comsumer_company_share_change_info_alipay/2019-08-20T06_22_32.json
2019/8/20 下午3:13:28[2019-08-20 15:13:28,138] WARNING  in logparser.logparser: New logfile: /code/logs/risk_control_info/comsumer_company_share_change_info_alipay/2019-08-20T06_22_32.log (5234 bytes) -> parse
2019/8/20 下午3:13:28[2019-08-20 15:13:28,158] WARNING  in logparser.logparser: Saved to /code/logs/risk_control_info/comsumer_company_share_change_info_alipay/2019-08-20T06_22_32.json
2019/8/20 下午3:13:28[2019-08-20 15:13:28,158] WARNING  in logparser.logparser: New logfile found: /code/logs/risk_control_info/job_51/2019-08-20T06_28_21.log (27072657 bytes)
2019/8/20 下午3:13:28[2019-08-20 15:13:28,158] WARNING  in logparser.logparser: Json file not found: /code/logs/risk_control_info/job_51/2019-08-20T06_28_21.json
2019/8/20 下午3:13:28[2019-08-20 15:13:28,158] WARNING  in logparser.logparser: New logfile: /code/logs/risk_control_info/job_51/2019-08-20T06_28_21.log (27072657 bytes) -> parse
2019/8/20 下午3:13:302019-08-20T15:13:30+0800 [twisted.python.log#info] "192.168.95.55" - - [20/Aug/2019:07:13:30 +0000] "GET /daemonstatus.json HTTP/1.1" 200 89 "-" "python-requests/2.22.0"
2019/8/20 下午3:13:342019-08-20T15:13:34+0800 [twisted.python.log#info] "192.168.95.55" - - [20/Aug/2019:07:13:33 +0000] "GET /daemonstatus.json HTTP/1.1" 200 89 "-" "python-requests/2.22.0"
2019/8/20 下午3:13:34[2019-08-20 15:13:34,183] WARNING  in logparser.logparser: Saved to /code/logs/risk_control_info/job_51/2019-08-20T06_28_21.json
2019/8/20 下午3:13:34[2019-08-20 15:13:34,184] WARNING  in logparser.logparser: Saved to http://127.0.0.1:6800/logs/stats.json
2019/8/20 下午3:13:34[2019-08-20 15:13:34,184] WARNING  in logparser.logparser: Sleep 10 seconds
2019/8/20 下午3:13:412019-08-20T15:13:41+0800 [twisted.python.log#info] "192.168.95.55" - - [20/Aug/2019:07:13:40 +0000] "GET /daemonstatus.json HTTP/1.1" 200 89 "-" "python-requests/2.22.0"
2019/8/20 下午3:13:432019-08-20T15:13:43+0800 [twisted.python.log#info] "192.168.95.55" - - [20/Aug/2019:07:13:43 +0000] "GET /daemonstatus.json HTTP/1.1" 200 89 "-" "python-requests/2.22.0"
2019/8/20 下午3:13:44[2019-08-20 15:13:44,186] WARNING  in logparser.logparser: Saved to http://127.0.0.1:6800/logs/stats.json
2019/8/20 下午3:13:44[2019-08-20 15:13:44,186] WARNING  in logparser.logparser: Sleep 10 seconds
2019/8/20 下午3:13:502019-08-20T15:13:50+0800 [twisted.python.log#info] "192.168.95.55" - - [20/Aug/2019:07:13:50 +0000] "GET /daemonstatus.json HTTP/1.1" 200 89 "-" "python-requests/2.22.0"
2019/8/20 下午3:13:532019-08-20T15:13:53+0800 [twisted.python.log#info] "192.168.95.55" - - [20/Aug/2019:07:13:53 +0000] "GET /daemonstatus.json HTTP/1.1" 200 89 "-" "python-requests/2.22.0"
2019/8/20 下午3:13:54[2019-08-20 15:13:54,198] WARNING  in logparser.logparser: Saved to http://127.0.0.1:6800/logs/stats.json
2019/8/20 下午3:13:54[2019-08-20 15:13:54,198] WARNING  in logparser.logparser: Sleep 10 seconds
2019/8/20 下午3:14:002019-08-20T15:14:00+0800 [twisted.python.log#info] "192.168.95.55" - - [20/Aug/2019:07:14:00 +0000] "GET /daemonstatus.json HTTP/1.1" 200 89 "-" "python-requests/2.22.0"
2019/8/20 下午3:14:042019-08-20T15:14:04+0800 [twisted.python.log#info] "192.168.95.55" - - [20/Aug/2019:07:14:03 +0000] "GET /daemonstatus.json HTTP/1.1" 200 89 "-" "python-requests/2.22.0"
2019/8/20 下午3:14:04[2019-08-20 15:14:04,210] WARNING  in logparser.logparser: Saved to http://127.0.0.1:6800/logs/stats.json
2019/8/20 下午3:14:04[2019-08-20 15:14:04,210] WARNING  in logparser.logparser: Sleep 10 seconds

My question： Still show information “'pip install -U logparser' on host '192.168.95.60:6800' and run command 'logparser' to update LogParser to v0.8.1” WX20190820-144308@2x.png

my8100 commented 5 years ago 
pip install --upgrade scrapydweb
luzihang123 commented 5 years ago

Perfect~ image.png

my dockerhub scrapydweb，update to v1.4 https://cloud.docker.com/u/chinaclark1203/repository/docker/chinaclark1203/scrapydweb

my8100 commented 5 years ago

Is it public?

https://cloud.docker.com/u/chinaclark1203/repository/docker/chinaclark1203/scrapydweb image

luzihang123 commented 5 years ago

https://hub.docker.com/r/chinaclark1203/scrapydweb

my8100 commented 5 years ago

https://hub.docker.com/r/chinaclark1203/scrapydweb

Good job.