scrapy / scrapyd

A service daemon to run Scrapy spiders
https://scrapyd.readthedocs.io/en/stable/
BSD 3-Clause "New" or "Revised" License
2.92k stars 569 forks source link

crawl: error: Invalid -s value, use -s NAME=VALUE #473

Closed saavedra29 closed 1 year ago

saavedra29 commented 1 year ago

When scrapyd runs a spider with scrapy==2.8.0 no log files are created inside the logs folder (logs_dir). Using scrapy==2.7.1 works perfectly.

imaxwen commented 1 year ago

+1

jpmckinney commented 1 year ago

Please upgrade to Scrapyd 1.4.0.

imaxwen commented 1 year ago

Using scrapy==2.8.0 and scrapyd==1.4.0 still got error , and logs not created in the logs_dir.

scrapyd-01  | 2023-02-10T12:07:39+0800 [-] Process started:  project='my-spiders' spider='test-spider' job='74738e4ea8f811ed9b1502420a110703' pid=13 log='logs/my-spiders/test-spider/74738e4ea8f811ed9b1502420a110703.log' items='file:///scrapyd/items/my-spiders/test-spider/74738e4ea8f811ed9b1502420a110703.jl'
scrapyd-01  | 2023-02-10T12:07:40+0800 [Launcher,13/stderr] Usage
scrapyd-01  |   =====
scrapyd-01  |     scrapy crawl [options] <spider>
scrapyd-01  |   crawl: error: Invalid -s value, use -s NAME=VALUE
imaxwen commented 1 year ago

@jpmckinney Please have a look, consider reopen this issue?

jpmckinney commented 1 year ago

crawl: error: Invalid -s value, use -s NAME=VALUE

That message is unusual. Can you edit your copy of scrapy/commands/__init__.py, and change the line raise UsageError("Invalid -s value, use -s NAME=VALUE", print_help=False), to raise UsageError(repr(opts.set)) ?

jpmckinney commented 1 year ago

@saavedra29 Please provide your scrapyd log (like in https://github.com/scrapy/scrapyd/issues/473#issuecomment-1425150676). Otherwise, this is impossible to debug. It was tested and it works in normal conditions.

Edit: Nevermind - if downgrading to Scrapy 2.7.1 fixed your issue, then you must not be using Scrapyd 1.4.0 yet.

jpmckinney commented 1 year ago

@imaxwen Alternatively, install this testing branch:

pip install -e git+https://github.com/scrapy/scrapyd.git@launcher-log#egg=scrapyd
jpmckinney commented 1 year ago

Aha, it is because the FEEDS argument wasn't properly encoded. 1.4.1 is released.

imaxwen commented 1 year ago

thanks Very efficient 😆