Python 3 support

[merito]

Hi,

scrapy 1.1.0 with Python 3 support is almost done. Are there any plans to add support for Python 3 for scrapyd?

Best regards

[lucaspottersky]

Also, the docs says it requires "Python 2.6 or above", which is not the case. I think it would be a good idea to update it in the meantime.

[xhijack]

I got this when running scrapyd

version_info = tuple(__version__.split('.')[:3])
TypeError: 'str' does not support the buffer interface

[Digenis]

@lucaspottersky, scrapyd-1.1 wasn't meant to support python 2.6 but leftover trove classifiers and documentation made it look like it does which is enough to assume that somebody used it in py2.6 and provide support. scrapyd-1.2 will definitely not support it.

[dionysio]

I tried running the python3-wip branch and it starts, but I cannot get the web console to work. The terminal is showing my requests, but responds with 404 to any url. I also tried setting debug = on, but it does not show any more info... got any tips?

[redapple]

@dionysio , I'm looking into it. As far as I can see, it's related to bytes vs. strings in twisted web Requests and Resources

[redapple]

@dionysio , I haven't tested extensively, but I've submitted a PR to make scrapyd more usable in Python 3 (i.e. not throwing 404 for everything): https://github.com/scrapy/scrapyd/pull/172 I'd be happy to have your feedback.

[dionysio]

@redapple Thank you! The updated version has been working great so far. I will let you know if some issue comes up.

[631068264]

Traceback (most recent call last):
  File "/Users/wyx/project/scrap/.env/bin/scrapyd", line 7, in <module>
    from scrapyd.scripts.scrapyd_run import main
  File "/Users/wyx/project/scrap/.env/lib/python3.5/site-packages/scrapyd/__init__.py", line 3, in <module>
    version_info = tuple(__version__.split('.')[:3])
TypeError: a bytes-like object is required, not 'str'

I hope scrapyd can install on Python3 beacause Scrapy support for Python3.

[Digenis]

Hello, scrapyd-1.2.0a1 is released today. This experimental release brings full python3 support. See the full announcement https://groups.google.com/forum/#!topic/scrapy-users/6xSS98svwe4

You can try the pre-release with pip install scrapyd'==1.2.0a1' Don't hesitate to report in this issue tracker any bugs you may discover (even if they are not related to python3). Your feedback is very important to the development of scrapyd.

I believe that it's a matter of weeks to get to a stable new release.

[xcf007]

@xhijack ，I got Same problem。Have you solved it？

[Digenis]

@xcf007, @xhijack What is your python version and scrapyd version? Also, please provide a full stacktrace.

[lt82654993]

As you say, I still can't get the version 1.2.0a1 of scrapyd. The pip can't find it. How can i get it? Thanks you response.

[Digenis]

@lt82654993, pre-releases are not installed automatically, see: https://www.python.org/dev/peps/pep-0440/#handling-of-pre-releases Execute the command from https://github.com/scrapy/scrapyd/issues/143#issuecomment-273893765 exactly as shown. See pip's documentation https://pip.pypa.io/en/stable/

[kedmenec]

The command in #143 didnt work for me, the following did: pip install scrapyd==1.2.0a1 probably obvious, but thought I would point it out. Thanks for your work.

[redapple]

An alternative is to use pip install --pre scrapyd

[Digenis]

@merito, @lucaspottersky, @xhijack, @dionysio, @631068264, @xcf007, @lt82654993, @kedmenec, @yilmazalican, @manukankani, @seozed,

scrapyd-1.2.0 (stable) with python3 support is just released so I'm closing this issue.

Try it out with pip install scrapyd'==1.2.*' and don't hesitate to report any bugs here or in the mailing list.

[blakeShalom]

Hi,

I was deploying a spider with the following python3 syntax:

https://www.python.org/dev/peps/pep-3102/

def yield_next_request(self, *, step=None, response=None):

And received this error code when deploying the project to scrapyd (used scrapyd-deploy):

def yield_next_request(self, *, step=None, response=None):\
                                  ^SyntaxError: invalid syntax\\n

Solution for me was to delete the keyword-only arguments however thought I would raise and open the issue again.

[Digenis]

Hi @blakeShalom, My guess is that your setup of scrapyd runs on python2. Can you verify it?

[blakeShalom]

Thanks @Digenis, how can I check the python version of my setup? I was running Scrapyd locally using the scrapyd command

[Digenis]

Hi @blakeShalom, Here's the easiest way: Among the first lines of scrapyd's log (or output if not running in daemon mode) you should see something similar to 2017-08-30T12:01:57+0300 [-] Loading /var/lib/scrapy/scrapyd-env/lib/python2.7/site-packages/scrapyd/txapp.py You can figure out what python it uses by looking at the path.

If you are having problems running scrapyd, don't forget to report them in an appropriate issue in this github repo.