search

binux / pyspider

A Powerful Spider(Web Crawler) System in Python.
http://docs.pyspider.org/
Apache License 2.0
16.49k stars 3.69k forks source link

Everything done well in debug,but always fectch error in dashbord #612

Open avayayu opened 7 years ago

avayayu commented 7 years ago

frist,i am using Ubuntu 16.04LTS python 3.5.2 pyspider 0.3.8

everything does well in debug,but always fectch error when i clicked run on dashbord.

A typical error like this:

image

github

image

image

PS：another error : from projects import XXX always cause keyError:'projects'

binux commented 7 years ago

pyspider will deduplicate tasks by it's url by default, so the task submitted in on_start and index_page are the same task for pyspider. The task info (post data) may be overridden by another callback. You can use taskid API to mark them as different task.

keyError:'projects' should be a but in python3.

avayayu commented 7 years ago

Do you mean due to the same url,my task has the same taskid ,so then the fetcher discard my task?

So I wonder what's different between itag and taskid. I tried set itag before but also failed.

thanks for your reply.

avayayu commented 7 years ago

i have tried add a tag and rewrite the get_taskid function.But still the same error.

binux commented 7 years ago

Could you

  1. change the POST url to 'http://httpbin.org/post'
  2. return response.json in the callback
  3. re-run in dashboard
  4. paste the result here