search

binux / pyspider

A Powerful Spider(Web Crawler) System in Python.
http://docs.pyspider.org/
Apache License 2.0
16.47k stars 3.69k forks source link

is there alternative project like this project? #976

Open socialpercon opened 3 years ago

socialpercon commented 3 years ago

Expected behavior

Actual behavior

How to reproduce

is there alternative project like this project?

i don't understand why this project no longer maintainece. i think alternative project more powerful... but i don't know...

Chaffy-0 commented 2 years ago

Scrapy

JermellB commented 2 years ago

This project isn't maintained any more because their javascript rendering capability is done by phantomjs which is no longer maintained.

Like @Chaffy-0 said, Scrapy is likely the best option if you wanted to do a spider like this.

These days, elasticsearch comes paired with one if you were doing something simple and didn't need to collect and process your own data from the wild.

Most places I've done stuff @ will use things like selenium + chrome or firefox, paired with beautiful soup for the rendered html parsing. Then you could keep track of where you'd spider with simple things like a bloom filter implemented on top of redis or something.

But yeah, Scrapy if you don't feel like getting too dirty.

milahu commented 2 years ago

some active python web scraper projects https://github.com/Gerapy/Gerapy https://github.com/howie6879/ruia

roniemartinez commented 2 years ago

Just in case people will be interested in my project 🙇 : https://github.com/roniemartinez/dude