sujitmandal / scrape-search-engine

Search anything on the different Search Engine's it will collect all the links.
MIT License
14 stars 5 forks source link

Ecosia broken? #3

Closed csecht closed 2 years ago

csecht commented 2 years ago

I had success with the Google and Duckduckgo modules, but the Ecosia module returns "Service Unavailable":

from ScrapeSearchEngine.SearchEngine import Ecosia

userAgent = 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:92.0) Gecko/20100101 Firefox/92.0'
search = 'cats'  #Enter Anything for Search

ecosia = Ecosia(search, userAgent)

print(ecosia)  # => (['HTTP Status : Service Unavailable'], ['HTTP Status : Service Unavailable'])

Do you have any suggestions to get the Ecosia engine working? I see that when a search works, 28 results are generally returned. Is there a parameter that can increase the number of results?

sujitmandal commented 2 years ago

@csecht i think they (Ecosia) change their source code due to that this is happening, i need to check it.

now you only get first page link (date), but if we want to increase the number of results then we need to try go for next page, for that i need to find out whether it is possible or not.

sujitmandal commented 2 years ago

{ "Page 1": [ "https://www.geeksforgeeks.org/python-programming-language/", "https://www.learnpython.org/", "https://www.coursera.org/learn/python", "https://www.coursera.org/specializations/python", "https://www.javatpoint.com/python-tutorial", "https://www.linkedin.com/learning/topics/python", "https://www.kaggle.com/learn/python", "https://www.udemy.com/topic/python/", "https://www.datacamp.com/courses/intro-to-python-for-data-science", "https://realpython.com/" ], "Page 2": [ "https://pythontutor.com/visualize.html", "https://www.jetbrains.com/pycharm/", "https://developers.google.com/edu/python", "https://pythoninstitute.org/what-is-python/", "https://github.com/python", "https://code.visualstudio.com/docs/languages/python", "https://www.netacad.com/courses/programming/pcap-programming-essentials-python", "https://opensource.com/resources/python", "https://cloud.google.com/appengine/docs/python", "https://swcarpentry.github.io/python-novice-inflammation/" ], "Page 3": [ "https://www.infoworld.com/article/3204016/what-is-python-powerful-intuitive-programming.html", "https://www.reddit.com/r/Python/", "https://aws.amazon.com/developer/language/python/", "https://www.toptal.com/python", "https://www.freecodecamp.org/news/tag/python/", "https://www.edx.org/learn/python", "https://www.anaconda.com/products/distribution", "https://www.pluralsight.com/paths/core-python", "https://www.py4e.com/", "https://www.hackerrank.com/domains/python" ], "Page 4": [ "https://docs.python-guide.org/intro/learning/", "https://docs.python-guide.org/", "https://pythonbasics.org/", "https://micropython.org/", "https://codingbat.com/python", "https://python.swaroopch.com/", "https://www.packtpub.com/in/tech/python", "https://scikit-learn.org/", "https://en.wikipedia.org/wiki/Python_(genus)", "https://www.djangoproject.com/" ], "Page 5": [ "https://jobtensor.com/Tutorial/Python/en/Introduction", "https://azure.microsoft.com/en-us/develop/python/", "https://nostarch.com/catalog/python", "https://www.udacity.com/course/introduction-to-python--ud1110", "https://open.sap.com/courses/python1", "https://python-course.eu/", "https://en.wikibooks.org/wiki/Python_Programming", "https://www.digitalocean.com/community/tutorial_series/how-to-code-in-python-3", "https://www.oreilly.com/library/view/learning-python-5th/9781449355722/", "https://py.processing.org/" ], "Page 6": [ "https://jupyter.org/", "https://pythonprogramming.net/", "https://www.intel.com/content/www/us/en/developer/tools/oneapi/distribution-for-python.html", "https://www.codingninjas.com/courses/python-data-structures-and-algorithms", "https://www.techtarget.com/whatis/definition/Python", "https://developer.oracle.com/python/what-is-python/", "https://www.fullstackpython.com/", "https://hourofpython.com/", "https://www.simplilearn.com/why-learn-python-a-guide-to-unlock-your-python-career-article", "https://python-graph-gallery.com/" ], "Page 7": [ "https://docs.python-requests.org/en/master/", "https://developers.arcgis.com/python/", "https://roadmap.sh/python", "https://www.nltk.org/", "https://www.guru99.com/python-tutorials.html", "https://dev.mysql.com/doc/connector-python/en/connector-python-examples.html", "https://www.interviewbit.com/python-interview-questions/", "https://users.astro.ufl.edu/~warner/prog/python.html", "https://www.w3resource.com/python-exercises/", "https://www.futurelearn.com/info/blog/what-is-python-used-for" ], "Page 8": [ "https://www.futurelearn.com/courses/programming-for-everybody-python", "https://www.facebook.com/pythonlang/", "https://www.mygreatlearning.com/academy/learn-for-free/courses/python-fundamentals-for-beginners", "https://scipy.org/", "https://data-flair.training/blogs/features-of-python/", "https://ocw.mit.edu/courses/6-0001-introduction-to-computer-science-and-programming-in-python-fall-2016/", "https://trinket.io/python", "https://thonny.org/", "https://home.pearsonvue.com/python", "https://www.daxx.com/blog/development-trends/python-developer-salary-usa" ], "Page 9": [ "https://datamites.com/blog/is-python-better-than-ruby/", "https://builtin.com/software-engineering-perspectives/python-symbol", "https://www.merixstudio.com/blog/8-skills-you-need-be-good-python-developer/", "https://towardsdatascience.com/this-little-python-exercise-taught-me-a-lot-65a60e1100bf", "https://www.analyticsinsight.net/top-10-deep-learning-python-courses-to-take-up-in-2022/", "https://www.dataquest.io/blog/learn-python-the-right-way/", "https://medium.com/tag/python?source=topics_v2----------84----------python----------f3a6e23f_2e4e_4a52_ab0e_4c722e8bfbe4-------19---", "https://www.kdnuggets.com/2022/04/python-libraries-data-scientists-know-2022.html", "https://sneakernews.com/2022/04/13/air-jordan-1-mid-white-vivid-green-python-dv2959-113/" ], "Page 10": [ "https://www.geeksforgeeks.org/python-programming-language/", "https://www.coursera.org/learn/python", "https://www.coursera.org/specializations/python", "https://www.learnpython.org/", "https://www.javatpoint.com/python-tutorial", "https://www.linkedin.com/learning/topics/python", "https://www.kaggle.com/learn/python", "https://www.udemy.com/topic/python/", "https://www.datacamp.com/courses/intro-to-python-for-data-science", "https://realpython.com/" ] }

@csecht this is the output of 10 number's of pages results, this is you asked for right....?

csecht commented 2 years ago

Thanks, but I'm still running into 'HTTP Status : Service Unavailable' issues. The problem must be at my end. I'll keep working on it.