search

RitikMody / Coursearch

A one stop solution to navigate the endless sea of online courses.
https://coursearch.herokuapp.com/
11 stars 6 forks source link

Integrate Udacity web scraper with Flask. #2

Closed rubyruins closed 3 years ago

rubyruins commented 4 years ago

Description

Coursearch is a web service that crawls MOOC platforms to scrape course data (title, level, taught by, reviews etc.) It uses Scrapy for Coursera and Udemy platforms and Selenium for Pluralsight. We are in the process of expanding the scope of the project.

Issue

Scraping the data using Selenium for Udacity gives the correct output on the command line, but does not integrate with Flask without causing errors.

To reproduce the behavior locally, run the Udacity spider on its own as a single python file. You will notice that it works without errors. Next, try running the Flask app by including the spider in it and notice that the code breaks.

Your job is to find a solution to this so that the entire application runs smoothly as expected along with the Udacity spider in the Flask app. For reference, you can check out the pluralsight.py file to see the code structure and how it works with Flask.

You can ask for resources and we will try to help you as best as we can. Please ask to be assigned before you start working on the issue.

rpdswtk commented 4 years ago

I would love to work on this issue. Please assign me.

RitikMody commented 4 years ago

Go for it

rubyruins commented 4 years ago

Hi @rpdswtk, any updates?

rpdswtk commented 4 years ago

Hi @rubyruins, I'm experimenting with a new approach. Expect my PR soon (probably tomorrow).

rubyruins commented 4 years ago

Sounds great. Looking forward to it!