Application installation plan

[bartaelterman]

Where will we install our application?

Installation of the scrapers

Option 1: internal Windows server

• Advantages: free
• Disadvantages: no experience with the platform. Will have to investigate how to install Scrapy and create jobs that run every day.

  Option 2: hosted server

• Advantages: more experienced with the platform. Can be installed side by side with the webapp so that the scrapers can make use of the django models.
• Disadvantages: possibly not free (although OpenShift could be a possibility). Need to implement a REST interface through which the Matlab user can fetch scraped data. This requires authentication.

  Installation of the webapp

  Option 1: the webserver where the client hosts its homepage

This is probably not an option since it's a one.com server and I cannot find any documentation on how to install django apps on one.com.

Option 2: a preferably free hosting platform

Currently our webapp is hosted on Heroku. We can leave it this way, but if we need to host the scrapers as well, it would be best to host them together and Heroku could get quite expensive then.

Questions

@niconoe and @peterdesmet could you share your ideas on this?

• What's your opinion on installing the scrapers on the Windows server of the lab?
• If we could find a free hosted solution, what would your preferred set up be?

[bartaelterman]

I would say if we can provide a free hosting solution on a platform like heroku or openshift, I see no reason to start messing on the Windows server.

[niconoe]

Just a few quick toughts:

• About the Windows server, I'd say the most important thing to assess feasibility is probably the compatibility of our dependencies (Scrapy, but maybe there are others). If it was for me, I'd probably try to avoid this option. In addition to the problem listed we also have to ensure there's no connectivity issues (univerisity firewalls, ...) between this server and the outside. Difficult to know without testing.
• About the "free hosting platform": I guess the "free" comes from some rule that allows one-shot payments, but not recurrent ones ? I've seen that a few times, especially in universities, but I think this doesn't make really sense. Infrastructures have a cost, and in my experience the free plans are always quite limited (or they lost the free attribute at some point later...). There are a lot of cheap but high-quality options (Digital Ocean for example - 5/10$ per month), I'd personally try to encourage them to bend the rules if possible instead of focusing on "free". And paying us to fight against the limitations of "free plans" is not exactly free, neither :)

[bartaelterman]

Thanks for you thoughts @niconoe. I'll spend a limited amount of time investigating the Windows environment for hosting the scrapers. I'll see how easy it is to install Scrapy, and whether there are no firewalls blocking our scrapers. I'll report on that later.

[bartaelterman]

Final plan

• Web app will run on a cloud service
• Scrapers will run on the same cloud service and write articles directly to the web apps database
• Articles will be classified using custom Python code that takes an articles text and a set of words with weights as input so there is no need to do the classification in Matlab anymore.