search

Netflix-Skunkworks / Scumblr

Web framework that allows performing periodic syncs of data sources and performing analysis on the identified results
Apache License 2.0
2.64k stars 319 forks source link

Pull RSS feed in environment with proxy configuration #230

Closed ManoftheSea closed 6 years ago

ManoftheSea commented 6 years ago

When I try to run an RSS search, I get an error about redirection. Which makes sense, in my environment I need to use a proxy to get to the internet. I tried the wiki configuration directions, but don't see how to set http_proxy/https_proxy for Scumblr. I'm on the suggested Ubuntu 14.04.

Could I trouble you to add a note?

sbehrens commented 6 years ago

Hi @ManoftheSea ,

Is the proxy for incoming connections to Scumblr or outgoing?

ManoftheSea commented 6 years ago

Outgoing. It seems it should be as easy as setting http_proxy in environment, but I'm clearly doing something wrong.

I have curl at the command-line working. Perhaps an option to bundle, or a config file somewhere?

sbehrens commented 6 years ago

I think a global config option for proxy makes sense. We maintain config files in the https://github.com/Netflix/Scumblr/blob/master/config/initializers/scumblr.rb. You could consider adding the right proxy settings and then passing in that config option (if set or initialized) to the http request portion of the rss sync. If you get that all working feel free to send a PR and we'll take a look.

ManoftheSea commented 6 years ago

So, the end result was, as expected, user error (on my part). I had the correct environment variable, HTTP_PROXY. But I never restarted sidekiq, only scumblr, after applying the necessary changes to get the environment variable to the service.