http 503 error when downloading through a squid proxy server

[GoogleCodeExporter]

First of, thanks so much for this great script!

What steps will reproduce the problem?
1. Install a squid proxy server that requires authentication
2. On a CentOS 5.5 server, ensure you have required perl modules
3. http_proxy=http://user:pass@prxy.domain.com:port
4. https_proxy=http://user:pass@prxy.domain.com:port 
5. Run pulledpork.pl with double verbose output.

What is the expected output? What do you see instead?
I expect to have the rules downloaded and the getstore() function to return a 
200 response code. Instead, the rules fail to download and I get a 503 
response.  

What version of the product are you using? On what operating system?

Snort Server
-------------
CentOS 5.5
pulledpork-0.4.2
perl-Crypt-SSLeay-0.51-11.el5
perl-libwww-perl-5.805-1.1.1
perl-Archive-Tar-1.39.1-1.el5_5.1

Proxy Server
-------------
squid-3.0.24

Please provide any additional information below.

I'm pretty sure it's the LWP::Simple->getstore() function not behaving 
correctly with proxies and redirects. Using wget to download the rules, does 
not have the same problem. Comparing the squid logs of wget and getstore().

1) wget
TCP_MISS/302 981 GET 
http://www.snort.org/reg-rules/snortrules-snapshot-2860.tar.gz/oinkcode 
TCP_MISS/200 20453072 CONNECT s3.amazonaws.com:443 

2) getstore()
TCP_MISS/302 981 GET 
http://www.snort.org/reg-rules/snortrules-snapshot-2860.tar.gz/oinkcode
GET 
https://s3.amazonaws.com/snort.org/rules/20100915/snortrules-snapshot-2860.tar.g
z?...

The difference is that wget uses the CONNECT method to tunnel the ssl request 
through the proxy (after the redirect). getstore(), on the other hand, tries 
another GET request, which will always fail since we're being redirected a site 
that uses ssl.

Without using a proxy, pulledpork and getstore() work correctly as expected 
(since CONNECT is not required).

For the time being, I'll be modifying my pulledpork.pl to use wget instead of 
getstore. Thought I'd mention this issue so you were aware.

Again, thanks for this great script!

James

Original issue reported on code.google.com by jdaf...@gmail.com on 20 Oct 2010 at 9:26

[GoogleCodeExporter]

Sorry, I chopped off the last squid log. It should be:

1) wget
TCP_MISS/302 981 GET 
http://www.snort.org/reg-rules/snortrules-snapshot-2860.tar.gz/oinkcode 
TCP_MISS/200 20453072 CONNECT s3.amazonaws.com:443 

2) getstore()
TCP_MISS/302 981 GET 
http://www.snort.org/reg-rules/snortrules-snapshot-2860.tar.gz/oinkcode
TCP_MISS/503 2561 GET GET 
https://s3.amazonaws.com/snort.org/rules/20100915/snortrules-snapshot-2860.tar.g
z?...

Original comment by jdaf...@gmail.com on 20 Oct 2010 at 9:32

[GoogleCodeExporter]

James, if you checkout the latest version that is committed to svn, this issue 
has been addressed and will be in the upcoming release, please feel free to 
test and provide any feedback

Original comment by Cummin...@gmail.com on 20 Oct 2010 at 11:12

• Changed state: Duplicate