Better support for HTTPS/certs (the curl_option_ settings can be used, but are kind of clunky)

[svanoort]

Add options to support HTTPS / certs on tests.

[ksreddy543]

What is the tentative release date for this enhancement?

[svanoort]

@ksreddy543 No formal release date -- this is only a day or two of coding, testing is the challenge.
If you're curious, the fork in the light blue platform has version of this working (with a legacy version of PyRestTest).

This is something a community member could take on easily and forward port + extend for a full feature in the current version, if you are interested...

[netjunki]

@svanoort I'd be willing to take a stab at doing the forward porting. I didn't realize there was even an implementation out there already. I need this for something I'm working on this week. I've got my own implementation which is trying to replicate the -E <cert>:<pass> syntax in curl but that doesn't seem to be working quite right... I keep getting SSL authentication errors. ERROR:Test Failure, failure type: Curl Exception, Reason: Curl Exception: (35, 'SSL authentication failed') but maybe the problem is that I'm not setting the CACERT. Will investigate.

Totally agreed that the testing part of this is the hard part. :-)

[ksreddy543]

@svanoort : I have taken lightblue platform added following lines in resttest.py. We have our own certs basis of openssl. our certs are placed under /mpsconfig/ssl/. Light Blue working fine with github only.

curl.setopt(pycurl.SSL_VERIFYPEER, 1)
curl.setopt(pycurl.SSL_VERIFYHOST, 2)
curl.setopt(pycurl.CAINFO, "/mpsconfig/ssl/sdx_cert")

Followed I am getting following error: Any clues how to resolve it? (77, 'error setting certificate verify locations:\n CAfile: /mpsconfig/ssl/sdx_cert\n CApath: /mpsconfig/ssl/\n')

[ksreddy543]

@svanoort Also tried with below options

curl.setopt(pycurl.SSL_VERIFYPEER, 1)
curl.setopt(pycurl.SSL_VERIFYHOST, 2)
curl.setopt(pycurl.SSLCERT, "/mpsconfig/ssl/sdx_cert")
curl.setopt(pycurl.SSLKEY, "/mpsconfig/ssl/sdx_key")

Getting below error: (58, 'unable to use client certificate (no key found or wrong pass phrase?)')

[netjunki]

If your cert of key uses a password you've need these respectively... curl.setopt(pycurl.SSLCERTPASSWD, test_config.cert_password) curl.setopt(pycurl.SSLKEYPASSWD, test_config.cert_password) If you're testing out... just swap out test_config.cert_password with a string (I have a modified branch locally which already pulls this data from a test yaml).

[ksreddy543]

@svanoort In "light blue platform" , I am able to run https cases without certificate details by adding authentication details in the resttest.py .but, it doesnt have response validator support. I would like to get this https version in the latest pyresttest. Would it be possible to give? If not please guide me to add https support to my private branch pyresttest. Thanks in advance..

[svanoort]

@netjunki Thank you for taking this on! By all means, go at it, and I'll make review/merge/release of this feature a priority since it's in high demand.

@ksreddy543 Light blue platform is a fork based off the legacy version of PyRestTest and has its own limited form of validators, but it is not compatible. It looks like netjunki is working on forward porting this feature though, so that should be available soon. I agree with netjunki's answer on the password.

[netjunki]

A quick update on this. So I seem to be running into some odd issue with pycurl and connecting with a password protected .p12. @svanoort Maybe you've encountered this before. But I'm stumped... and it seems like making this stuff work across the different backend variants (let alone testing them) is going to be a major challenge.

So we have the following curl call: curl -svk -E ~/clientcert.p12:XXXXXX 'https://example.com'

And this works and everything is happy (urls and names changed since I've been testing this against company servers).

If I convert this to some C code using the --libcurl out.c option I can then compile and run that code and everything seems to work okay as well.

But when I perform the operation inside pycurl... we get that 'SSL authentication failed' I mentioned a few posts back.

All the version information seems to be the same. $ curl --version curl 7.37.1 (x86_64-apple-darwin14.0) libcurl/7.37.1 SecureTransport zlib/1.2.5 Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smtp smtps telnet tftp Features: AsynchDNS GSS-Negotiate IPv6 Largefile NTLM NTLM_WB SSL libz $ gcc out.c -lcurl $ ./a.out libcurl/7.37.1 SecureTransport zlib/1.2.5 (added a call to out.c which prints the curl version)

$ python -c "import pycurl; print pycurl.version" PycURL/7.19.5.1 libcurl/7.37.1 SecureTransport zlib/1.2.5 Trying to throw this on a debian jessie box I have at home I noticed that pycurl there is built against gnutls... and there's support for pycurl to be built against openssl and one other library which I'm not remembering the name of at the moment.

If anyone's got some bright idea about what the problem is I could use the insight. :-)

Here's the boiled down version of just the pycurl code which I've been using to experiment:

import pycurl
print pycurl.version
pycurl_connect = pycurl.Curl()
pycurl_connect.setopt(pycurl.URL, "https://example.com")
pycurl_connect.setopt(pycurl.VERBOSE, True)
pycurl_connect.setopt(pycurl.NOPROGRESS, True)
pycurl_connect.setopt(pycurl.USERAGENT, "curl/7.37.1")
pycurl_connect.setopt(pycurl.MAXREDIRS, 50)
pycurl_connect.setopt(pycurl.SSLCERT,"clientcert.p12")
pycurl_connect.setopt(pycurl.SSLKEYPASSWD,"XXXXXX")
pycurl_connect.setopt(pycurl.SSL_VERIFYPEER,0)
pycurl_connect.setopt(pycurl.SSL_VERIFYHOST,0)
pycurl_connect.perform()

[svanoort]

@netjunki I am still trying to figure this one out myself (as someone not expert on working with certs/SSL configs, it's a bit slow). I do know that there are some subtle differences in curl vs. libcurl in pycurl: I hit issues with handling of request size and continuation with Django, where it worked in curl but not the PyCurl and I had to explicitly set an option to override it.

General rule of thumb is that at least one pycurl option exists to override a specific handling.

If it's any help, the included Django test app offers a testing environment you can use to validate behavior with at least one webserver, and it runs from the functionaltest.py tests.

I do see that in the libcurl docs there are many reasons for that particular error:

CURLE_SSL_CONNECT_ERROR (35)

A problem occurred somewhere in the SSL/TLS handshake. You really want the error buffer and read the message there as it pinpoints the problem slightly more. Could be certificates (file formats, paths, permissions), passwords, and others.

One useful troubleshooting tip: this will print out the fill HTTP traffic back and forth (everything!) and has been very useful for me in debugging HTTP configuration issues!

pycurl_connect.setopt(pycurl.VERBOSE,True)

[svanoort]

@netjunki @ksreddy543 This is to some extent covered by the Custom curl options now present (assuming pycurl supports the SSL algorithms).

It appears that it may be fixed by explicitly setting the SSL ciphers to use:

curl_option_SSL_CIPHER_LIST:  'rsa_rc4_128_sha'   # Or similar

[netjunki]

@svanoort haha. Actually I just got this working on Saturday. Apparently between issues with the server I was talking to and out of date libraries on my machine (just got a new laptop) this seems to all be working now. Still need to figure out a sane way to test it. :-)

That custom curl options seems like a much more flexible approach than the one I was taking. Let me play around with it and see how it works.

[svanoort]

@netjunki I'm glad you figured out what was up, and like the new options! :-) You may want to check out the other goodies in 1.6.0 release (biggest new addition besides generic opts is the types validator for content): https://github.com/svanoort/pyresttest/blob/master/CHANGELOG.md

Also, if you're curious, I'm organizing next steps into milestones. Next up is Python 3 support and a major refactoring to how configs are handled (to make new options easy to add). Feedback is welcome as far as the roadmap. This one had a lot of python3 prep and CI/docker infrastructure, and next one promises to be even meatier.

As far as testing goes, the Django-tastypie testapp could be modified with one client-cert protected SSL endpoint, which this could be tested as against.... but for now I'm using validation parsing side that options are legitimate, and using mocks to test that they're passed to pycurl correctly.

Also, based on this: http://stackoverflow.com/questions/1942719/pycurl-tls-handshake-error there's a way to rebuild pycurl to use OpenSSL if the GnuTLS lib has issues:

export PYCURL_SSL_LIBRARY=openssl
pip install --upgrade pycurl

[svanoort]

@netjunki If your new options are good to go, would love to roll them in with the next release (since that's more elegant than the rather ugly curl_option approach); your effort certainly isn't going to waste!

There's some work to re-do parsing (for https://github.com/svanoort/pyresttest/issues/75) that I've begun on branch feature-refactor-parsing, which might facilitate making HTTPS options exposed at the test, config, or command-line level more easily.

[j796160836]

Hi, I use this test framework and encountered SSL certificate problems. The target website which is provide by Godaddy and it runs normally in iOS client and Mac Chrome but it fails in pyresttest.

I found the solution here https://stackoverflow.com/questions/16192832/pycurl-https-error-unable-to-get-local-issuer-certificate Which use certifi (root certificate exports from firefox) maybe it's helpful.