search

abusesa / abusehelper

A framework for receiving and redistributing abuse feeds
MIT License
118 stars 18 forks source link

abusehelper.core.utils.fetch_url: SNI support #90

Closed ics closed 7 years ago

ics commented 7 years ago

The custom HTTPS handler introduced at b7e5f72 doesn't support SNI. Fetching a resource from a host enforcing SNI yield the following error:

2017-03-28 00:40:31Z alienvault[32602] INFO Poll skipped: Download failed: 'https://reputation.alienvault.com/reputation.data' (<urlopen error [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:661)>)

Using Python's (2.7.9+) urllib2.urlopen to fetch the same resource works as expected.

larihuttunen commented 7 years ago

Thanks for reporting this to us. We were able to reproduce the problem and will investigate it further in due time.

mseppanen commented 7 years ago

Proposed fix in #93.

ics commented 7 years ago

This fixes the SNI issue. However, certificate validation fails with: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)> I have tested on Python 2.7.13, OpenSSL 1.0.2l.

Calling urllib2.urlopen in an idiokit thread works as expected. Let me know if you need more details.

mseppanen commented 7 years ago

Which OS (and distribution if Linux) you have and can the fetch_url() verify any certificate?

There might be problem with finding system's CA bundle. Currently it is done using code in idiokit, as older Python versions do not provide generic way to do that. However I could modify the patch to use Python's builtin functionality for that when using modern Python (2.7.9+).

mseppanen commented 7 years ago

I modified #93 a bit. Could you check if it now works better on your environment?

ics commented 7 years ago

Running FreeBSD 11.0-RELEASE-p8. fetch_url isn't able to verify any certificate indeed. I think you're right about finding the CA certs because the modified #93 works on macOS.

mseppanen commented 7 years ago

That explains. The code in idiokit has support only for bunch of Linux distributions and OpenBSD: https://github.com/abusesa/idiokit/blob/master/idiokit/ssl.py#L110

ics commented 7 years ago

Is that in the idiokit ssl module? I tried elif platform.system().lower() in ["openbsd", "freebsd"]: but it doesn't seem to work. Certs path exists.

ics commented 7 years ago

Modified #93 works on Debian and derivates.

mseppanen commented 7 years ago

That should work, if the root CAs are there and in correct format for OpenSSL.

ics commented 7 years ago

It works on FreeBSD too (PEBKAC). I'll submit a PR to idiokit.