Unable to post to ssl endpoint with custom CA

[LordBrain]

We have a elastic stood up using a internal CA. This plugin does not like that. Is there a way where we can pass in the cert, or have it ignore it?

[jayzeng]

@blee1170 so you have custom cert any key files? I can expose a few more ssl related settings like what's provided in example (https://elasticsearch-py.readthedocs.io/en/master/#ssl-and-authentication), will these be sufficient?

[LordBrain]

Something like this:

es = Elasticsearch(
    ['localhost', 'otherhost'],
    http_auth=('user', 'secret'),
    port=443,
    use_ssl=True,
    ca_certs='/path/to/cacert.pem',
    client_cert='/path/to/client_cert.pem',
    client_key='/path/to/client_key.pem',
)

Would help a ton.

[jayzeng]

ok, I will add these over the weekend.

[LordBrain]

That would be fantastic.

[jayzeng]

@blee1170 I checked in an open PR (https://github.com/knockrentals/scrapy-elasticsearch/pull/68), since I don't have a testing environment to test custom CA files, do you mind pulling this pr to your local environment to give it a test before I merge?

You will need to clone this repo, check out the custom_ca branch and issue a python setup.py install. It accepts a new setting ELASTICSEARCH_CA (dictionary, see https://github.com/knockrentals/scrapy-elasticsearch/pull/68/files#diff-88b99bb28683bd5b7e3a204826ead112R49) for you to pass in ca files.

Let me know if it works and your thoughts.

[LordBrain]

@jayzeng We did some testing and it seems to work for us!

This is the config we used:

ELASTICSEARCH_CA = { 'CA_CERT': os.environ['CA_BUNDLE_PATH'], 'CLIENT_KEY': None, 'CLIENT_CERT': None }

We are using a bundle, so we need not need the key and cert. Still had to set those values to 'None'. But everything is working well. Let me know when its released.

[jayzeng]

@blee1170 thanks for testing and glad it works, I just cut the latest release (https://pypi.python.org/pypi/ScrapyElasticSearch/0.9.0), cheers!