To run ScrapydWeb in HTTPS mode

[ohenrik]

How do i enable https? I have a certificate file from let's encrypt and would like to start the server using https.

Is this possible at this time? It should be possible since using basic auth without HTTPS is dangerous.

[ohenrik]

Maybe adding config paths for the certificate files and optionally adding them here: https://github.com/my8100/scrapydweb/blob/master/scrapydweb/run.py#L92

Would be enough?

You might also need to require pyopenssl (pip install pyopenssl)

app.run with ssl_context:

app.run(ssl_context=('cert.pem', 'key.pem'))

[my8100]

This works for me, and no need to install pyopenssl. Could you try and tell me your result?

context = ('/path/to/the/cert.pem', '/path/to/the/cert.key')
app.run(host=app.config['SCRAPYDWEB_BIND'], port=app.config['SCRAPYDWEB_PORT'], ssl_context=context)

[ohenrik]

It seems to work :)

[my8100]

What's the file type of your certificate? (Mine is '.pem')

[my8100]

Fixed in v1.1.0: Now you can set up ENABLE_HTTPS, CERTIFICATE_FILEPATH, and PRIVATEKEY_FILEPATH to enable HTTPS mode.

[jwjhdev]

Just wanted to point out to anyone having issues with SSL that scrapydweb uses assert os.path.isfile() to check the certificate and key files which only returns true if the path exists and is a file and you have (at least) read permissions.

Popular certificate aggregators like LetsEncrypt chmod the certificate as root only read/write.

That means you will need to run scrapydweb as root in order for the certificate to parse.

[my8100]

The below assert line could be commented out. https://github.com/my8100/scrapydweb/blob/7a3b81dba2cba4279c9465064a693bb277ac20e9/scrapydweb/utils/check_app_config.py#L84-L91

[jwjhdev]

Thanks @my8100 👍

I can't think of a clean way to fix this issue other than acknowledging the file exists manually and commenting out that line as you say, would you agree?

• It should rightly fail without permission to read the cert/key, the assert os.path.isfile() is the correct method.
• Non-root users shouldn't change the permission of the cert/key as it is set to root as a layer of security
• Installing scrapyd, logparser and scrapydweb shouldn't be installed as root as a layer of security
• Running scrapydweb as a root user should also be avoided as a layer of security

Kind of leaves you in no man's land.

Thanks for a great tool by the way @my8100 and hello from Australia!

[my8100]

@jwjhdev As the below comment said, HTTPS mode was not fully tested. Thank you so much for the feedback. https://github.com/my8100/scrapydweb/blob/7a3b81dba2cba4279c9465064a693bb277ac20e9/scrapydweb/default_settings.py#L82-L92