Upgrade to python>=3.6, update requirements

[emkll]

SSLyze (used by pshht) has a hard requirement for Python >=3.6. Currently we use Python 3.5.3 for SecureTheNews. Because these dependencies are highly coupled, it will make updating them difficult as time goes by.

We are also far behind on pshtt version https://github.com/freedomofpress/securethenews/blob/master/securethenews/requirements.txt#L40 (latest as of this writing is 0.5.2)

Recently there have been several security fixes for SSL-related libraries. While they do not directly affect SecureTheNews, they will make upgrades more complex should a rapid version be required.

Let's move to Python 3.6 to ensure we can timely update dependencies, should a security vulnerability require rapid patching.

[conorsch]

In addition to updating the dependencies, we must also ensure that the CI environment dev tooling uses at least python 3.6—it's currently pinned to 3.5.2 https://github.com/freedomofpress/securethenews/blob/6f056c4b5dcd5373133db4dcfcbe3cd2201acf13/.circleci/config.yml#L13-L14 .

[eloquence]

@conorsch Could you clarify on this ticket what you see as the next step here? Should this wait on the Kubernetes transition, or is this a change the web team can help with now?

[conorsch]

Yes, first goal is get STN on in containers deployed to production, via k8s, then use that setup to upgrade the Python version inside the app container. That'll be significantly less work than upgrading the full VM OS to get a newer version of Python.

It's worth noting that Python 3.5 EOLs on 2020-09-13: https://devguide.python.org/#status-of-python-branches Even though we're using an LTS OS, we should still aim to have all sites (not just STN) upgraded to >3.5 by that time. If migration the other sites to containers by that date isn't feasible, we'll proceed with VM updates where required. Recommend final decision by 2020-07-01.

[eloquence]

Discussed today at web sync. We agreed that this is high priority for next week. The change itself, in a best case, should be pretty straightforward; we may also want to use this deploy as an opportunity to try to use a branch strategy for STN deploys, as opposed to tag pushes.

[harrislapiroff]

Just want to add one clarifying comment here: as part of this change, we would like to upgrade what packages we can to reduce the list of ignored safety vulns. Ideally we want to reduce it to zero, but if any must be left, please note which ones and why.

[eloquence]

as part of this change, we would like to upgrade what packages we can

Retitled accordingly, but please don't hesitate to split into a separate issue if that makes more sense.

[chigby]

IMO this was fixed with #251