Consider fixing the python version for development

[will-snavely]

We can discuss the pros and cons of this. Currently, we are running three different python versions through CI (3.6, 3.7, 3.8), and our documentation says that you can install any of those python versions. It might be worth just choosing a reasonable version of python and fixing it. It would help with Pipfile issues, for example. It would also make setup a little easier to debug, if we just enforce a single, known-to-work version of Python. Any thoughts?

[will-snavely]

Some text from pipenv's website:

General Recommendations & Version Control

Generally, keep both Pipfile and Pipfile.lock in version control.

Do not keep Pipfile.lock in version control if multiple versions of Python are being targeted.

Specify your target Python version in your Pipfile’s [requires] section. Ideally, you should only have one target
Python version, as this is a deployment tool. python_version should be in the format X.Y and python_full_version 
should be in X.Y.Z format.

From: https://pipenv-fork.readthedocs.io/en/latest/basics.html#general-recommendations-version-control

[ben-nathanson]

If this makes it easier for us to focus on shipping spiders and less on fixing dependencies I'm all for it.

I wonder what the context for this decision was originally - supporting three versions was inherited when we forked the Pittsburgh City Scrapers project. This might be a mechanism for having some flexibility e.g. if a dependency doesn't yet support Python 3.8, we can just fall back to 3.7. That said, I can't recall any problems so far with changes to Python itself.

To the points raised by the Pipenv docs, I would really like to keep Pipfile.lock because it gives us reproducible builds. I'm less attached to having multiple Python versions.

tl;dr I see no problem with just sticking to Python 3.7 and bumping up the version number regularly.

[wsnavely]

I talked about this with Bonnie a little bit after the meeting. I agree that keeping the lock file is a good idea, especially since we are running a daily cron job. We want that to have a consistent setup.

Beyond that, I think we have some options. Some ideas are:

• Keep things more or less are they are, with multiple python versions, but don't advise developers to set up their environment using the Pipfile.lock. In this scenario, the lock-file is more or less exclusively for production. Developers would setup their local Pipenv using just the Pipfile. I think this would mean they would run pipenv install opposed to pipenv sync. The reason: if a developer is using a different version of python than was used to generate the lock file, there is a chance the lock file won't work for them (though perhaps CI tries to ensure this doesn't happen by testing multiple version). The risk here is that the developer's local Pipenv differs from production in some way that leads to the cron job breaking upon pushing. But this risk can be managed by carefully configuring the Pipfile, I think. Also, CI would probably catch such things. Additionally, in this scenario, we would .gitignore the Pipfile.lock, and only update it when we need to push some update to production.

• Switch to a fixed python version. The Pipfile would include a [requires] section with an appropriate version of python (probably a 3.7.X, as Ben suggested). Developers would need to make sure they have an appropriate version of python available on their system. In this scenario, we could probably continuing using the lock file to setup local environments, though I am unsure the extent to which multiple OS's complicates the situation. Perhaps it would still make sense to use install over sync, and .gitignore the lock file to prevent accidental changes to the lock file.

• Something else. Still learning about all the options here.

[will-snavely]

Some more things I learned:

• To install dependencies locally from the Pipfile opposed to the Pipfile.lock, the command seems to be: pipenv install --skip-lock --dev. This will also not change the lockfile locally.

• Tracked files can't be gitignored (https://www.git-tower.com/learn/git/faq/ignore-tracked-files-in-git). Therefore, if we wish to track Pipfile.lock (which I think we might), we can't gitignore it, but can take other measure to protect it--e.g., installing locally from Pipfile, or using the 'git update-index' command (https://stackoverflow.com/questions/3319479/can-i-git-commit-a-file-and-ignore-its-content-changes).