WIP: Add Docker support

[MrDiggles2]

Hey all,

Like I mentioned in the Slack channel, I took a stab at making a development-focused set of Docker.

I've updated the development docs to reflect the new setup steps. But if you're in a hurry, you can test the setup with:

# To run tests
docker-compose run city-scrapers pytest
# To get help text of scrapy
docker-compose run city-scrapers scrapy -h

Explanations on some things:

1. A fully built image was pushed up to the Docker registry. Developers will not have to build locally, just download.
2. I locked the requirements.txt and versioned the image. That way, dependencies and code will be in lock-step and developers don't have to worry about managing their packages.
3. The VSCode docs on IDE configuration was updated to be compatible with Docker.
4. I included a .bashrc in the workspace. It's a standard one I use everywhere; we can remove if it you'd prefer not to have it in or we can add the ability to extend it with gitignored files.

This is very much still a WIP; let me know if you have any suggestions/questions.

[pjsier]

Thanks for putting this together! People have mentioned this a few times so it's good to see what a potential development environment could look like. I have a few notes from looking at it initially:

• If we do go this route it would be in addition to the usual installation and not replacing it
• I'm not sure we need docker-compose here, since we'd only be working with a single container
• We aren't storing anything on disk other than if people decide to send the output to a file, and I'm not sure that use case warrants the additional complexity of volumes
• pipenv is useful outside of pinning requirements, and I think that's something we would want to keep regardless. requirements.txt was left in to make installation easier for people who wanted it, but pipenv is pretty quickly becoming the standard for dependency management in Python
• The .bashrc file is useful information, but I think don't think we would want that in the repo
• If we do have a Docker image, we would use one based in this GitHub organization, and we would need it to build and deploy on every push

I don't think this is a direction we want to go in though. In my experience, Docker simplifies setup for developers working on multiple projects with complicated dependencies, but is a lot more opaque for beginners who are just getting an environment set up (maybe for the first time). We also haven't had many issues with environment setup in the past few months since removing some packages that didn't work well with Windows like invoke. I'm also worried that rather than simplify things it could create another set of problems to troubleshoot.

The docs could use an update, especially for things like virtualenv-wrapper which is probably not worth supporting at this point, but the current environment setup is likely simpler than this. I wanted to be transparent about that for now, but I'll also leave this open for discussion. If anyone has thoughts on this feel free to comment here!

[ahmad380360]

شكرا لجمييع.

[MrDiggles2]

After reading through the updated documentation and your comments, I agree that pipenv does make more sense for this project. The initial setup is simpler and development environment (i.e. VSCode settings) is definitely more straightforward without having another technology in the mix.

I hadn't considered the fact that opaqueness could be a negative but it makes sense that it could mislead first-time contributors.

I'll close out this PR.

Thanks for the feedback. Definitely learned something new!

[pjsier]

Thanks for doing the research here, and glad to hear it was useful!