Consolidate the two development environments to a hybrid (virtualenv + Docker) approach

[glennmatthews]

Proposed Changes

Instead of two supported development environments (1: everything in Docker, 2: virtualenv + locally installed and managed postgres and redis), there is some momentum towards moving to a hybrid environment in which the Python/Django code runs natively in a virtualenv and Postgres and Redis are run as Docker containers. Ideally, rather than providing a third option for developers, this hybrid environment would replace both existing environments as the only officially recommended and documented developer environment.

Justification

Discussions amongst the core Nautobot development team indicate that a majority of the team are already using some form of hybrid approach rather than either of the documented workflows.

Moving away from volume-mounting live Python code into Docker would also reduce the likelihood of issues such as #148.

[jathanism]

One of the biggest issues with the currently documented virtualenv workflow is the fact that we instruct the reader to install both PostgreSQL & Redis "locally per the documentation". The documentation they are referred to is the production install documentation for Ubuntu or CentOS. This tends to infer that we are saying one must use Ubuntu or CentOS for their development environment, which is certainly not the intent in my opinion.

If we had it my way, we'd use the development environment approach I am using, which is a hybrid:

• Docker used for providing the services required (PostgreSQL & Redis) with their ports mapped to the host OS
• Localhost virtualenv via poetry install used to run the application code, and configured to talk to PG & Redis services running in containers.

Additionally, if we decide to go the hybrid route, we could still leave the underpinnings for using Docker or otherwise there for people to use if they desire, but just express that it is not our officially supported method.

For users of the Docker workflow

The change puts responsibility to the developer to manage their own runserver instance, but I've found more often than not, people are doing this anyhow so that they can keep the development_nautobot_1 container running, by commenting out nautobot-server run server and replacing it with bash. They then docker exec into the container and run nautobot-server runserver manually. If this happens often enough, the net benefit simplifies this workflow to not require that.

For users of the virtualenv workflow

The change clarifies that services should be ran using Docker, and eliminates the confusion of instructing the reader to "install locally" and pointing them to the production deployment docs, which is confusing at best.

For plugin authors

The plugin development guide needs to be moved into the "Development" section of the docs. We'd then need to clarify and streamline the guide around the single officially supported development workflow.

[jifox]

Hello Glenn,

I fully agree with prioritizing the hybrid approach. But I'm not convinced that dropping the docker development environment will be the right step. Especially when thinking of the upcoming "Dev Environments" feature for docker desktop.

see video: Docker Developer Environments - Feature Preview

[jathanism]

@jifox It's not that we would drop it, just more that it wouldn't be our primary and fully-documented option. Right now we are managing two and it's kind of not fun.

[jvanderaa]

I'd vote for having a "supported" methodology of the container methodology to maintain best practices as it relates to 12-factor app methodology, that the app is run in the same fashion regardless of the platform it is on. That doesn't stop any one from doing development in a virtual environment, but it is way easier to switch between versions of Python via containerization rather than local venvs.

By using containers, one can simulate the test process and get the same results as a CI system executing the tests.

[glennmatthews]

It's worth noting that post-#514, we no longer have a supported pure-virtualenv approach as we currently only document/support running Selenium as a Docker container and not as a natively managed process.

So perhaps we will still be unable to converge to a single official development workflow but instead just need to replace the pure-virtualenv documentation with documentation for the hybrid approach used by @jathanism and @lampwins., but also retaining the pure-Docker option as a supported alternative?

[tim-fiola]

Per @glennmatthews 's suggestion, I am adding this comment in this issue versus a new issue.

I am using a pure venv (no docker). I hit several problems when docker desktop was not running.

In unit tests section, I'd suggest that this comment below go in the Python Virtual Environment Workflow section instead of this section. It's more appropriate for a user to see that in a virtual environment setup section as they may not read all the way down to the Unit Tests section

By default invoke unittest will start and run the unit tests inside the Docker development container; this ensures that PostgreSQL and Redis servers are available during the test. However, if you have your environment configured such that nautobot-server can run locally, outside of the Docker environment, you may wish to set the environment variable INVOKE_NAUTOBOT_LOCAL=True to execute these tests in your local environment instead. See the Invoke configuration for more information.

I hit this error below when running in my local venv when trying a commit because docker desktop did not happen to be running:

(nautobot-6LHZxDho-py3.9) PycharmProjects/nautobot % git commit                                                   (tim-fiola-getting-started-webui)nautobot
Validating PEP8 compliance...
Running docker-compose command "ps --services --filter status=running"
Traceback (most recent call last):
  File "urllib3/connectionpool.py", line 670, in urlopen
  File "urllib3/connectionpool.py", line 392, in _make_request
  File "http/client.py", line 1255, in request
  File "http/client.py", line 1301, in _send_request
  File "http/client.py", line 1250, in endheaders
  File "http/client.py", line 1010, in _send_output
  File "http/client.py", line 950, in send
  File "docker/transport/unixconn.py", line 43, in connect
FileNotFoundError: [Errno 2] No such file or directory

Otherwise, we are adding a requirement for docker to be running for the venv setup that is not easy to find in the docs.