Recommendations for Python Development

[zbyte64]

Request for comments: what practices should we include in the writing of python code?

Here are a few to get the ball rolling:

• Use black for automatic code formatting: https://github.com/psf/black
• Write tests with pytest: https://docs.pytest.org/en/latest/
• All projects should have a requirements.txt , preferably with pinned requirements

[stephengtuggy]

I don't use black for code formatting. I use Visual Studio Code for that.

@zbyte64 what are your thoughts on this?

[stephengtuggy]

Write tests with pytest? Absolutely. requirements.txt? Definitely.

I might also add:

• For Django projects, at least, use Docker whenever possible. When this is not feasible, use virtualenv.
• Use spaces, not tabs, for indentation. 4 column "tab stops."

[DropsOfSerenity]

My advice is:

• Don't recommend code formatting tools unless you enforce them with automation (git hooks, CI)
• Django has django.test.* and test cases should inherit from the Django test cases as they handle things like test database teardown spinup, etc.
• pip freeze > requirements.txt should be used to generate requirements file. Pipfile was going to be a thing but it looks abandoned (https://github.com/pypa/pipfile)

[DropsOfSerenity]

Other recommendations:

• Make POPO's (plain old python objects,) basically classes that follow the single responsibility principle.
• Agreed with use docker, or use virtualenvwrapper a nice shell wrapper that lets you manage multiple python virtualenvs.
• Don't use the django admin as a user facing administration tool, they recommend against this. Instead use it as a low cost administration tool for developers.
• Avoid N+1 queries and other simple but costly errors by using https://github.com/jazzband/django-debug-toolbar

[stephengtuggy]

• Don't use the django admin as a user facing administration tool, they recommend against this. Instead use it as a low cost administration tool for developers.

I think I have seen that recommendation before. Can you link to a source? Thanks.

[DropsOfSerenity]

[DropsOfSerenity]

From django documentation. https://docs.djangoproject.com/en/2.1/ref/contrib/admin/#module-django.contrib.admin

[stephengtuggy]

@DropsOfSerenity Hmm. That's interesting. I had a different takeaway after reading the exact same paragraph. It sounded to me like trusted users / administrators within a client organization could be given access to the admin interface, while the average user would not have that access. That's how it is set up for Ultra Gro.

[stephengtuggy]

Also for Ultra Gro, I ended up writing tests that fit both the pytest and django.test.* paradigms. Example file here.

[DropsOfSerenity]

They explain the position in some django tickets on their issue tracker as well. The general jist of the thing is:

• Because of how much is autogenerated it is usually more information than a typical administrator of a system would need.
• Without configuration the administrator can easily mess up the system, because operations are entirely unrestricted by default.
• Configuring the admin for use by a non-technical user would take more time and effort than building a purpose built admin interface, a custom built admin is less error prone, less effort to maintain, and more robust.
• Custom workflows and widgets are a headache in django admin and are often very difficult to implement due to the large amount of magic going on behind the scenes.
• The developers are the only users than understand the data, and how they fit together, users do not understand unspoken rules of how the data should fit together, without that information built in as actual rules in the system, actual UX, the admin user are prone to messing up the data.

From experience stripping down the admin to a place where the UX is good for a non-technical administrator is not worth the time.

[DropsOfSerenity]

@stephengtuggy You can access the testing database in a regular django TestCase that test should work without that decorator I believe, is there any reason you stuck with both?

[stephengtuggy]

As I recall, the original tests were written for pytest. I wanted to keep my tests backwards-compatible, while also making them compatible with the more usual Django test framework.

In addition, the test code seemed more elegant that way. If one can make the same (test) code serve two different purposes without duplication, why not, y'know?

[carlosvargas]

• I second the use of black.

  @stephengtuggy the Python VSCode plugin includes it. By default it uses autopep8, but you can change it to black.

  @DropsOfSerenity Corey wants to start making sure we use a CI tool for all of our projects. We can me this part of it. Worst case, we can always just add a git pre-commit hook to the repo. edit: I'm actually going to make CI tool a hard requirement for BWRD projects.

• I vote for containers as well. For those of us using VSCode, MS just release a nice extension that handles remoting to a container to do debugging and a bunch of other things: https://devblogs.microsoft.com/python/remote-python-development-in-visual-studio-code/

  They had a talk at DjangoCon where they went in depth into this extension, but the videos aren't up yet.

• We could have another discussion about Django Admin.

• @dmpayton wrote a nice logging and performance utility similar to the Django-debug-tool bar that he's willing to share (right, Derek? lol).

• We should also pick gunicorn or uwsig. Although I know that @dmpayton you are using something else in Ordrslip because of your use of Channels, right?

• We should decide on a specific middleware for Django to do REST (ie DjangoRestFramework, etc) and GraphQL (ie graphene-django, etc).

• I'm also in favor of Static Type Cheking, so I would recommend the use of mypy or something similar.

• There is also a nice tool that we learned about at DjangoCon called: https://pypi.org/project/automagic-rest/. It generates your Models and Serializers from an existing PostgreSQL db. It won't be useful in all of the projects, but I think it will come in handy once in a while.

[kennethkoontz]

couple of thoughts

• pipenv which creates a pipfile.lock, think of npm or yarn for python
• gunicorn is pretty much defacto for wsgi in production and is pretty solid
• for channels or any server that needs asgi, daphne or uvicorn is good, i've used daphne in production and i think ordrslip uses uvicorn for asgi
• i agree that any type of code formatter should be checked in CI/automated or at the very least able to run as a command line tool and agnostic of any editor, black is a great candidate
• DRF is a very mature, and stable kit. i highly recommend it for REST requirements. It's also super flexible to extend.

[stephengtuggy]

• pipenv sounds great.
• gunicorn is what Ultra Gro uses. I would be happy to make that our standard.
• I have also heard good things about Django Rest Framework.

[zbyte64]

I like gunicorn but would encourage others to experiment, especially with all the new asynchronous python going around.

In the same theme of experimentation, I wouldn't want to prevent someone from trying new libraries like: https://github.com/strawberry-graphql/strawberry ; as long as it is shown to have a healthy community and fits the requirements.

[DropsOfSerenity]

Code quality is to me less about a set of perscriptive technologies, and more about a set of techniques to follow to keep your code clean. I agree with @zbyte64 perhaps we should define the borders of what we want "Python Code Quality" to accomplish and where it's boundaries should be.

General recommendation IMO should be preferred, the things I see people running into most in django is things like, not using select_related (not understanding sql), running into n+1 queries, not understanding what the query language can do for you (looping over result sets instead of querying for what you want), not using null, blank, etc. correctly on the models/migrations, not using django admin for purpose, being afraid of class based views, being afraid of django forms, and being afriad to construct their own plain old python classes, so personally that's where i would target the effort.

[carlosvargas]

I agree @zbyte64 and @DropsOfSerenity. We have to point out all those things and let people experiment, but we also need to provide at least some recommended libraries/frameworks/tools that they can use.

The reason is that a lot of people here don't have much experience with Python/Django or simple don't want to spend hours doing research on what's best to use. So we need to provide something for them to get started.

Maybe it should be in a separate Github issue, but it's something that we need to have.

[stephengtuggy]

This may be relevant: https://www.toptal.com/django/django-top-10-mistakes

[stephengtuggy]

As a follow-up to what @carlosvargas said, what about renaming this issue to "Shift3 Recommendations for Python Development" or similar?

[DropsOfSerenity]

The reason is that a lot of people here don't have much experience with Python/Django or simple don't want to spend hours doing research on what's best to use. So we need to provide something for them to get started.

Good point.

[michaelachrisco]

Revisiting this in slack. Looking for feedback from those of you that use Python/Django!

[michaelachrisco]

As per feedback from above, renamed the issue to Recommendations for Python Development. Look forward to getting more feedback from everyone!