search

ONSdigital / ons-python-template

A basic Python template to jump start a Python project
MIT License
3 stars 2 forks source link
ons-template

readme

ONS Python Template

Build Status Build Status License - MIT

This repository serves as a template for creating a Python project, complete with fundamental tooling and configuration. It is meant to help new python projects get started quickly, letting the user focus on writing code. The template takes care of the minutia of directory structures, tool configurations, and automated testing until the user is ready to take over.

This template is generated using Copier, an open source tool for rendering project from templates and natively supports updating projects as the original template matures.

See this demo repository for an example created from this template.

Table of Contents

Features

This template includes a number of features to help you get started developing your Python project quickly:

Getting Started

You have two options for project generation from this template:

Using GitHub Template Feature

[!NOTE] DO NOT FORK this repository. Instead, use the Use this template feature.

To get started:

  1. Click on Use this template
  2. Name your new repository and provide a description, then click Create repository. Note: the repository name should be lowercase and use hyphens (-) instead of underscores.
  3. GitHub will now copy the contents over and GitHub Actions will process the template and commit to your new repository shortly after you click Create repository.
  4. Wait until the "Rename Project from Template" job in GitHub Actions finished running!
  5. Once the Rename Project from Template action has run, you can clone your new repository and start working on your project. :rocket:

Known Limitations

Using Copier Locally

Prerequisites

  1. Python 3.10+: We recommend using pyenv for managing Python versions.

  2. Pip or Pipx

  3. Copier: Install Copier using pip or pipx.

    pip install --user copier
# OR
# Install pipx and add it to your PATH and then install Copier
pip install --user pipx && pipx ensurepath
pipx install copier

  4. Operation System: Ubuntu/MacOS

  5. Git: Ensure Git is installed and configured.

  6. GitHub CLI: [OPTIONAL] Ensure GitHub CLI is installed and you are authenticated (gh auth login) if you would like to automate the repository creation and configuration like branch protection.

Generate Project from Template

Copier will ask you a series of questions to customise the project to your needs. Once you have answered all the questions, Copier will generate the project for you.

To generate the project run:

copier copy --trust gh:ONSdigital/ons-python-template /path/to/your/new/project

Replace /path/to/your/new/project with the path to the directory where you want to create your new project. This directory should match the name of the repository you want to create.

Initialising a Git Repository and Pushing to GitHub

This step is only required if you answered No to the Do you want to set up the git repository? question. Otherwise, this would have been automatically done for you.

  1. Go to your project directory, and initialise a git repository and make the initial commit

    cd /path/to/your/new/project
git init -b main
git add .
git commit -m "Initial commit"

  2. Create a new repo in GitHub. See [GitHub How-to](https://docs.github.com/en/repositories/creating-and-managing-repositories/quickstart-for-repositories]

  3. Push your project to the repository on GitHub:

    git remote add origin https://github.com/<repository_owner>/<repository_name>.git
git push -u origin main

Now you can start working on your project. :rocket:

To update your project when the template changes, see Updating Project with Template Changes

Post-Clone Steps

There are a few steps you should take after cloning your new repository to ensure it is fully configured and ready for use.

1. Private Internal Reasoning Record (PIRR)

If your repository is private/internal, you should update the PIRR.md file in the root of your repository with the reasoning for the private/internal status of the repository.

2. Repository Settings

Familiarise yourself with the ONS GitHub Policy and ensure your repository is compliant with the policy. Few key points to note are:

If you answered Yes to the Do you want to set up the git repository? question, then these settings would have been automatically configured for you. However, it is recommended to review these settings to ensure they meet your requirements.

3. MegaLinter

Reducing the Docker image size for MegaLinter

MegaLinter is set up to use the largest Docker image by default, containing all available linters and code analysis tools. This setup is designed for comprehensive coverage and serves as a solid starting point for new projects. However, you might not need every tool included in this image, as it can be quite large.

To save space and optimise your setup, you can choose a more specific MegaLinter Docker image, called a flavor. Each flavor includes a subset of linters and tools suited for particular languages or frameworks.

If you're unsure which flavor is best for you, try running MegaLinter with the default set up after your project has matured. After the run, MegaLinter will analyse the output and suggest a more suitable flavor if necessary. This helps you customise your setup to include only the tools you need, reducing the Docker image size and improving efficiency.

Auto-fixing linting issues via GitHub Actions

If you would like to auto-fix issues using MegaLinter and commit the changes back to the PR, you can will need to create a Personal Access Token and add it as a secret to your repository. Without a PAT token, commits/PRs made by workflows do not trigger other workflows, including the MegaLinter workflow. This is a security feature of GitHub Actions to prevent infinite loops. For more info, see MegaLinter: Auto-fixing issues.

Updating Project with Template Changes

[!CAUTION] CURRENTLY UNSUPPORTED: This is currently unsupported due to an upstream issue with Copier. Once the issue is resolved, this section will be updated with instructions on how to update your project with changes.

View Details You can update your project with changes made to the template since you generated your project. This is useful to keep your project up to date with the latest tooling and configuration. If you always used Copier with this project, getting last updates with Copier is simple: ```bash cd ~/path/to/your/project make copier-update ``` Copier will ask you all questions again, but default values will be those you answered last time. Just hit Enter to accept those defaults, or change them if needed or you can use `poetry run copier update --force` instead to avoid answering the questions again. For more see [Copier docs](https://copier.readthedocs.io/en/stable/updating/) and `poetry run copier --help-all`.

Structure

The structure of the templated repo is as follows:

├── .github                           # Contains GitHub-specific configurations, including Actions workflows for CI/CD processes.
│   ├── linters                       # Directory for config files used by linter run via MegaLinter, e.g. .markdown-lint.json, .yaml-lint.yml etc.
│   ├── workflows                     # Directory for GitHub Actions workflows.
│   │   ├── ci.yml                    # Workflow for Continuous Integration, running tests and other checks on commits to `main` and on pull requests.
│   │   ├── codeql.yml                # CodeQL workflow for automated identification of security vulnerabilities in the codebase. (Public Repos Only)
│   │   └── mega-linter.yml           # MegaLinter workflow for linting the project.
│   ├── dependabot.yml                # Configuration for Dependabot, which automatically checks for outdated dependencies and creates pull requests to update them.
│   ├── ISSUE_TEMPLATE.md             # Template for issues raised in the repository.
│   ├── PULL_REQUEST_TEMPLATE.md      # Template for pull requests raised in the repository.
│   └── release.yml                   # Configuration on how to categorise changes into a structured changelog when using 'Generate release notes' feature.
├── app                               # Main Python package directory for the project, containing source code.
│   ├── __init__.py                   # Initialises the directory as a Python package, allowing its modules to be imported.
│   └── calculator.py                 # A simple Python class for demonstration purposes.
└── tests                             # Contains all test files.
│   ├── __init__.py                   # Marks the directory as a Python package, enabling the discovery of test modules by testing frameworks.
│   └── unit                          # Directory for unit tests, containing tests for individual components of the project.
│       ├── __init__.py               # Further organises tests into a Python package structure.
│       ├── conftest.py               # Contains pytest fixtures and configurations, applicable to all tests in the directory.
│       └── test_calculator.py        # Unit tests for the functionality provided by sample python code for demonstration purposes.
├── .copier-answers.yml               # Configuration file for Copier, specifying the answers to prompts when generating the project. Required for project updates.
├── .editorconfig                     # Configuration file for maintaining consistent coding styles for multiple developers working on the same project across various editors and IDEs.
├── .gitattributes                    # Git attributes file for defining attributes per path, such as line endings and merge strategies.
├── .gitignore                        # Specifies intentionally untracked files to ignore when using Git, like build outputs and temporary files.
├── .mega-linter.yml                  # Configuration file for MegaLinter, specifying the linters and code analysers to be used.
├── .pylintrc                         # Configuration file for Pylint.
├── .python-version                   # Specifies the Python version to be used with pyenv.
├── CODE_OF_CONDUCT.md                # A code of conduct for the project, outlining the standards of behaviour for contributors.
├── CONTRIBUTING.md                   # Guidelines for contributing to the project, including information on how to raise issues and submit pull requests.
├── LICENSE                           # The license under which the project is made available.
├── Makefile                          # A script used with the make build automation tool, containing commands to automate common tasks.
├── PIRR.md                           # Private Internal Reasoning Record (PIRR) for the repository, documenting the reasoning for the private/internal status of the repository. (Private/Internal Repos Only)
├── poetry.lock                       # Lock file for Poetry, pinning exact versions of dependencies to ensure consistent builds. (Exists if poetry is selected as the package manager)
├── pyproject.toml                    # Central project configuration file for Python, used by Poetry package manager and tools like Ruff, black etc.
├── Pipfile                           # Used by pipenv package manager to specify dependencies and their versions. (Exists if pipenv is selected as the package manager)
├── Pipfile.lock                      # Lock file for Poetry, pinning exact versions of dependencies to ensure consistent builds. (Exists if pipenv is selected as the package manager)
├── README.md                         # The main README file providing an overview of the project, setup instructions, and other essential information.
└── SECURITY.md                       # A security policy for the project, providing information on how to report security vulnerabilities.

Design Decisions

Although this template is opinionated, there are many alternatives to the tools used in this template which you may prefer. See the Alternatives Software/Tools section for more information.

1. Why use Poetry?

2. What is Ruff and why use Ruff?

3. Why use pylint and black when Ruff offers similar functionality?

4. Why use mypy for type checking instead of pytype, pyright, or pyre?

5. Why use pytest for testing instead of unittest?

6. What is MegaLinter?

7. Why is MegaLinter not used for Python?

8. Why not use SuperLinter?

9. Why does this not provide a full example of a Python project, i.e a Flask app, FastAPI app, Python package etc.?

10. My projects do not have a CodeQL workflow. Why?

11. Why is Secret Scanning and Push Protection not enabled?

Alternatives Software/Tools

There are many alternatives to the tools used in this template, and you may prefer to use some of these alternatives.

Future Plans

Development

:TODO: Add instructions for development

Contributing

See CONTRIBUTING.md for details.

License

See LICENSE for details.