search

PCMDI / pcmdi_utils

Collection of utilities for the PCMDI project
https://pcmdi.github.io/pcmdi_utils/
Apache License 2.0
1 stars 0 forks source link

readme

PCMDI Utils

PCMDI Utils aims to host reuseable functions across the PCMDI project in one place.

The target audience was originally developers from the PCMDI, E3SM Project, and LLNL Climate Program_, but it can be used by anyone in the open-source community!

The repository is initiated by using the Python Template Repo_ generated by @tomvothecoder

.. E3SM Project: https://github.com/E3SM-Project .. xCDAT: https://github.com/xCDAT/xcdat .. _PCMDI Metrics: https://github.com/PCMDI/pcmdi_metrics .. PCMDI: https://pcmdi.llnl.gov .. LLNL Climate Program: https://climate.llnl.gov .. _Python Template Repo: https://github.com/tomvothecoder/python-template-repo

CURRENTLY UNDER DEVELOPMENT!!

Features

.. raw:: html

Python logo Anaconda logo

Getting Started

CURRENTLY UNDER CONSTRUCTION

Resources for developers

  1. Clone this repository

Follow the steps below to get this repository cloned to your local environment

.. code-block:: console

  git clone git@github.com:PCMDI/pcmdi_utils.git
  1. Set up the Anaconda Development Environment

Follow the steps below to get the Anaconda development environment set up for your library. We recommend using mamba_ instead of conda because it is significantly faster and more reliable.

  1. Download and Install Anaconda (Mac OS & Linux)

    .. code-block:: console

    curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh" bash Mambaforge-$(uname)-$(uname -m).sh

  2. Update /conda-env/*.yml by follow each file's #TODO comments (don't worry, it's straightforward)

  3. Create and activate the Conda development environment

    .. code-block:: console

    mamba env create -f conda-env/dev.yml mamba activate template-dev

.. _mamba: https://mamba.readthedocs.io/en/latest/

  1. Set up the Documentation

The library's documentation is setup with Sphinx and Read the Docs Sphinx Theme.

You just need to do a few things to get up and running:

  1. Update README.rst, AUTHORS.rst, HISTORY.rst, and /docs/index.rst as needed

  2. Decide how to deploy the documentation. This step is left up to you based on your needs. Follow the instructions in the provided links.

    • Option 1: Read the Docs_
    • Option 2: GitHub Pages_ (via GitHub Actions)

.. _Read the Docs: https://docs.readthedocs.io/en/stable/tutorial/index.html .. _GitHub Pages: https://coderefinery.github.io/documentation/gh_workflow/

To build the documentation locally (useful for reviewing):

  1. Activate the conda developer environment

    .. code-block:: console

    mamba activate template-dev

  2. Build documentation

    • Option 1: make docs from the root of repo (also cleans up and opens docs in your browser)
    • Option 2: cd docs && make html

.. _Sphinx: https://www.sphinx-doc.org/en/master/ .. _Read The Docs Sphinx Theme: https://sphinx-rtd-theme.readthedocs.io/en/stable/

  1. Set up Quality Assurance Tools

This repository includes quality assurance (QA) tools for code formatting (black, isort), linting (flake8), unit testing (pytest), and optional static type checking (mypy). These tools ensure that you can easily catch issues and follow good Python practices without sacrificing energy on them. These tools are integrated in the pre-commit_ package as "hooks" that automatically run when committing changes to files.

To run these QA tools through pre-commit:

  1. Activate the conda development environment

    .. code-block:: console

    mamba activate template-dev

  2. Install pre-commit hooks into the Git repository

    .. code-block:: console

    $ pre-commit install pre-commit installed at .git/hooks/pre-commit

    • pre-commit will now run automatically on git commit!

  3. (optional) Run against all files

    • it's usually a good idea to run the hooks against all of the files when adding new hooks (usually pre-commit will only run on the changed files during git hooks)

    .. code-block:: console

    $ pre-commit run --all-files [INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks. [INFO] Initializing environment for https://github.com/psf/black. [INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks. [INFO] Once installed this environment will be reused. [INFO] This may take a few minutes... [INFO] Installing environment for https://github.com/psf/black. [INFO] Once installed this environment will be reused. [INFO] This may take a few minutes... Check Yaml...............................................................Passed Fix End of Files.........................................................Passed Trim Trailing Whitespace.................................................Failed

    • hook id: trailing-whitespace

    • exit code: 1

      Files were modified by this hook. Additional output:

      Fixing sample.py

      black....................................................................Passed

Useful commands:

Information on QA tools:

.. _pre-commit: https://pre-commit.com/ .. _black: https://black.readthedocs.io/en/stable/ .. _isort: https://pycqa.github.io/isort/ .. _flake8: https://flake8.pycqa.org/en/latest/ .. _mypy: https://mypy.readthedocs.io/en/stable/ .. _PEP (Python Enhancement Proposals): https://peps.python.org/pep-0000/

  1. Decide How to Distribute Software on Anaconda

There are several ways of distributing software on Anaconda including:

  1. Using conda-forge_ (recommended)
  2. Managing your own Anaconda channel_ and uploading packages there

I recommend using conda-forge. "conda-forge is a GitHub organization containing repositories of conda recipes. Thanks to some awesome continuous integration providers (AppVeyor, Azure Pipelines, CircleCI and TravisCI), each repository, also known as a feedstock, automatically builds its own recipe in a clean and repeatable way on Windows, Linux and OSX.""

.. conda-forge: https://conda-forge.org/ .. Anaconda channel: https://docs.anaconda.com/navigator/tutorials/manage-channels/

Helpful Knowledge

Reminder to Routinely Maintain the Software Infrastructure


It is generally good practice to maintain your software infrastructure. Here is a
routine software maintenance checklist. I recommend doing these before every new
software version release.

1. Maintain conda environment dependencies in `.yml` files
2. Update `pre-commit` hooks in `.pre-commit-config.yml`

How GitHub Actions is Used for CI/CD

This repository uses GitHub Actions to run the CI/CD build workflow. This workflow is automatically triggered with commits on pull requests to main and merging pull requests to main.

Jobs include:

  1. Run pre-commit for formatting, linting, and type checking
  2. Build conda CI/CD environment with different Python versions, build and install the package, and run unit test suite with pytest

Here's an example of GitHub Actions in "action": https://github.com/tomvothecoder/python-template-repo/actions

To save time and resources on build cycles, GitHub Actions has been configured to automatically skip jobs instead of re-executing based on the files that are committed. For example, if docs are committed then the unit tests will not run.

Additional Configuration



- Feel free to modify the configuration for QA tools in ``pyproject.toml`` and ``setup.cfg``
- You can also remove some or all the QA tools if you want (not recommended though).
  Just make sure to delete them from your Anaconda ``.yml`` files and remove their entries
  in ``pre-commit-config.yaml`` and their configs in ``pyproject.toml`` and/or ``setup.cfg``.
- `mypy` is disabled by default with `pre-commit`. Enable `mypy` by uncommenting the
  lines related lines in ``pre-commit-config.yaml``.