A cookiecutter for Billinge-group packages.
This repository contains a cookiecutter template for generating a group-standard Python project complete with CI testing and documentation.
To do so, in a conda env that includes the cookiecutter package run:
cookiecutter https://github.com/billingegroup/cookiecutter
and follow the instructions.
This may only be a one-time deal as we bring all our packages up to the same level of uniformity (summer 2024), but these instructions will be left here in case we need to do this again in the future because of some desired change in our package structure.
Do not delete/remove any files before confirming that it is absolutely not necessary. contact Simon (or Andrew) for assistance. When copying over documentation files, make sure you include any additional package-specific information that may be in those files. For instance, there may be a more verbose description of what the package does, or tutorial/example/utility files. DO NOT REMOVE THEM.
Finally take a glance at the API/documentation workflow below. This should be done as the last step after the Pytest
tests are passing.
In your dev
folder, fork and clone the package that you are preparing for release
cd
into the top level directory of that project
git pull upstream main
(make sure you are synchronized)
Double check that no bug-fix etc. pull-requests are waiting to be merged. May as well get them merged before doing this. Check with Simon if not sure.
Before running the cookiecutter, we want to run black
and flake8
on the top level directory. Create a new branch and call it black
. Preferrably, do not initialize pre-commit.
Edit the pyproject.toml
so the section [tools.black]
is consistent with pyproject.toml
in {{ cookiecutter.repo_name }}
.
Activate an env that contains black and run black src
(note: some of the older packages do not have an src
directory, so you may have to run black on a different directory).
If it runs successfully and makes changes, commit the changes (if your pre-commit is activated you can override it with -n
to make these commits).
Let's run black on everything else. Run black .
and commit any edits that are made.
19 Now, edit the .flake8
file so it's consistent with .flake8
in {{ cookiecutter.repo_name }}
.
When this passes, open a PR and alert Simon.
When the PR is merged, update your main and create a new branch called precommit
.
Make sure that .pre-commit-config.yaml
is in your current directory. If it is not, copy it over from the cookiecutter.
In an env containing pre-commit, run pre-commit run --all-files
Make a new PR before doing any manual changes to files and have Simon merge it. Make sure tests are passing first though
Fix any errors and make periodic commits.
Make necessary edits and make periodic commits to make review easier.
Once complete, open a PR and alert Simon. When merged, you can continue on with the cookiecutter.
cookiecutter https://github.com/billingegroup/cookiecutter
setup.py
or pyproject.toml
in comma-separated string format, e.g., diffpy, pdf, something, something else
(no quotes around it)diffpy.pdfmorph
)<package_name>
, e.g., diffpy.pdfmorph
. Type ls
to check it is there.diffpy.<package_name>/
directory created by cookiecutter under the main directory (e.g., in our example pwd
would return ~/dev/diffpy.pdfmorph/diffpy.pdfmorph
) (we will refer to this as the cookiecutter directory).ls -als
(if you have the alias, this is ll
) compare the directory structures in this directory tree to that in the original repo to see what is different (ignore files at this point). Nothing to do here, just get familiar with the differences..git
directory from the main directory to the cookiecutter directory. From the main directory type mv ../.git .
.git checkout -b cookierelease
.cp -n
or cp --no-clobber
. e.g., from the cookiecutter directory where you currently are, type cp -n -r ../src .
if the old code is already in a directory src
. If there is no src directory, it will be something like cp -n -r ../diffpy ./src
.git status
. You will see a list of files that have been modified, deleted or are untracked. Untracked files are in the cookiecutter but not in the original repo, deleted files are in the original but haven't been moved over, and modified files are in both but have been changed.doc/<path>/source
file from the old repo to the doc/source
file in the new repo. In some old pacakges, <path>
will be something like manual
and in others it will just not exist.
manual
directory, go to your cookiecutter directory. Then run cp -n -r ../doc/manual/source/* ./doc/source
.../
or ..
and look at all relative path instances.cookierelease
branch by pushing your fork and opening a PR.git status
are in the old repo but not in the new cookiecutter. We took care of most these by moving over the src tree, but let's do the rest now. Go down the list and for git status
"delete" files type cp -n ../<filepath>/<filename> ./<target_filepath>
. Note that, generally, <filepath>
and <target_filepath>
will be the same, but may differ in cases such as /doc/manual/source
and /doc/source
respectively. If there are files there we don't want, don't move them over. Whenever files are copied over using the -n
command, you may delete them in the main repository to keep your file-tree clean.Git|current file|show diff
and the differences will show up. Select anything you want to inherit from the original file. In many cases we don't want to bring over things that have been polished in the cookiecutter, so you are mostly looking for code specific things, such as extended descriptions of the package in README and things like that.doc/manual/source/
in the old repo but are not doc/source
we correct by typing git add doc/manual/source
.This should be done only when the above steps are finished.
When you see files with ..automodule::
within them, these are API documentation. However, these are not populated. We will populate them using our release scripts.
dev
and running git clone https://github.com/Billingegroup/release-scripts.git
.cd ./diffpy.pdfmorph/diffpy.pdfmorph
.python -m build
. You may have to install python-build
first.diffpy.pdfmorph
, this is ./src/diffpy/pdfmorph
. In general, for a.b.c
, this is ./src/a/b/c
.python <path_to_auto_api> <package_name> <path_to_package_proper> <path_to_api_directory>
.
python ../../release-scripts/auto_api.py <package_name> <path_to_package_proper> ./doc/source/api
.Make sure you build the documentation by going to /doc
and running make html
.
The error "No module named" (e.g. WARNING: autodoc: failed to import module 'tools' from module 'diffpy.pdfmorph'; the following exception was raised: No module named 'diffpy.utils'
) can be resolved by adding autodoc_mock_imports = [<pkg>]
to your conf.py
right under imports. This file is located in /doc/source/conf.py
.
In the case of PDFmorph
, this was done by adding autodoc_mock_imports = ["diffpy.utils",]
.
Congratulations! You may now commit the changes made by auto_api.py
(and yourself) and push this commit to the cloud!
Make a PR! It will be merged, trust!
This should be done only when the above steps are finished.
cookierelease
activity make a <branchname>.rst
file by copying TEMPLATE.rst
in the news folder and under "fixed" put Repo structure modified to the new diffpy standard
README
and make sure that all parts have been filled in and all links resolve correctly.OK to release diffpy.<package-name>
"We are using diffpy.utils as a template for building the cookie cutter. To make sure the cookie cutter is giving output that is consistent with diffpy utils please use the following workflow:
dev
or sthg like thatdev
create a scratch area called scratch
diffpy.utils
in the scratch area of my hard-drivecookiecutter
and pexpect
in it.scratch/diffpy.utils
python /path/to/your/local/cookiecutter_project/test_utils.py
which for me was python ../../cookiecutter/test_utils.py
diffpy.utils
under the old one using the current version of the cookie cutter (watch out, cookie cutter sometimes caches so if you make a change and it is not reflected, clear the cache).cd .
)cp diffpy.utils/.pre-commit-config.yaml .
). It will overwrite the version that is already there.diffpy.utils
.cookiecutter
Adapted from the NSLS-II scientific cookiecutter, thanks guys!: https://github.com/nsls-ii/scientific-python-cookiecutter
Here are more in-depth docs for using this from NSLS-II, adapted for this repr: