Migrating packaging system from setup.py to pyproject.toml

ganow commented 1 year ago

Changes

Migrated packaging system from setup.py (setuptools) to pyproject.toml (hatch)
Introduced optional requirements
Dropped caffe import in dl/caffe module
Introduced GitHub workflow to {lint w/ flake8, test w/ pytest, type check w/ mypy}

Verification

I have tested the following procedures work in the local environment.

Package building

$ git clone https://github.com/KamitaniLab/bdpy.git
$ cd bdpy && git fetch && git checkout update-packaging
$ hatch build
[sdist]
dist/bdpy-0.19.tar.gz

[wheel]
dist/bdpy-0.19-py3-none-any.whl

(update on 6/22) The following works:

$ pip install build
$ git clone https://github.com/KamitaniLab/bdpy.git
$ cd bdpy && git fetch && git checkout update-packaging
$ python -m build
* Creating venv isolated environment...
* Installing packages in isolated environment... (hatchling)
* Getting build dependencies for sdist...
* Building sdist...
* Building wheel from sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (hatchling)
* Getting build dependencies for wheel...
* Building wheel...
Successfully built bdpy-0.19.tar.gz and bdpy-0.19-py3-none-any.whl

Until I investigated the Python build system this time, I was unaware that the hatchling specified in pyproject.toml is a build backend, and hatch is a command line frontend used to initiate hatchling. It is not necessarily required to match the frontend with the build backend. The build tool, managed by PyPA, uses the appropriate build backend according to the specifications in pyproject.toml, so it was not necessary to install the hatch frontend.

Installation

$ pip install 'dist/bdpy-0.19-py3-none-any.whl[fig]'
...
Installing collected packages: pytz, tzdata, threadpoolctl, six, pyyaml, pyparsing, pillow, numpy, kiwisolver, joblib, fonttools, cycler, scipy, python-dateutil, h5py, contourpy, scikit-learn, pandas, matplotlib, hdf5storage, bdpy
Successfully installed bdpy-0.19 contourpy-1.1.0 cycler-0.11.0 fonttools-4.40.0 h5py-3.9.0 hdf5storage-0.1.19 joblib-1.2.0 kiwisolver-1.4.4 matplotlib-3.7.1 numpy-1.24.3 pandas-2.0.2 pillow-9.5.0 pyparsing-3.1.0 python-dateutil-2.8.2 pytz-2023.3 pyyaml-6.0 scikit-learn-1.2.2 scipy-1.10.1 six-1.16.0 threadpoolctl-3.1.0 tzdata-2023.3

I have NOT tested the following works (since it'll affect the production environment), but conceptually it must work.

Deploy

$ hatch publish

(update on 6/26)

$ python -m twine upload dist/*

Notes

I have created initial version of the pyproject.toml by mitsuhiko/rye (and I privately use it for managing my dev environment). After the creation of pyproject.toml, I have manually edited it so that it can handle extra deps. ~I demonstrated how to build&publish the package by using hatch command because I thought that currently, rye is a newly developed tool and hatch might be a better option to use as a repository default.~ I have updated the example build&deploy scripts from hatch to build and twine, which seem to be widely used tools in recent years.

ganow commented 1 year ago

Random notes

pip>19.02 can install deps directly from pyproject.toml
- we can do like: git clone ...; cd ...; pip install .[option] for the project only containing pyproject.toml without requirements.txt
- https://orolog.hatenablog.jp/entry/2019/03/24/223531
GitHub workflow can cache deps
- https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows
- https://github.com/actions/setup-python#caching-packages-dependencies

ganow commented 1 year ago

What I've found via GitHub workflow:

pip install nipy fails for Python >= 3.10
We have several syntax errors in bdpy/mri module

(update on 6/22) I've summarized why the nipy installation fails for Python >=3.10 in https://github.com/KamitaniLab/bdpy/issues/68#issuecomment-1602175865

(update on 6/26) I closed the issue on the nipy installation failure for Python >= 3.10 since we can expect that it will be resolved for the next nipy release. (see https://github.com/KamitaniLab/bdpy/issues/68#issuecomment-1606582724 for the details)

github-actions[bot] commented 1 year ago

Coverage Report

File	Stmts	Miss	Cover	Missing
bdpy
__init__.py	3	3	0%	10–12
bdpy/bdata
__init__.py	2	2	0%	7–8
bdata.py	402	402	0%	26–924
featureselector.py	64	64	0%	8–124
metadata.py	67	67	0%	8–154
utils.py	113	113	0%	4–258
bdpy/dataform
__init__.py	4	4	0%	7–10
datastore.py	107	107	0%	7–265
features.py	275	275	0%	8–516
pd.py	9	9	0%	7–44
sparse.py	67	67	0%	6–126
bdpy/distcomp
__init__.py	1	1	0%	6
distcomp.py	92	92	0%	7–127
bdpy/dl
caffe.py	60	60	0%	4–129
bdpy/dl/torch
__init__.py	2	2	0%	1–2
base.py	43	43	0%	6–105
models.py	104	104	0%	1–369
torch.py	109	109	0%	3–258
bdpy/evals
metrics.py	95	95	0%	3–179
bdpy/feature
__init__.py	1	1	0%	3
feature.py	30	30	0%	1–74
bdpy/fig
__init__.py	4	4	0%	6–9
draw_group_image_set.py	90	90	0%	3–182
fig.py	88	88	0%	16–164
makeplots.py	336	336	0%	1–729
tile_images.py	59	59	0%	1–193
bdpy/ml
__init__.py	7	7	0%	8–14
crossvalidation.py	59	59	0%	7–196
ensemble.py	13	13	0%	5–46
learning.py	296	296	0%	4–574
model.py	134	134	0%	4–277
regress.py	11	11	0%	6–38
searchlight.py	16	16	0%	4–51
bdpy/mri
__init__.py	7	7	0%	7–13
fmriprep.py	497	497	0%	4–866
glm.py	40	40	0%	4–95
image.py	24	24	0%	4–54
load_epi.py	28	28	0%	7–88
load_mri.py	19	19	0%	4–36
roi.py	248	248	0%	4–499
spm.py	158	158	0%	1–300
bdpy/opendata
__init__.py	1	1	0%	1
openneuro.py	178	178	0%	1–281
bdpy/preproc
__init__.py	3	3	0%	8–10
interface.py	52	52	0%	8–217
preprocessor.py	129	129	0%	8–236
select_top.py	22	22	0%	8–61
util.py	6	6	0%	6–22
bdpy/recon
utils.py	55	55	0%	4–146
bdpy/recon/torch
__init__.py	1	1	0%	1
icnn.py	159	159	0%	15–473
bdpy/stats
__init__.py	1	1	0%	14
corr.py	42	42	0%	7–116
bdpy/util
__init__.py	3	3	0%	8–10
info.py	46	46	0%	1–76
math.py	12	12	0%	1–32
utils.py	35	35	0%	8–146
TOTAL	4529	4529	0%

Tests	Skipped	Failures	Errors	Time
110	0 :zzz:	13 :x:	6 :fire:	12.347s :stopwatch:

ganow commented 1 year ago

The reason why coverage==0% even though we have several test cases

Perhaps the test directory structure matters. We can get correct coverage (≒30%) in the branch that refactors the test directory structure (#66). (both situations tested locally)

ganow commented 1 year ago

Memo: The GitHub workflow created in this PR only reports lint errors and type errors and does not actively intervene in existing code. Running the following commands (black and ruff with fix option) in {workflow/local environment} would resolve the error somewhat automatically.

$ black bdpy  # formatting w/ Black
$ ruff check --fix bdpy  # (partial) formatting w/ Ruff by enabling autofix option

As a project, Ruff is designed to be used alongside Black and, as such, will defer implementing stylistic lint rules that are obviated by autoformatting.

KamitaniLab / bdpy