search

brain-microstructure-exploration-tools / abcd-microstructure-pipelines

Processing pipelines to extract brain microstructure from ABCD Study dMRI
Apache License 2.0
0 stars 1 forks source link

cannot publish to pypi with direct HDBET dependency #43

Open ebrahimebrahim opened 3 months ago

ebrahimebrahim commented 3 months ago

Upon v0.1.0 release, got the following from our publish action:

ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/        
         Can't have direct dependency: HD-BET@                                  
         https://github.com/brain-microstructure-exploration-tools/HD-BET/archiv
         e/refs/tags/v1.0.0.zip#sha[25](https://github.com/brain-microstructure-exploration-tools/abcd-microstructure-pipelines/actions/runs/9003846883/job/24735448038#step:4:26)6=d48908854207b839552f2059c9cf2a48819b847bc
         1eb0ea4445d1d589471a1f5. See                                           
         https://packaging.python.org/specifications/core-metadata for more     
         information.

This is because we point to hdbet on github here and pypi seems to want all dependencies to be on pypi.

Originally posted by @ebrahimebrahim in https://github.com/brain-microstructure-exploration-tools/abcd-microstructure-pipelines/issues/42#issuecomment-2100763789

ebrahimebrahim commented 3 months ago

@allemangD I have an idea and I want to run it by you before trying it. Maybe we can make HD_BET a git submodule in this repo and then declare it to be a package that is to be installed alongside the abcdmicro package within this repo. Then somehow, IDK how yet, we can point to that locally packaged HD_BET as a dependency of abcdmicro... do you think there's any promise for such an idea? I just want your initial thoughts before jumping in. It'd be nice to not have to publish an extra thing to pypi just for this.

allemangD commented 3 months ago

A single distribution package (abcd-microstructure-pipelines) can contain multiple top-level import packages (abcdmicro, HD_BET). I don't think there's a way to "include" the HD BET setup.py, though, we'd need to port all that to our own pyproject.toml. See setuptools configuration. I think we'd set: 

[tool.setuptools.packages.find]
where = ["src", "vendor/HD-BET"]
include = ["abcdmicro", "HD_BET"]

If we did that we should change the import package name, though, so our HD_BET fork doesn't potentially clash with an upstream HD_BET installation...

I'm generally hesitant to use submodules, especially since we planned to modify parts of HD_BET API for our needs. Editing content of submodules always gets very tricky to coordinate in my experience.

We'll encounter all the same issues with the https://github.com/pnlbwh/conversion dependency.

ebrahimebrahim commented 3 months ago

ah "import package" and "distribution package" ... ty for giving me the more precise words for what I was trying to say

Okay ... I am convinced. I think I still like that idea of publishing our fork better than this.

We can call it HD-BET-abcdmicro-fork. Then for the "conversion" package we can use a similar naming scheme pnlbwl-conversion-abcdmicro-fork. Large and inconvenient names, but ones that exist just for the purpose of our project pointing to them.