Installation of ANIMA via pip

[jcohenadad]

currently, ANIMA is installed with git clone syntax (see here).

It would be much better if instead we could get ANIMA installed with a pip command that would run the git clone downlaod. Is that something that is doable? @kousu @lifetheater57 @RignonNoel

[lifetheater57]

Why would it be better with a pip command?

[kousu]

It means other projects can build on us in a conventional way and makes it a lot easier to get integrated into people's environments whether conda-based, apt-based, or brew-based. It also imposes or at least strongly encourages many best practices like separating your input data from your source code and using semantic versioning.

I've got the scripts for this ready in other repos. I'll make you a PR and you can see if it makes sense for you.

[kousu]

You were asking for more than I realized, because we need some way to wrap a different team's large collection of C programs inside of python somehow. So it took me all day, but I had a good time anyway working through it. The relationship is almost identical to SCT par rapport de ANTs, and I should be able to repurpose these lessons into there.

I have ANIMA packages over at

• https://github.com/ivadomed/anima-bin/releases/

You can try them out with

pip install --find-links https://github.com/ivadomed/anima-bin/releases/ ivadomed-anima-bin

Two major problems to resolve:

1. I don't have a PyPI token for ivadomed; the PyPI token in ivadomed's secrets is evidently locked down to that specific project.

   ``` INFO Response from https://upload.pypi.org/legacy/: 403 Invalid API Token: InvalidMacaroonError('invalid macaroon') INFO

   403 Invalid API Token: InvalidMacaroonError('invalid macaroon')

   Access was denied to this resource.
   
   Invalid API Token: InvalidMacaroonError('invalid macaroon') ERROR HTTPError: 403 Forbidden from https://upload.pypi.org/legacy/ Invalid API Token: InvalidMacaroonError('invalid macaroon') ```

   Who has the rights to the ivadomed PyPI project? I could use my personal PyPI tokens but I'd rather not leave that in place long term.

2. There's a 100MB limit on PyPI, and ANIMA is ~600MB so even if I do use my personal PyPI token it fails.

   ``` INFO Response from https://test.pypi.org/legacy/: 400 File too large. Limit for project 'ivadomed-anima-bin' is 100 MB. See https://test.pypi.org/help/#file-size-limit for more information. INFO

   400 File too large. Limit for project 'ivadomed-anima-bin' is 100 MB. See https://test.pypi.org/help/#file-size-limit for more information.

   The server could not comply with the request since it is either malformed or otherwise incorrect.
   
   File too large. Limit for project 'ivadomed-anima-bin' is 100 MB. See https://test.pypi.org/help/#file-size-limit for more information. ERROR HTTPError: 400 Bad Request from https://test.pypi.org/legacy/ File too large. Limit for project 'ivadomed-anima-bin' is 100 MB. See https://test.pypi.org/help/#file-size-limit for more information. ```

   I can solve this by splitting it up into sub-packages, but that's going to take more than one day of work.

And two minor problems:

3. It doesn't work on macOS 10.13

   ``` (anima-test) parker:~ p115628$ sw_vers ProductName: Mac OS X ProductVersion: 10.13.6 BuildVersion: 17G14042 (anima-test) parker:~ p115628$ pip install --find-links https://github.com/ivadomed/anima-bin/releases/ ivadomed-anima-bin Looking in links: https://github.com/ivadomed/anima-bin/releases/ ERROR: Could not find a version that satisfies the requirement ivadomed-anima-bin (from versions: none) ERROR: No matching distribution found for ivadomed-anima-bin ```

   But I think this is more an issue of we need to upgrade our own systems; Github doesn't even provide 10.13 images, 10.15 is the minimum. And actually, despite asking for macos-latest which is supposed to be macOS 11, the script built for "macos_10_15". So I think 10.13 is probably just too old or something.

   I can try building on runs-on: macos-10.15 but I'm skeptical that will make a difference.

4. I need someone to test Windows for me.

Once this is working, ms-challenge-2021 will have to be reworked to use the new location of the ANIMA apps. It should even simplify the code, since using pip automatically adds the apps to $PATH. And of course by using pip it means we can pin the version down, if we need to. But there's still some hours left before we're there.

And that gives me a fifth thing to solve:

5. The "Usage" section should cover how to access the apps from python code: what to put in install_requires and what to put at runtime (import anima.bin; subprocess.check_output([Path(anima.bin.__file__).parent / "animaKMeansClustering", ....])?).

[kousu]

@dyt811 reports that the Windows packages install and at least run --version without crashing. So that's one issue checked off.

[kousu]

I've split up the package and that successful got it published to (test)PyPI. (there's another copy of everything on GitHub).

There was a bunch of python packaging weeds I had to learn about -- like that the manylinux tags are not generated by wheel -- you need to either do them manually or have auditwheel help you figure them out, but even then it's only half helpful in this case because it was written with the strong assumption that it would be running on the same system the compilation happened on -- which is not true here, I'm just repacking someone else's work. But once I got through all that I got it working.

There's still a couple of kinks -- I keep breaking the macOS or Windows builds in subtle ways -- so this version only comes with 6 apps. But the version with all 102 is really close to being ready.

You can try it out with

pip install -i https://test.pypi.org/simple/ ivadomed-anima-bin

And that will get you animaGeneralizedFA, animaGraphCut, and some others on your $PATH (only 6 at the moment; but maybe by the time you see this it will get all 102). It should work on all Linux, mac and Windows machines anyone might care to try it on. Not like, a really old Mac system, but I tried it on some of our macOS 10.13 systems (which aren't getting updates anymore and should really be upgraded!) and it even worked there, so I'm feeling pretty confident.

[kousu]

:tada: :+1:

https://github.com/ivadomed/anima-bin/runs/6952221852?check_suite_focus=true:

https://test.pypi.org/project/ivadomed-animaConvertImage/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaDiceMeasure/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaFDRCorrectPValues/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaPatientToGroupComparisonOnTracks/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaT1SERelaxometryEstimation/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaDTIScalarMaps/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaMajorityLabelVoting/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaDTIProbabilisticTractography/4.1.1rc 3/ https://test.pypi.org/project/ivadomed-animaMorphologicalOperations/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaNLMeansTemporal/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaLinearTransformArithmetic/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaConnectedComponents/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaDenseTransformArithmetic/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaFibersFDRCorrectPValues/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaGeneralizedFA/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaMCMApplyTransformSerie/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaLinearTransformToSVF/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaODFApplyTransformSerie/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaGcStremMsLesionsSegmentation/4.1.1rc 3/ https://test.pypi.org/project/ivadomed-animaSimuBlochIR-SE/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaNyulStandardization/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaFibersDiseaseScores/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaCollapseImage/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaRemoveTouchingBorder/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaCreateImage/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaMultiT2RelaxometryEstimation/4.1.1rc 3/ https://test.pypi.org/project/ivadomed-animaODFEstimator/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaSymmetryConstrainedRegistration/4.1. 1rc3/ https://test.pypi.org/project/ivadomed-animaSymmetryPlane/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaLowMemCramersTest/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaT2RelaxometryEstimation/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaDWISimulationFromMCM/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaMCMScalarMaps/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaTransformSerieXmlGenerator/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaT1RelaxometryEstimation/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaDTIEstimator/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaMCMProbabilisticTractography/4.1.1rc 3/ https://test.pypi.org/project/ivadomed-animaMergeBlockImages/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaLowMemNLMeansPatientToGroupCompariso n/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaSimuBlochGRE/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaMaskImage/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaCramersTest/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaLocalPatchCovarianceDistance/4.1.1rc https://test.pypi.org/project/ivadomed-animaNLMeans/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaExpTensors/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaT2EPGRelaxometryEstimation/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaFibersCounter/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaTensorApplyTransformSerie/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaIsosurface/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaSimuBlochIR-GRE/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaPyramidalBMRegistration/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaAnalyticAssociatedLegendre/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaImageMosaicing/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaFibersApplyTransformSerie/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaROIIntensitiesStats/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaFillHoleImage/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaConcatenateImages/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaSegPerfAnalyzer/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaInfluenceZones/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaLogTensors/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaApplyTransformSerie/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaFibersFilterer/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaImageArithmetic/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaSimuBlochCoherentGRE/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaSimuBlochSE/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaGraphCut/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaAverageImages/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaLesionEvolutionDetection/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaKMeansClustering/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaImageResolutionChanger/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaBMDistortionCorrection/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaDetectedComponents/4.1.1rc3/ https://test.pypi.org/project/ivadomed-animaMCMAverageImages/4.1.1rc3/

Now just to tidy the README to make it more presentable and actually do the real builds that publish to real PyPI, and then modify ms-challenge-2021 to actually make use of it.

[naga-karthik]

Hey @kousu, just a quick clarification, does it install the latest anima version when we use pip install ivadomed-anima-bin? The version that we used for the ms-challenge-2021 was Anima v4.1 but now I see that Anima v4.2 is released.