[🐕 Batch]: Submit Ersilia package to Conda Forge in a GitHub Action

miquelduranfrigola commented 7 months ago

Summary

We would like to offer Ersilia as a Conda package. Updates should happen automatically at the same time that submission to PyPi happens (there is a specific GitHub Action for this).

Some considerations:

Should we use conda-forge?
All dependencies are centralized through the poetry specifications file. Therefore, we have to create the characteristic meta.yaml file dinamically. One possibility is to create the .conda folder dynamically as part of the GitHub Action.

Objective(s)

In principle, the GitHub Action should do the following

[ ] Be triggered at the same time as the PyPi action.
[ ] Extract the requirements.txt from the poetry specs in Ersilia.
[ ] Create the .conda folder, containing the build.sh and meta.yaml files. The meta.yaml file should refer to the requirements.txt file created above.
[ ] Versioning should mirror the PyPi version.
[ ] Submit the package to Anaconda (see this GitHub Action) or Conda-Forge.

I am unsure if we do have an Ersilia organization already in Conda-Forge or Anaconda, and if we are using it at all. If not, let's create one.

Documentation

https://conda-forge.org/docs/maintainer/adding_pkgs/

DhanshreeA commented 3 months ago

There are three ways of generating a conda recipe (meta.yaml), viz. grayskull, conda skeleton, and from scratch. Each of these approaches present their own limitations, discussed below:

Grayskull has an issue with multi constraint dependencies when using a pyproject.toml with a poetry build backend. Works with others (eg Willow, and pytest)
Conda skeleton wants there to exist a setup.py in the sdist, however Poetry generated builds do not have setup.py in the sdist. Conda skeleton also does not take GitHub as a source URL:
For grayskull, there can be two options:
1. We move away from the build backend of poetry and use setup tools
2. We only support conda installations over Python 3.8 (by rewriting the pyproject.toml)
We try writing a recipe from scratch - not generally in favor of this option, but we can try as a worst case scenario.

Now there are two solutions from here:

We completely drop support for Python 3.7, which is probably not a good idea, or we support conda builds only with Python3.8 and above, maintaining a separate pyproject.toml for that (although this seems a bit hacky)
We could write a meta.yaml from scratch mentioning multi constraint dependencies as done here however based on the discussion in the same issue, it seems it might not work with other conda tools such as conda-smithy. However, still worth a try nonetheless.

Update: I could generate a meta.yaml with multi-constraint dependencies based on the discussion linked above, however I could not get conda-build to actually generate a package with this syntax in the meta.yaml. So, I am proceeding with only keeping support for Python 3.8 and above.

DhanshreeA commented 3 months ago

Here's a meta.yaml generated with dependencies support for Python3.8 and above using my fork:

{% set name = "ersilia" %}
{% set version = ".0.1.33" %}

package:
  name: {{ name|lower }}
  version: {{ version }}

source:
  url: https://github.com/DhanshreeA/ersilia/archive/v{{ version }}.tar.gz
  sha256: c7317678ae3050f7e82cee2d069a6f077818ae33da720e0166ff7dbd5fe57516

build:
  entry_points:
    - ersilia = ersilia.cli:cli
    - bentoml = ersilia.setup.requirements:check_bentoml
  noarch: python
  script: {{ PYTHON }} -m pip install . -vv --no-deps --no-build-isolation
  number: 0

requirements:
  host:
    - python >=3.8
    - poetry-core
    - pip
  run:
    - python >=3.8
    - inputimeout >=1.0.4,<2.0.0
    - emoji >=2.8.0,<3.0.0
    - validators >=0.21.0,<0.22.0
    - pandas >=1.3.0,<1.4.0
    - h5py >=3.7.0,<4.0.0
    - loguru >=0.6.0,<0.7.0
    - pyairtable <3
    - pyyaml >=6.0.1,<7.0.0
    - dockerfile-parse >=2.0.1,<3.0.0
    - tqdm >=4.66.1,<5.0.0
    - click >=8.1.7,<9.0.0
    - docker-py >=6.1.3,<7.0.0
    - boto3 >=1.28.40,<2.0.0
    - requests >=2.31.0,<3.0.0
  run_constrained:
    - isaura 0.1
    - pytest >=7.4.0,<8.0.0
    - fuzzywuzzy >=0.18.0,<0.19.0
    - sphinx >=5.3.0
    - jinja2 >=3.1.2,<4.0.0

test:
  imports:
    - ersilia
  commands:
    - pip check
    - ersilia --help
    - bentoml --help
  requires:
    - pip

about:
  home: https://github.com/ersilia-os/ersilia
  license: GPL-3.0 AND GPL-3.0-or-later
  license_file:
    - LICENSE
    - ersilia/hub/content/metadata/license.txt

extra:
  recipe-maintainers:
    - DhanshreeA

DhanshreeA commented 3 months ago

The above adapted for ersilia's org account:

{% set name = "ersilia" %}
{% set version = "0.1.34" %}

package:
  name: {{ name|lower }}
  version: {{ version }}

source:
  url: https://github.com/ersilia-os/ersilia/archive/v{{ version }}.tar.gz
  sha256: 1714492189d07471370d6c11fe4d820cb51d4635009b2b9e73dcf70458f81125

build:
  entry_points:
    - ersilia = ersilia.cli:cli
    - bentoml = ersilia.setup.requirements:check_bentoml
  noarch: python
  script: {{ PYTHON }} -m pip install . -vv --no-deps --no-build-isolation
  number: 0

requirements:
  host:
    - python >=3.8
    - poetry-core
    - pip
  run:
    - python >=3.8
    - inputimeout >=1.0.4,<2.0.0
    - emoji >=2.8.0,<3.0.0
    - validators >=0.21.0,<0.22.0
    - pandas >=1.3.0,<1.4.0
    - h5py >=3.7.0,<4.0.0
    - loguru >=0.6.0,<0.7.0
    - pyyaml >=6.0.1,<7.0.0
    - dockerfile-parse >=2.0.1,<3.0.0
    - tqdm >=4.66.1,<5.0.0
    - click >=8.1.7,<9.0.0
    - docker-py >=6.1.3,<7.0.0
    - boto3 >=1.28.40,<2.0.0
    - requests <=2.31.0
  run_constrained:
    - isaura 0.1
    - pytest >=7.4.0,<8.0.0
    - fuzzywuzzy >=0.18.0,<0.19.0
    - sphinx >=5.3.0
    - jinja2 >=3.1.2,<4.0.0

test:
  imports:
    - ersilia
  commands:
    - pip check
    - ersilia --help
    - bentoml --help
  requires:
    - pip

about:
  home: https://ersilia.io
  license: GPL-3.0-only
  license_file:
    - LICENSE
  summary: 'A hub of AI/ML models for open source drug discovery and global health'
  dev_url: https://github.com/ersilia-os/ersilia

extra:
  recipe-maintainers:
    - DhanshreeA

I created this recipe by forking ersilia, and editing the pyproject.toml to reflect support only for Python 3.8 and above. Then I downloaded the tarball from the github release and calculated its sha256 sum to put here, otherwise the auto generated stuff from grayskull was reflecting the shasum of the release on my fork.

To automate this, we can use this skeleton to edit these values in an action.

Edits as of June 4, 2024:

License is now has a valid SPDX identifier
Removed pyairtable as a hard dependency
Replaced version v0.1.33 with v0.1.34 and its corresponding SHA256sum

miquelduranfrigola commented 3 months ago

Thanks @DhanshreeA - I would add this in this GitHub Action, which is currently only uploading to PyPi: https://github.com/ersilia-os/ersilia/blob/master/.github/workflows/python-publish.yml

DhanshreeA commented 3 months ago

Depends on #1143 and #1142 and currently missing a summary line

DhanshreeA commented 3 months ago

Latest fully functional PR: https://github.com/conda-forge/staged-recipes/pull/26552 (Note windows package obviously doesn't work and we don't claim to support Ersilia on windows anyway)

miquelduranfrigola commented 2 months ago

Hey @DhanshreeA what is the status of this?

miquelduranfrigola commented 2 months ago

Hi @DhanshreeA can this issue be closed?

ersilia-os / ersilia