Conda Bump version

[jburel]

This PR introduces a new command scc bump-version-conda to update the version and sha256 in the meta.yaml files in the conda- repositories. The command will be run in a GitHub action using a scheduled job in the conda- repositories i.e. similar to a bot. If changes are detected, the meta files are modified, committed and a PR is created. Note that the method bump-version-conda does not create the PR, this done via GitHub action in order to reduce the boiler plate required to create a PR

What does this PR do?

• Find the meta.yaml files in the conda-* repositories
• Find the latest tag from the target repository using git ls-remote. rc, dv, m* are not considered
• Check the version in the meta file.
• If no version changed, the process stops
• If version changed, determine which sha256 to use depending on the package. This could be from GH, pypi, downloads.o.org
• Update the version/sha256 in the meta.yml files
• commit changes.

To test this PR:

• In a virtual environment, install this branch
• Activate the environment

Test 1:

• Go to conda-omero-py:
• run scc bump-version-conda
• nothing should be committed.
• modify the version in meta file, save, do not commit.
• run the command again
• the process will run, the file will be updated but the commit will not occur

Test 2:

• Go to conda-bftools:
• run scc bump-version-conda
• the 2 meta files should be modified: version and sha256. Check that the version is 6.7.0, and the sh256 is the value from https://downloads.openmicroscopy.org/bio-formats/6.7.0/artifacts/bftools.zip.sha1
• Changes should be committed

Test 3:

• conda-raw2ometiff
• run scc bump-version-conda
• Check that nothing is changed i.e. rc is not applied.

Workflow in action https://github.com/jburel/conda-omero-py/pull/4

cc @joshmoore @sbesson

[jburel]

A minor formatting issue is that the YAML file is written with different indentation style, making the diff longer than it should be.

The BF conda file does not follow any of the conventions used in the other repositories, leading to the indentation style difference

[jburel]

In addition to the various inline comment, the primary issue for repositories with multiple packages like conda-raw2ometiff or conda-bftools is that the transformation works for one of the packages but the other package is completely overwritten with the same YAML.

Ha, the change I made to fix the writing introduced a bug for repositories with multiple meta files. Thanks for checking that.

[sbesson]

The BF conda file does not follow any of the conventions used in the other repositories, leading to the indentation style difference

This is unrelated to the style of the conda-bftools YAML file to the best of my knowledge, running the command against conda-omero-py set to the 5.9.1-0 tag gives me

diff --git a/meta.yaml b/meta.yaml
index e38e08d..787dca5 100644
--- a/meta.yaml
+++ b/meta.yaml
@@ -1,13 +1,14 @@
 {% set name = "omero-py" %}
-{% set version = "5.9.1" %}
+{% set version = 5.10.0 %}

 package:
   name: "{{ name|lower }}"
   version: "{{ version }}{{ environ.get('VERSION_SUFFIX', '') }}"

 source:
-  url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/{{ name }}-{{ version }}.tar.gz
-  sha256: 173b09411b45e4f728b54fd8203c246e2e114d0cc75fa083c5eecee76456cb21
+  url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/{{ name }}-{{ version
+    }}.tar.gz
+  sha256: b9e8d71ca56131367045e976c490e6c0913525e0331a504d639a6d10f771e42d

 build:
   # Can't use noarch because Windows requirements are different
@@ -16,46 +17,46 @@ build:
   number: 1
   script: "{{ PYTHON }} -m pip install . --no-deps --ignore-installed -vv "
   script_env:
-    - VERSION_SUFFIX
+  - VERSION_SUFFIX
   entry_points:
-    - omero = omero.main:main
+  - omero = omero.main:main

 requirements:
   host:
-    - pip
-    - python
+  - pip
+  - python
   run:
-    - appdirs
-    - future
-    - numpy
-    - pillow
-    - python
-    - pywin32  # [win]
-    - requests
-    - pyyaml
-    - zeroc-ice36-python
+  - appdirs
+  - future
+  - numpy
+  - pillow
+  - python
+  - pywin32    # [win]
+  - requests
+  - pyyaml
+  - zeroc-ice36-python

 test:
   commands:
-    - omero --help
-    - omero version
+  - omero --help
+  - omero version
   imports:
-    - omero
-    - omero.api
-    - omero.clients
-    - omero.cmd
-    - omero.constants
-    - omero.fs
-    - omero.gateway
-    - omero.grid
-    - omero.install
-    - omero.metadatastore
-    - omero.model
-    - omero.plugins
-    - omero.romio
-    - omero.sys
-    - omero.util
-    - omero_ext
+  - omero
+  - omero.api
+  - omero.clients
+  - omero.cmd
+  - omero.constants
+  - omero.fs
+  - omero.gateway
+  - omero.grid
+  - omero.install
+  - omero.metadatastore
+  - omero.model
+  - omero.plugins
+  - omero.romio
+  - omero.sys
+  - omero.util
+  - omero_ext

 about:
   home: https://www.openmicroscopy.org/
@@ -67,4 +68,4 @@ about:

 extra:
   recipe-maintainers:
-    - ome
+  - ome

[jburel]

aha, my file must have been already "correctly" formatted during my tests

[jburel]

The BF conda file does not follow any of the conventions used in the other repositories, leading to the indentation style difference

This is unrelated to the style of the conda-bftools YAML file to the best of my knowledge, running the command against conda-omero-py set to the 5.9.1-0 tag gives me

The package ruamel.yaml will force your output to be consistent with regards to indentation. So we will have to add some custom code to keep specific indentation for lines starting with -, # etc. I think it will be preferable to unify the files, set on the indentation (yaml.indent(mapping=2) by default)

[sbesson]

:+1: for YAML consistency across all files and hacing it enforced by the writing tool.

I was just surprised by the indentation of the block list item mostly because other places like the Ansible roles use yamllint which by default expect the 2 spaces indentation. Reading further I discovered that both forms (non indented and indented) seem to be valid according to the spec especially The “-”, “?” and “:” characters used to denote block collection entries are perceived by people to be part of the indentation.

[jburel]

All the points have now been fixed (I think). Comment https://github.com/ome/scc/pull/273#discussion_r716610015, the code could be removed after changing the logic in the yaml files across the various repositories

[sbesson]

Retested on the various repository

• for conda-omero-py, running git checkout 5.9.1-0 && scc bump-version-conda && git diff -w 5.10.0-0

  
  diff --git a/meta.yaml b/meta.yaml
  index 081bff7..787dca5 100644
  --- a/meta.yaml
  +++ b/meta.yaml
  @@ -1,12 +1,13 @@
  {% set name = "omero-py" %}
  -{% set version = "5.10.0" %}
  +{% set version = 5.10.0 %}
  
  package:
  name: "{{ name|lower }}"
  version: "{{ version }}{{ environ.get('VERSION_SUFFIX', '') }}"
  
  source:

• url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/{{ name }}-{{ version }}.tar.gz

• url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/{{ name }}-{{ version

• }}.tar.gz sha256: b9e8d71ca56131367045e976c490e6c0913525e0331a504d639a6d10f771e42d

  build:

• for conda-bftools, running git checkout 6.6.0 && /opt/ome/scc/venv/bin/scc bump-version-conda --repo ome/bioformats && git diff -w 6.6.0

diff --git a/bftools-libs/meta.yaml b/bftools-libs/meta.yaml
index 568c893..9b30479 100644
--- a/bftools-libs/meta.yaml
+++ b/bftools-libs/meta.yaml
@@ -1,6 +1,6 @@
 package:
   name: "bftools-libs"
-  version: "6.6.0"
+  version: "6.7.0"

 build:
   number: 0
@@ -8,7 +8,7 @@ build:

 source:
   url: https://downloads.openmicroscopy.org/bio-formats/{{ PKG_VERSION }}/artifacts/bftools.zip
-  sha256: d13553047ac552327107e029a077e32b48a7e0f7e028cf4c04d81539f2f56a21
+  sha256: 31a4200f7c5271bd11640743244b15ecb5166214

 extra:
   recipe-maintainers:
diff --git a/bftools/meta.yaml b/bftools/meta.yaml
index 1801170..8f18f73 100644
--- a/bftools/meta.yaml
+++ b/bftools/meta.yaml
@@ -1,13 +1,13 @@
 package:
   name: "bftools"
-  version: "6.6.0"
+  version: "6.7.0"

 build:
   number: 0

 source:
   url: https://downloads.openmicroscopy.org/bio-formats/{{ PKG_VERSION }}/artifacts/bftools.zip
-  sha256: d13553047ac552327107e029a077e32b48a7e0f7e028cf4c04d81539f2f56a21
+  sha256: 31a4200f7c5271bd11640743244b15ecb5166214

 requirements:
   run:

• for conda-raw2ometiff, running git checkout f15cdb1 && scc bump-version-conda --repo glencoesoftware/raw2ometiff leads to

  $ /opt/ome/scc/venv/bin/scc bump-version-conda --repo glencoesoftware/raw2ometiff
  Traceback (most recent call last):
  File "/opt/ome/scc/venv/lib/python3.8/site-packages/scc/main.py", line 64, in entry_point
  main("scc", items=[
  File "/opt/ome/scc/venv/lib/python3.8/site-packages/yaclifw/framework.py", line 188, in main
  ns.func(ns)
  File "/opt/ome/scc/venv/lib/python3.8/site-packages/scc/git.py", line 4090, in __call__
  sha = self.get_sha_from_github(jinja2[self.KEY_NAME],
  KeyError: 'name'

All non-whitespace diff are related to the formatting style discussed earlier. Two outstanding issue:

• for conda-bftools the SHA is still a SHA1 rather than a SHA256
• the command fails against conda-raw2ometiff

[jburel]

sorry forgot about the sha1 vs sha256

[jburel]

the lack of consistency between the repositories makes the handling of the various cases very tricky.

[sbesson]

the lack of consistency between the repositories makes the handling of the various cases very tricky.

I agree. As this PR is trying to handle all these cases, the two options I can think of would be:

• limit the scope to one recipe first and then work on the other use cases - but that likely escapes the hard work
• make the decisions to define the specification for our recipes and enforce consistency between the repositories first and then have this command expect to meet this layout

Is there a shortlist of the divergences that need to be reviewed?

[jburel]

Differences

• usage or not of jinja2 for setting variables
• when jinja2 is used: version or name or both are used
• url to determine the sha256 is formatted differently: version is used, name sometimes, other variable at time.

We may also want to add the repo i.e. ome/omero-py for example so we do not have to pass the parameter

The problems reported for conda-raw2ometiff and conda-bftools have now been fixed. Looking at the usage of quotes for jinja2

[jburel]

@sbesson the problems described in https://github.com/ome/scc/pull/273#issuecomment-930049266 should now be fixed

[jburel]

We already use the about/home for example https://github.com/ome/conda-bftools/blob/master/bftools/meta.yaml#L34 with a different purpose The dev_url could be used e.g. https://github.com/ome/bioformats

[jburel]

Actually I am going to push another commit using the about/dev_url since it is set in all the repositories. This means that we won't have to pass any parameter

[sbesson]

:+1: dev_url is definitely be a viable option - it is one of the accepted keys of the about section - as per https://docs.conda.io/projects/conda-build/en/latest/resources/package-spec.html#info-about-json and is linked from the Conda page https://anaconda.org/ome/omero-py/.

A separate issue is that these terms are fairly undefined. And at least I feel defining https://ww.openmicroscopy.org/ as the home of https://anaconda.org/ome/bioformats2raw is a bit of a stretch for instance.

[jburel]

With 75e59d2ef63b4939ed9670758f89f93b234a4c09, it is no longer necessary to use --repo. The parameter is no longer supported

[jburel]

A separate issue is that these terms are fairly undefined. And at least I feel defining https://ww.openmicroscopy.org/ as the home of https://anaconda.org/ome/bioformats2raw is a bit of a stretch for instance.

Yes, this should be changed.

[sbesson]

Briefly discussed with @jburel. The only addition I would suggest would be to have some logic to also reset the build.number to 0 when the version has been detected as different.

[jburel]

The fact that the changes were made even if the version was the same has now be fixed i.e. running scc on the latest conda-omero-py should stop at the version check

[jburel]

The first step:

• open PR for repositories not declaring version as a jinja2 variable.

[jburel]

No more commits. Reformatting will occur when the version is changed