conda / conda-build

Commands and tools for building conda packages
https://docs.conda.io/projects/conda-build/
Other
381 stars 420 forks source link

Cannot use Jinja2 for loop in outputs section #2815

Closed StefansM closed 1 year ago

StefansM commented 6 years ago

It is not possible to use a Jinja2 for loop in the outputs section of meta.yaml due to an incompatibility between rendered and unrendered output.

Actual Behavior

This can be reproduced using this minimal example:

{% set subtypes = [1, 2, 3] %}

package:
  name: dummy

source:
  path: dummy.txt

outputs:
{% for subtype in subtypes %}
  - name: foo-{{ subtype }}
  - name: bar-{{ subtype }}
{% endfor %}

Running conda build ., I get the following error:

Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Traceback (most recent call last):
  File "/data/sm2708/conda/bin/conda-build", line 11, in <module>
    sys.exit(main())
  File "/data/sm2708/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 420, in main
    execute(sys.argv[1:])
  File "/data/sm2708/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 411, in execute
    verify=args.verify)
  File "/data/sm2708/conda/lib/python3.6/site-packages/conda_build/api.py", line 199, in build
    notest=notest, need_source_download=need_source_download, variants=variants)
  File "/data/sm2708/conda/lib/python3.6/site-packages/conda_build/build.py", line 2105, in build_tree
    notest=notest,
  File "/data/sm2708/conda/lib/python3.6/site-packages/conda_build/build.py", line 1167, in build
    output_metas = expand_outputs([(m, need_source_download, need_reparse_in_env)])
  File "/data/sm2708/conda/lib/python3.6/site-packages/conda_build/render.py", line 586, in expand_outputs
    for (output_dict, m) in _m.get_output_metadata_set(permit_unsatisfiable_variants=False):
  File "/data/sm2708/conda/lib/python3.6/site-packages/conda_build/metadata.py", line 1830, in get_output_metadata_set
    for k in out_metadata.get_used_vars()}))] = out, out_metadata
  File "/data/sm2708/conda/lib/python3.6/site-packages/conda_build/metadata.py", line 1935, in get_used_vars
    meta_yaml_reqs = self._get_used_vars_meta_yaml(force_top_level=force_top_level)
  File "/data/sm2708/conda/lib/python3.6/site-packages/conda_build/metadata.py", line 1958, in _get_used_vars_meta_yaml
    recipe_text = self.extract_single_output_text(self.name(), apply_selectors=False)
  File "/data/sm2708/conda/lib/python3.6/site-packages/conda_build/metadata.py", line 1562, in extract_single_output_text
    output = output_matches[output_index] if output_matches else ''
IndexError: list index out of range

The issue seems to be is because of the discrepancy in the number of rendered and "unrendered" outputs. The rendered outputs are parsed from the YAML after applying templates, but the "unrendered" outputs are found via regular expression matching, which obviously won't find outputs added by Jinja2.

Expected Behavior

I don't understand why conda build cares about the "unrendered" YAML. Requiring the unrendered YAML to rigidly map to the rendered YAML throws away a lot of the useful features of Jinja2. In general, parsing a YAML file with regexes ignores lots of features of YAML: the output_re in metadata.py wouldn't cope with flow-style outputs or YAML anchors (see #2809 for another example of regex parsing).

Steps to Reproduce

Run conda build . with the minimal meta.yaml file above.

Output of conda info
     active environment : None
       user config file : /bmm/home/sm2708/.condarc
 populated config files : /bmm/home/sm2708/.condarc
          conda version : 4.5.0
    conda-build version : 3.8.0
         python version : 3.6.4.final.0
       base environment : /data/sm2708/conda  (writable)
           channel URLs : file:///bmm/phyreengine/software/conda/linux-64
                          file:///bmm/phyreengine/software/conda/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/linux-64
                          https://repo.anaconda.com/pkgs/pro/noarch
                          https://conda.anaconda.org/bioconda/linux-64
                          https://conda.anaconda.org/bioconda/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /data/sm2708/conda/pkgs
                          /bmm/home/sm2708/.conda/pkgs
       envs directories : /data/sm2708/conda/envs
                          /bmm/home/sm2708/.conda/envs
               platform : linux-64
             user-agent : conda/4.5.0 requests/2.18.4 CPython/3.6.4 Linux/3.10.0-514.26.2.el7.x86_64 ol/7.3 glibc/2.17
                UID:GID : 1487:500
             netrc file : None
           offline mode : False
github-actions[bot] commented 1 year ago

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

  1. Verify that you can still reproduce the issue at hand
  2. Comment that the issue is still reproducible and include:
    • What OS and version you reproduced the issue on
    • What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!