search

ReproNim / neurodocker

Generate custom Docker and Singularity images, and minimize existing containers
https://www.repronim.org/neurodocker/
Apache License 2.0
325 stars 97 forks source link

no longer produces `/neurodocker/startup.sh` in docker recipes, in effect ignores neurodocker/templates/_header.yaml? #620

Closed yarikoptic closed 3 weeks ago

yarikoptic commented 1 month ago

Discovered while working on

although there is neurodocker/templates/_header.yaml stating that 

# Instructions to be run in the beginning of every Dockerfile generated by
# Neurodocker.

such promise is not fulfilled:

$ docker run -i --rm repronim/neurodocker:1.0.0 generate docker --base-image ubuntu:20.04 --pkg-manager apt
# Generated by Neurodocker and Reproenv.

FROM ubuntu:20.04

# Save specification to JSON.
RUN printf '{ \
  "pkg_manager": "apt", \
  "existing_users": [ \
    "root" \
  ], \
  "instructions": [ \
    { \
      "name": "from_", \
      "kwds": { \
        "base_image": "ubuntu:20.04" \
      } \
    } \
  ] \
}' > /.reproenv.json
# End saving to specification to JSON.
yarikoptic commented 1 month ago

archaeology expedition notes:

in 0.7.0 we had --base not --base-image and that worked ```shell ❯ neurodocker generate docker --pkg-manager apt --base neurodebian [NEURODOCKER 2024-06-12 10:37:09,907 WARNING]: A newer version (1.0.0) of ReproNim/neurodocker is available. You are using 0.7.0 # Generated by: Neurodocker version 0.7.0 # Latest release: Neurodocker version 1.0.0 # Timestamp: 2024/06/12 14:37:09 UTC # # Thank you for using Neurodocker. If you discover any issues # or ways to improve this software, please submit an issue or # pull request on our GitHub repository: # # https://github.com/ReproNim/neurodocker FROM neurodebian USER root ARG DEBIAN_FRONTEND="noninteractive" ENV LANG="en_US.UTF-8" \ LC_ALL="en_US.UTF-8" \ ND_ENTRYPOINT="/neurodocker/startup.sh" RUN export ND_ENTRYPOINT="/neurodocker/startup.sh" \ && apt-get update -qq \ && apt-get install -y -q --no-install-recommends \ apt-utils \ bzip2 \ ca-certificates \ curl \ locales \ unzip \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* \ && sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen \ && dpkg-reconfigure --frontend=noninteractive locales \ && update-locale LANG="en_US.UTF-8" \ && chmod 777 /opt && chmod a+s /opt \ && mkdir -p /neurodocker \ && if [ ! -f "$ND_ENTRYPOINT" ]; then \ echo '#!/usr/bin/env bash' >> "$ND_ENTRYPOINT" \ && echo 'set -e' >> "$ND_ENTRYPOINT" \ && echo 'export USER="${USER:=`whoami`}"' >> "$ND_ENTRYPOINT" \ && echo 'if [ -n "$1" ]; then "$@"; else /usr/bin/env bash; fi' >> "$ND_ENTRYPOINT"; \ fi \ && chmod -R 777 /neurodocker && chmod a+s /neurodocker ENTRYPOINT ["/neurodocker/startup.sh"] RUN echo '{ \ \n "pkg_manager": "apt", \ \n "instructions": [ \ \n [ \ \n "base", \ \n "neurodebian" \ \n ] \ \n ] \ \n}' > /neurodocker/neurodocker_specs.json ```
in 0.8.0 neurodocker switched to --base-image and lost that addition: ```shell ❯ neurodocker generate docker --pkg-manager apt --base-image neurodebian # Generated by Neurodocker and Reproenv. FROM neurodebian # Save specification to JSON. RUN printf '{ \ "pkg_manager": "apt", \ "existing_users": [ \ "root" \ ], \ "instructions": [ \ { \ "name": "from_", \ "kwds": { \ "base_image": "neurodebian" \ } \ } \ ] \ }' > /.reproenv.json # End saving to specification to JSON. ```

unfrortunately there were no changelog whatsoever back in those dark times, so hard to tell the intent etc.

trying to use --_header KEY=VALUE revealed that it leads to reading that _header template ```shell ❯ neurodocker generate docker --pkg-manager apt --base-image neurodebian --_header KEY=VALUE Traceback (most recent call last): File "/home/yoh/proj/repronim/neurodocker/neurodocker/reproenv/renderers.py", line 212, in from_dict renderer.add_registered_template(method_or_template, **kwds) File "/home/yoh/proj/repronim/neurodocker/neurodocker/reproenv/renderers.py", line 364, in add_registered_template self.add_template(template=template, method=method) File "/home/yoh/proj/repronim/neurodocker/neurodocker/reproenv/renderers.py", line 263, in add_template template_method.validate_kwds() File "/home/yoh/proj/repronim/neurodocker/neurodocker/reproenv/template.py", line 145, in validate_kwds raise TemplateKeywordArgumentError( neurodocker.reproenv.exceptions.TemplateKeywordArgumentError: Keyword argument provided is not specified in template: 'KEY'. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/yoh/proj/repronim/neurodocker/venv-3/bin/neurodocker", line 8, in sys.exit(cli()) ^^^^^ File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/decorators.py", line 21, in new_func return f(get_current_context(), *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/yoh/proj/repronim/neurodocker/neurodocker/cli/generate.py", line 464, in docker _base_generate( File "/home/yoh/proj/repronim/neurodocker/neurodocker/cli/generate.py", line 441, in _base_generate r = renderer.from_dict(renderer_dict) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/yoh/proj/repronim/neurodocker/neurodocker/reproenv/renderers.py", line 214, in from_dict raise RendererError( neurodocker.reproenv.exceptions.RendererError: Error on template '_header'. Please see above for more information. ```
apparently neurodocker adds that odd --name_header automagically as per each template, so since it is named _header, even if there is no parameters -- it adds it ```shell def _get_params_for_registered_templates() -> list[click.Parameter]: """Return list of click parameters for registered templates.""" params: list[click.Parameter] = [] names_tmpls = list(registered_templates_items()) names_tmpls.sort(key=lambda r: r[0]) # sort by name for name, tmpl in names_tmpls: hlp = _create_help_for_template(Template(tmpl)) param = OptionEatAll( [f"--{name.lower()}"], type=KeyValuePair(), multiple=True, help=hlp ) params.append(param) return params ```
and trying --_header method=source without anu ```shell ❯ neurodocker generate docker --pkg-manager apt --base-image neurodebian --_header method=source Traceback (most recent call last): File "/home/yoh/proj/repronim/neurodocker/neurodocker/reproenv/renderers.py", line 76, in _render_string_from_template source = tmpl.render(template=template) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/jinja2/environment.py", line 1304, in render self.environment.handle_exception() File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/jinja2/environment.py", line 939, in handle_exception raise rewrite_traceback_stack(source=source) File "