PertuyF
commented
5 years ago You may find interesting things in conda-forge documentation. Not sure it will solve, but it could help.
Closed nfahlgren closed 5 years ago
PertuyF
commented
5 years ago You may find interesting things in conda-forge documentation. Not sure it will solve, but it could help.
nfahlgren
commented
5 years ago Thanks @PertuyF! That is a really useful link. It seems to me like the libGL stuff should be taken care of in the OpenCV conda-forge package, and indeed they have the configuration discussed in the conda-forge documentation in their recipe: https://github.com/conda-forge/opencv-feedstock/tree/master/recipe.
I tried various forms of adding yum_dependencies.txt with mesa-libGL or mesa-libGL-devel and separately also adding the libGL packages to the build requirements, but none of those things changed the test result of circleci build. I always get the error:
ImportError: libGL.so.1: cannot open shared object file: No such file or directory
Here's a link to my configuration files with the last updates I tried: https://github.com/bioconda/bioconda-recipes/tree/update-plantcv-3.2.0/recipes/plantcv
PertuyF
commented
5 years ago From what I understand, you need to add {{ cdt('mesa-libgl-devel') }} # [linux] to your build requirements. Then likely set a yum_requirements.txt for mesa-libGL so that the tests will be able to access this library.
I think this is a workaround to allow conda packages to depend on system-level libraries. It is done for openCV at build time, and likely it should be done for building any package that depends on openCV as well, as these libs aren't included in the docker image.
But note I'm not an expert on this part, it is just based on deduction and experimentation.
nfahlgren
commented
5 years ago Thanks @PertuyF, the last iteration I tried looked like this but didn't work:
meta.yml:
{% set name = "plantcv" %}
{% set version = "3.2.0" %}
{% set file_ext = "tar.gz" %}
{% set hash_type = "sha256" %}
{% set hash_value = "a2800a9749d66f6376a8e05939979ed185daf5aa63cbbaefef99c26da4ca0c1e" %}
package:
name: '{{ name|lower }}'
version: '{{ version }}'
source:
url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/{{ name }}-{{ version }}.{{ file_ext }}
'{{ hash_type }}': '{{ hash_value }}'
build:
noarch: python
number: 0
requirements:
build:
- {{ cdt('mesa-libgl-devel') }} # [linux]
- {{ cdt('mesa-dri-drivers') }} # [linux]
- {{ cdt('libselinux') }} # [linux]
- {{ cdt('libxdamage') }} # [linux]
- {{ cdt('libxxf86vm') }} # [linux]
host:
- python
- pip
run:
- python
- matplotlib >=1.5
- numpy >=1.11
- pandas
- python-dateutil
- scipy
- scikit-image
- plotnine
- opencv <4
test:
imports:
- plantcv
- plantcv.learn
- plantcv.plantcv
- plantcv.plantcv.roi
- plantcv.plantcv.threshold
- plantcv.plantcv.transform
about:
home: https://plantcv.danforthcenter.org
license: MIT License
license_family: MIT
license_file: 'LICENSE'
summary: An image processing package for plant phenotyping.
description: "PlantCV is a Python package for building modular image analysis workflows for plant phenotyping."
doc_url: 'https://plantcv.readthedocs.io'
dev_url: 'https://github.com/danforthcenter/plantcv'
extra:
recipe-maintainers: 'PlantCV team'
identifiers:
- doi:10.1016/j.molp.2015.06.005
- doi:10.1109/WNYIPW.2016.7904790
- doi:10.7717/peerj.4088
- doi:10.7717/peerj.5727yum_requirements.txt:
mesa-libGL
mesa-dri-drivers
libselinux
libXdamage
libXxf86vmLast error is because libjasper cannot be found, so likely the mesa-libGL issue is resolved?
nfahlgren
commented
5 years ago Something I missed in the documentation is using the extended container: https://bioconda.github.io/troubleshooting.html#using-the-extended-image. A specific use case is when you need libGL.so.1. Adding
extra:
container:
extended-base: truedid not fix a local circleci build for me but I'm thinking that's because I would need to pull the extended Docker container. I went ahead and submitted a pull request #14002 to test in CircleCI directly.
nfahlgren
commented
5 years ago I am totally stuck at this point. Local testing is not working for me at all. To get away from potentially clashing environments on my MacBook I created a clean AWS instance to work in. I installed git and docker, cloned the bioconda-recipes repo and updated the plantcv recipe (just bumping the version number only). I then follow the testing instructions at https://bioconda.github.io/contribute-a-recipe.html#test-locally
This pulls the container successfully:
docker pull bioconda/bioconda-utils-build-env
This installs the circleci client correctly:
sudo curl -fLSs https://circle.ci/cli | bash
Running circleci build in the bioconda-recipes fails:
Docker image digest: sha256:0f93072c4564ac0992ed5fe4f1c33f4c39ad58c56ea811a7dc59f02be99d2594
====>> Spin up Environment
Build-agent version 1.0.8916-7902072c (2019-03-12T16:48:37+0000)
Starting container bioconda/bioconda-utils-build-env
using image bioconda/bioconda-utils-build-env@sha256:c93c972a7ed72fa0bc203fa901ad43262398aae67e35ef25c5a62f6f37776a83
Using build environment variables
BASH_ENV=/tmp/.bash_env-localbuild-1552414108
CI=true
CIRCLECI=true
CIRCLE_BRANCH=master
CIRCLE_BUILD_NUM=
CIRCLE_JOB=build
CIRCLE_NODE_INDEX=0
CIRCLE_NODE_TOTAL=1
CIRCLE_REPOSITORY_URL=https://github.com/bioconda/bioconda-recipes.git
CIRCLE_SHA1=aea75b2f5896f34af33ad50de5421e15ec7d555f
CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1552414108
CIRCLE_WORKING_DIRECTORY=~/project
====>> Checkout code
#!/bin/bash -eo pipefail
mkdir -p /root/project && cd /tmp/_circleci_local_build_repo && git ls-files -z | xargs -0 tar -c | tar -x -C /root/project && cp -a /tmp/_circleci_local_build_repo/.git /root/project
/bin/bash: git: command not found
tar: Cowardly refusing to create an empty archive
Try `tar --help' or `tar --usage' for more information.
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
Error: Exited with code 2
Step failed
Error: runner failed
{"Runner":true,"level":"error","msg":"runner failed","task-id":"localbuild-1552414108","time":"2019-03-12T18:08:30Z"}
Task failed
Error: task failedIf I try the boostrap method instead:
Creating the miniconda environment works:
./bootstrap.py /tmp/miniconda
source ~/.config/bioconda/activateLinting works:
bioconda-utils lint recipes config.yml --git-range master
Mulled tests fail because the tmp-bioconda-builder cannot be created?
bioconda-utils build recipes config.yml --docker --mulled-test --git-range master
18:12:01 BIOCONDA INFO Recipes newly unblacklisted:
18:12:01 BIOCONDA INFO Recipes modified according to git: plantcv
18:12:26 BIOCONDA INFO DOCKER: Building image "tmp-bioconda-builder" from /tmp/tmpqq3_8e9q
..............................18:51:41 BIOCONDA ERROR COMMAND FAILED: docker build --network host -t tmp-bioconda-builder /tmp/tmpqq3_8e9q
18:51:41 BIOCONDA ERROR STDOUT+STDERR:
Sending build context to Docker daemon 3.584kB
Step 1/2 : FROM bioconda/bioconda-utils-build-env:2019-02-01
---> 15419aba89cc
Step 2/2 : RUN /opt/conda/bin/conda install -y conda=4.5.11 conda-build=3.15.1
---> Running in 3144594c1580
Solving environment: ...working... The command '/bin/sh -c /opt/conda/bin/conda install -y conda=4.5.11 conda-build=3.15.1' returned a non-zero code: 137
18:51:41 BIOCONDA ERROR DOCKER FAILED: Error building docker container tmp-bioconda-builder.
Traceback (most recent call last):
File "/tmp/miniconda/miniconda/bin/bioconda-utils", line 10, in <module>
sys.exit(main())
File "/tmp/miniconda/miniconda/lib/python3.6/site-packages/bioconda_utils/cli.py", line 840, in main
bioconductor_skeleton, clean_cran_skeleton, autobump
File "/tmp/miniconda/miniconda/lib/python3.6/site-packages/argh/dispatching.py", line 328, in dispatch_commands
dispatch(parser, *args, **kwargs)
File "/tmp/miniconda/miniconda/lib/python3.6/site-packages/argh/dispatching.py", line 174, in dispatch
for line in lines:
File "/tmp/miniconda/miniconda/lib/python3.6/site-packages/argh/dispatching.py", line 277, in _execute_command
for line in result:
File "/tmp/miniconda/miniconda/lib/python3.6/site-packages/argh/dispatching.py", line 260, in _call
result = function(*positional, **keywords)
File "<boltons.funcutils.FunctionBuilder-2>", line 2, in build
File "/tmp/miniconda/miniconda/lib/python3.6/site-packages/bioconda_utils/cli.py", line 41, in wrapper
func(*args, **kwargs)
File "/tmp/miniconda/miniconda/lib/python3.6/site-packages/bioconda_utils/cli.py", line 430, in build
keep_image=keep_image,
File "/tmp/miniconda/miniconda/lib/python3.6/site-packages/bioconda_utils/docker_utils.py", line 349, in __init__
self._build_image(image_build_dir)
File "/tmp/miniconda/miniconda/lib/python3.6/site-packages/bioconda_utils/docker_utils.py", line 430, in _build_image
raise e
File "/tmp/miniconda/miniconda/lib/python3.6/site-packages/bioconda_utils/docker_utils.py", line 425, in _build_image
p = utils.run(cmd, mask=False)
File "/tmp/miniconda/miniconda/lib/python3.6/site-packages/bioconda_utils/utils.py", line 427, in run
raise e
File "/tmp/miniconda/miniconda/lib/python3.6/site-packages/bioconda_utils/utils.py", line 407, in run
**kwargs)
File "/tmp/miniconda/miniconda/lib/python3.6/subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['docker', 'build', '--network', 'host', '-t', 'tmp-bioconda-builder', '/tmp/tmpqq3_8e9q']' returned non-zero exit status 137.
18:51:43 BIOCONDA ERROR COMMAND FAILED: docker rmi tmp-bioconda-builder
18:51:43 BIOCONDA ERROR STDOUT+STDERR:
Error: No such image: tmp-bioconda-builder
Exception ignored in: <bound method RecipeBuilder.__del__ of <bioconda_utils.docker_utils.RecipeBuilder object at 0x7ff25335f400>>
Traceback (most recent call last):
File "/tmp/miniconda/miniconda/lib/python3.6/site-packages/bioconda_utils/docker_utils.py", line 358, in __del__
self.cleanup()
File "/tmp/miniconda/miniconda/lib/python3.6/site-packages/bioconda_utils/docker_utils.py", line 510, in cleanup
utils.run(cmd, mask=False)
File "/tmp/miniconda/miniconda/lib/python3.6/site-packages/bioconda_utils/utils.py", line 427, in run
raise e
File "/tmp/miniconda/miniconda/lib/python3.6/site-packages/bioconda_utils/utils.py", line 407, in run
**kwargs)
File "/tmp/miniconda/miniconda/lib/python3.6/subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['docker', 'rmi', 'tmp-bioconda-builder']' returned non-zero exit status 1.I think I got around the issue with the new PlantCV build in #14002. Looks like conda was installing OpenCV 3.1.x and I think that was causing the error with the missing libjasper.so.1. Maybe related to the issue described in the OpenCV feedstock (https://github.com/conda-forge/opencv-feedstock/issues/147).
I pinned OpenCV to 3.4.4 in our recipe and that seems to have fixed it. I'm testing removing some other configuration options that I added that I think are not needed now.
Local testing with the circleci client and the mulled-build tests in Docker are still not working for me but I will close this anyway as those issues are not related to the original issue.
Hi all, I'm working on updating the
plantcvrecipe and I'm hitting an error I can't quite figure out. Steps I have taken:bioconda/bioconda-recipescloneplantcvrecipe files and committed changes to the new branchdocker pull bioconda/bioconda-utils-build-envsudo circleci update installcircleci buildWith the recipe conda solves the following environment:
The build returns the following error:
So this looks like OpenCV is looking for libGL but doesn't find it. From some older issues (e.g. #489) it sounds like libGL is meant to be provided at the system level and not as a conda package. I'm not sure if there's something I can specify in the recipe, if this has to do with
mesa-libGLU-develin the build environment, if it's an issue with OpenCV inconda-forge, or something else.Any thoughts? Much appreciated!