search

bioconda / bioconda-recipes

Conda recipes for the bioconda channel.
https://bioconda.github.io
MIT License
1.65k stars 3.28k forks source link

trouble in Azure DevOps test #47547

Open yanyew opened 6 months ago

yanyew commented 6 months ago

I have been experiencing this error continuously during the automatic testing process since submitting the update, but I have not been able to find the cause. It seems that there is an issue with the azure pipelines? It showed "[Errno 28] No space left on device" when TEST START after BUILD SUCCESS Here is the link to the test in Azure DevOps https://dev.azure.com/bioconda/bioconda-recipes/_build/results?buildId=57320&view=logs&j=e14e69ff-a0ae-55c4-b71d-229b239cfb2f&t=4dddc55b-10bc-50fc-ac68-7ff899b7a030&l=2079

76f79b3bb9c5fa0e2ed2965ef2fbb54 9db977f7391738fdf24be76dcbf8324

martin-g commented 6 months ago

Can you add df -h before/after the build/test phases ? Do you download some big data package/file in your tests ? Which PR is failing with this error ?

yanyew commented 6 months ago

Can you add df -h before/after the build/test phases ? Do you download some big data package/file in your tests ? Which PR is failing with this error ?

I didn't download big data file in the test. And this PR have been completed successfully in the previous PR. This error occurred this time in https://github.com/bioconda/bioconda-recipes/pull/47492

TobyBaril commented 6 months ago

I am experiencing the same thing with #47498 , and this has also been an issue continuously with RepeatMasker and RepeatModeler too

martin-g commented 6 months ago

Did anyone of you test df -h in the build/test scripts ?

TobyBaril commented 6 months ago

yes, added in the build script, is it worth adding to the test as well?

conda_log.txt

martin-g commented 6 months ago

You can help yourself by adding more df -h in your build.sh and see when the used disk space grows.

E.g. I would bet on https://github.com/bioconda/bioconda-recipes/pull/47498/files#diff-d357f966c7237a539f146074ad4c151fc5fcc6e2514524512aea9a92b6bd692dR20. Here you clone a Git repository. I am pretty sure you don't need all its history, so you can use git clone --depth=1 ...

Once you are ready with it you can delete the local copy of this repo, at line https://github.com/bioconda/bioconda-recipes/pull/47498/files#diff-d357f966c7237a539f146074ad4c151fc5fcc6e2514524512aea9a92b6bd692dR24 Same for https://github.com/bioconda/bioconda-recipes/pull/47498/files#diff-d357f966c7237a539f146074ad4c151fc5fcc6e2514524512aea9a92b6bd692dR59 - I guess you can restore some disk space by deleting the .tar.gz and its uncompressed folder once it is no more needed.

TobyBaril commented 6 months ago

Great thanks, I've made those changes.

I do wonder if this is an issue of RepeatMasker and RepeatModeler dependencies though, as RepeatMasker requires a relatively large database, and the error seems to arise after pulling the container from quay.io. RepeatModeler sources the LTRretriever databases as far as I know - this was also not an issue prior to a small update to the meta.yaml - has something changed recently in the DevOps system?

2024-04-30T09:12:04.3454955Z 09:11:35 BIOCONDA INFO (ERR) [Apr 30 09:11:35] DEBU Pull Image [quay.io/bioconda/base-glibc-busybox-bash:3.0]
2024-04-30T09:12:04.3455747Z 09:11:45 BIOCONDA INFO (ERR) [Apr 30 09:11:45] DEBU Packing failed due to open build/dist/conda-meta/font-ttf-ubuntu-0.83-h77eed37_1/licenses/LICENCE.txt: permission denied
2024-04-30T09:12:04.3456525Z 09:11:47 BIOCONDA INFO (ERR) [Apr 30 09:11:47] WARN Local execution errorred with error [open build/dist/conda-meta/font-ttf-ubuntu-0.83-h77eed37_1/licenses/LICENCE.txt: permission denied], retrying with remote execution
2024-04-30T09:12:04.3457112Z 09:11:47 BIOCONDA INFO (ERR) [Apr 30 09:11:47] DEBU Creating container [step-f224588f60]
2024-04-30T09:12:04.3457921Z 09:11:47 BIOCONDA INFO (ERR) [Apr 30 09:11:47] DEBU Image [involucro/tool:latest] not present, pulling it
2024-04-30T09:12:04.3458380Z 09:11:47 BIOCONDA INFO (ERR) [Apr 30 09:11:47] DEBU Pull Image [involucro/tool:latest]
2024-04-30T09:12:04.3458864Z 09:11:49 BIOCONDA INFO (ERR) [Apr 30 09:11:49] DEBU Created container [43bfdb6c19ed step-f224588f60], starting it
2024-04-30T09:12:04.3459403Z 09:11:49 BIOCONDA INFO (ERR) [Apr 30 09:11:49] DEBU Container [43bfdb6c19ed step-f224588f60] started, waiting for completion
2024-04-30T09:12:04.3460056Z 09:11:49 BIOCONDA INFO (ERR) [Apr 30 09:11:49] SERR [Apr 30 09:11:49] DEBU Creating container [step-89d93840f9]
2024-04-30T09:13:04.1952681Z .09:12:32 BIOCONDA INFO (ERR) [Apr 30 09:12:32] SERR [Apr 30 09:12:32] DEBU Packing succeeded
2024-04-30T09:14:04.1956392Z .09:13:27 BIOCONDA INFO (ERR) [Apr 30 09:13:27] SERR [Apr 30 09:13:27] ERRO Task processing failed: API error (500): {"message":"write /usr/local/share/LTR_retriever/database/alluniRefprexp082813: no space left on device"}
martin-g commented 6 months ago

I am not aware of any changes in the AzureCI config, but there might be something ... You have added several new dependencies - https://github.com/bioconda/bioconda-recipes/pull/47498/files#diff-c7b20a81f85b48bd9d321f136d14ff451fe825ca2cc1eee9b478a4ade0a2cc35R49-R54 . Maybe they take some space ?!

You can compare the contents of https://anaconda.org/bioconda/earlgrey/4.1.1/download/linux-64/earlgrey-4.1.1-h4ac6f70_0.tar.bz2 > info/about.json against the build logs and see whether something big changed between the builds Unless you pin a dependency to a specific version, like - mydependency 1.2.3 any following build may use a newer version of this dependency with its new transitive dependencies.

TobyBaril commented 6 months ago

Great, thanks for the information! I'll look into this and see if I can figure something out to successfully merge into Bioconda

TobyBaril commented 6 months ago

Have made the suggested changes and still running into the same issues. Not sure of a way around this to get the new version of the package into bioconda. Currently hosting on my own conda channel in the interim.

https://dev.azure.com/bioconda/bioconda-recipes/_build/results?buildId=57459&view=results

yanyew commented 6 months ago

Did anyone of you test df -h in the build/test scripts ?

Sorry for the late reply. I have test 'df -h' in the test scripts, and found that the disk space occupied reached 97% after the build was completed. Which seems to be wired because the packages used in this build shouldn't occupie so much space. The link to azure test is https://dev.azure.com/bioconda/bioconda-recipes/_build/results?buildId=57667 image

aliciaaevans commented 2 weeks ago

We've now switched to GitHub Actions for PR checks, so I wonder if that makes any improvement to these disk space issues. Merge master into the PR to trigger the new checks.