search

bcbio / bcbio-nextgen

Validated, scalable, community developed variant calling, RNA-seq and small RNA analysis
https://bcbio-nextgen.readthedocs.io
MIT License
994 stars 354 forks source link

v1.1.9 install is hanging at conda environment solve #3040

Closed mjsteinbaugh closed 4 years ago

mjsteinbaugh commented 4 years ago

I'm seeing the bcbio-nextgen 1.1.9 installer get stuck trying to figure out the conda environment recipe, when these flags are used:

    --upgrade="stable"
    --datatarget="rnaseq"
    --datatarget="variation"
    --nodata
    --isolate 
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working...

I'll re-run this with just the rnaseq datatarget and report back.

roryk commented 4 years ago

Thanks, this isn't really anything we can handle unfortunately. If you let it sit for a while it will figure it out, but it can take hours to eventually finish.

mjsteinbaugh commented 4 years ago

Yeah it's hanging with only --datatarget="rnaseq" too

roryk commented 4 years ago

Yeah, it's the tools that run into this problem, should be independent of the data.

mjsteinbaugh commented 4 years ago

mamba to the rescue?

roryk commented 4 years ago

Could be, there have been some other solutions proposed too. For now we're just waiting to see what shakes out as something that is faster and works stably.

mjsteinbaugh commented 4 years ago

Following up, I let the installer run overnight and the solver does eventually complete and run through.

erinijapranckeviciene commented 4 years ago

Dear colleagues,

I am installing v1.1.9 on a fresh Ubuntu 18 LTS system with a simple command

python3 bcbio_nextgen_install.py ~/bcbio/v1.1.9 --tooldir=~/bcbio/v1.1.9 --genomes GRCh37 --aligners bwa

and am experiencing the install hanging specifically at this step:

# Installing into conda environment python2: bismark=0.22.1, cpat, cutadapt=1.16, dkfz-bias-filter, gemini, gvcf-regions, hap.py, hisat2, htseq=0.9.1, lumpy-sv, macs2, manta, metasv, mirge, optitype, peddy, phylowgs, platypus-variant, sentieon, smcounter2, smoove, strelka, svtools, svtyper, theta2, tophat, vawk, vcf2db Collecting package metadata (current_repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Collecting package metadata (repodata.json): done Solving environment: \

This step was taking more than 12 hours, so I cancelled. I have retried several times. Every time it is stuck at the same step. Is there an approximate estimate of how long it actually should take? I might be wrong, but It does not seem right for installer to be stuck for such a long time.

Many thanks in advance!

mjsteinbaugh commented 4 years ago

@erinijapranckeviciene Try letting it run for a full day (~24 hours) and see if the solver completes. It took longer than 12 hours for me last time on an Azure VM.

erinijapranckeviciene commented 4 years ago

@mjsteinbaugh That is very helpful. Many thanks! It means that next time to proceed on a fresh system with v1.1.9 the cost-efficient way is to just transfer the existing install.

mjsteinbaugh commented 4 years ago

@erinijapranckeviciene If the install path is the same on a different machine, rsync will work. If the installation path is different, you may run into problems.

roryk commented 4 years ago

Thanks, sorry about that. There isn't anything we can do ATM to fix this, it is an issue with complex conda environments. They've been working on improving the solver, but we're mostly at their mercy for it completing.

erinijapranckeviciene commented 4 years ago

Yes indeed, it is fully understandable, NP. Just an update - in my case the long time taking step completed in about 15 hours.