search

daviesdrew / variantcalling

0 stars 0 forks source link

Setup CI #13

Open peterk87 opened 4 years ago

peterk87 commented 4 years ago

Setup CI via Github Actions, Travis CI, CircleCI or some other service.

Small test dataset run with test profile using Conda profile.

daviesdrew commented 4 years ago

Currently working on this using Travis CI.

Ran into a small error with building my Dockerfile and solved using this thread https://github.com/docker/for-linux/issues/563

daviesdrew commented 4 years ago

Okay I am pretty close to having this working correctly. Took a little longer than I thought. Should be able to finish it tomorrow, if not tonight.

daviesdrew commented 4 years ago

The build keeps failing when attempting to run nextflow in the scripts part of the travis.yml file.

It exits with status 127 and says the nextflow command is not found.

On my local machine I can get the pipeline to run in the Dockerfile image until it reaches the first process bbduk. But that exits with status 1.

peterk87 commented 4 years ago

Are you installing Nextflow in your CI setup?

https://github.com/daviesdrew/variantcalling/blob/e823d4c1e6b3dc44149cca981a60c5e4497d4eb5/.github/workflows/ci.yml#L18-L21

daviesdrew commented 4 years ago

I am going to try doing that now.

I have figured out that nextflow is being installed into the TRAVIS_BUILD_DIR during the travis build but it doesn't seem to be in the user path so I am going to take the sudo mv instruction you suggested and see if that makes nextflow accessible.

One thing I tried was to put nextflow into the conda environment activated by docker but that doesn't seem to work.

daviesdrew commented 4 years ago

Thanks Peter.

The sudo mv instruction made the difference.

nextflow is now found.

daviesdrew commented 4 years ago

The pipeline initiates in the container but it cant seem to find bbduk in REMOVE_PHIX even though I have bbmap dependency in the conda environment which executes in the container.

Will investigate more Monday as I should spend the rest of my time this week on MITACS.

I am sure it is just a config problem I haven't noticed yet.

peterk87 commented 4 years ago

Have you tried running the bbduk.sh command using the Docker image?

$ docker run <name of docker image> bbduk.sh

I would try getting the built Docker image into Docker Hub and pulling from there for CI.

daviesdrew commented 4 years ago

ddbuk.sh can be found in the environment and when I run "which bbduk.sh" in the conda environment it points to the correct conda environment.

Maybe using docker hub will provide a simple solution. I will look into using dockerhub.