Replace 'sudo docker' with 'udocker'

[sebastian-luna-valero]

Hi,

The changes proposed on this PR should help us get rid of the sudo docker pre-requisite.

To try this out, please follow these steps:

# go to specific working dir
cd /path/to/working-dir

# get 'run.sh' from 'udocker' branch
curl -O https://raw.githubusercontent.com/AMIGA-IAA/hcg-16/udocker/run.sh

# replace all 'hcg-16/master' with 'hcg-16/udocker' in run.sh
sed -i 's/hcg-16\/master/hcg-16\/udocker/g' run.sh

# test it
bash run.sh

udocker downloads container images to $HOME/.udocker so it's important to check usage with:

du -sh $HOME/.udocker
du -sh working-dir

@jonesmg before merging, could you please run this branch on your end and let me know whether the output is exactly the same?

Best regards, Sebastian

[jonesmg]

Hi Sebastian,

I tried to test this (just on amiga-ui, as I shouldn't need sudo access for udocker). However, it failed before it even got to udocker. I think it failed to activate the conda environment.

# run.sh log | amiga-ui | Fri Feb 26 19:56:32 CET 2021 |  Download conda environment...  
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   269  100   269    0     0    772      0 --:--:-- --:--:-- --:--:--   770
# run.sh log | amiga-ui | Fri Feb 26 19:56:33 CET 2021 |  Install hcg-16 environment...  
run.sh: line 109: mamba: command not found
# run.sh log | amiga-ui | Fri Feb 26 19:56:33 CET 2021 |  Download pipeline...  
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3849  100  3849    0     0  11843      0 --:--:-- --:--:-- --:--:-- 11843
# run.sh log | amiga-ui | Fri Feb 26 19:56:33 CET 2021 |  Download pipeline config...  
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   460  100   460    0     0   1356      0 --:--:-- --:--:-- --:--:--  1356
# run.sh log | amiga-ui | Fri Feb 26 19:56:33 CET 2021 |  Run pipeline...  
Traceback (most recent call last):
  File "pipeline.py", line 3, in <module>
    from ruffus import *
ImportError: No module named ruffus

So I think this current issue is before it ever gets to udocker and is probably the result of the switch to mamba?

Cheers,

Mike

[sebastian-luna-valero]

Hi Mike,

That's strange, the steps above work for me no only on amiga-ui but also on my personal laptop, and 4 virtual machines on the SPSRC private cloud with Centos 7, 8, Ubuntu 18.04 and 20.04.

Have you configured your $HOME/.bashrc on amiga-ui to load an existing conda configuration automatically? If so, it might be worth disabling that temporarily and re-run the steps above from scratch on a new terminal.

On the other hand, you could also sent me a private email with the path on amiga-ui where you downloaded and modified run.sh so I can compare with mine.

Best regards, Sebastian

[jonesmg]

Hi Sebastian,

Yes that was the issue. I commented out the existing conda installation in my .bashrc and it's running now. I'll let you know how the final products look.

This seems like an issue though (maybe we should open a new issue for it) as many people will have existing conda installations configured in their .bashrc files (or equivalents). At the very least we need to add to the documentation to tell people to comment out any existing installation before running the pipeline, but ideally we'd find a solution that avoids the need for it.

Cheers,

Mike

[jonesmg]

Output all looks good to me. I'll go ahead with the merge.

[sebastian-luna-valero]

Hi Mike,

Great, I am glad to know manually disabling the previous conda installation solved the issue! I would like to modify the run.sh script to automatically detect and disable any inherited conda installation. I think it's possible, but I would need to test it first.

We also need to update the instructions in the README.md to reflect that sudo docker is no longer a pre-requisite. Also, did you check the disk space in $HOME/.udocker? I think we also need to warn users about it, since it is not a negligible amount.

I will work on updating the run.sh script, could you please update README.md?

Best regards, Sebastian

[jonesmg]

Hi

I just updated the readme. I think the pre-requisites section was the only thing that needed changing. I checked and udocker used 8.8 GB additional space when I ran it, so I double the recommended available HDD space to 20 GB.

Cheers,

Mike

[sebastian-luna-valero]

Thanks, Mike!

Follow up in https://github.com/AMIGA-IAA/hcg-16/pull/9