MitoFish DB fails to download

[cement-head]

$ crabs db_download --source mitofish --output mitofish.fasta --keep_original yes

downloading sequences from the MitoFish database
Traceback (most recent call last):
File "/home/andor/miniconda3/envs/CRABS/bin/crabs", line 1372, in
main()
File "/home/andor/miniconda3/envs/CRABS/bin/crabs", line 1369, in main
args.func(args)
File "/home/andor/miniconda3/envs/CRABS/bin/crabs", line 96, in db_download
dl_file = mitofish_download(url)
File "/home/andor/miniconda3/envs/CRABS/lib/python3.6/site-packages/function/module_db_download.py", line 139, in mitofish_download
os.remove('complete_partial_mitogenomes.zip')
FileNotFoundError: [Errno 2] No such file or directory: 'complete_partial_mitogenomes.zip'

[gjeunen]

Hello @cement-head,

Can you please let me know what version you're working on? Please make sure to use the most up-to-date version crabs --version 0.1.8, which is available on GitHub. The Docker version should be fine as well, while the conda version is too out of date.

Best regards, Gert-Jan

[a4000]

I'm also having an issue downloading the MitoFish DB. I tried using the docker container 'quay.io/swordfish/crabs:0.1.4'. Is there a more up to data docker container?

The error with the docker container is:

downloading sequences from the MitoFish database Traceback (most recent call last): File "/usr/local/bin/crabs", line 1428, in main() File "/usr/local/bin/crabs", line 1425, in main args.func(args) File "/usr/local/bin/crabs", line 114, in db_download dl_file = mitofish_download(url) File "/usr/local/bin/function/module_db_download.py", line 140, in mitofish_download results = sp.run(['unzip', 'complete_partial_mitogenomes.zip'], stdout = sp.DEVNULL, stderr = sp.DEVNULL) File "/usr/lib/python3.8/subprocess.py", line 493, in run with Popen(*popenargs, **kwargs) as process: File "/usr/lib/python3.8/subprocess.py", line 858, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: 'unzip'

[gjeunen]

Hello @a4000,

Unfortunately, there would not be a more up-to-date Docker container at this stage. My apologies, I do not have personal experience with Docker. @hughcross, could you please have a look at the conda and Docker versions? They will likely need to be updated.

Best, Gert-Jan

[a4000]

I found the line causing the error "No such file or directory: 'unzip'". I sent a pull request that should fix the issue. I have experience with Docker, so I can help updating the docker container if needed.

[cement-head]

@gjeunen Opps - just saw that CONDA has not been updated. My bad - I'll try the git hub version.

[a4000]

@gjeunen I figured out why the docker container causes "FileNotFoundError: [Errno 2] No such file or directory: 'unzip'" when downloading the MitoFish database, it's because unzip isn't installed in the container. That's something to keep in mind for future versions of the container.

[gjeunen]

@a4000,

Thank you very much for the update! @hughcross, could you please add unzip to the Docker container?

Best, Gert-Jan

[billzt]

Hello everyone. I am maintaining the MitoFish DB. It should be noticed that only the newest GitHub version of CRABS works.

[hughcross]

Hi @cement-head, we now have a workable conda version that uses pip to install crabs. I am still testing it but if you want to check it out see the instructions on issue 32. Once we get it tested we will change the instructions on the main page. If you do try it, please let us know how it goes.

And thanks for the heads up, @a4000, I will add unzip to the next Docker version, which I will try to get out next week.

[ThomasLuypaert]

Hi all!

I was wondering whether there have been any updates to the docker image since the aforementioned issues with downloading the MitoFish database due to the absence of unzip?

I downloaded the docker image using singularity today, I am still getting the issue reported above:

Traceback (most recent call last):
  File "/usr/local/bin/crabs", line 1428, in <module>
    main()
  File "/usr/local/bin/crabs", line 1425, in main
    args.func(args)
  File "/usr/local/bin/crabs", line 114, in db_download
    dl_file = mitofish_download(url)
  File "/usr/local/bin/function/module_db_download.py", line 140, in mitofish_download
    results = sp.run(['unzip', 'complete_partial_mitogenomes.zip'], stdout = sp.DEVNULL, stderr = sp.DEVNULL)
  File "/usr/lib/python3.8/subprocess.py", line 493, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'unzip'

As a temporary workaround, I can manually unzip the file and extract the mito-all file, but I am not entirely sure how to emulate the post-processing steps by the mitofish_format function.

Other than that, the docker image is working great, thanks for this cool package!

[gjeunen]

Hello @ThomasLuypaert,

Thank you very much for your message. We'll try to sort this out asap.

@hughcross, could you please check if the Docker image is running the latest version of CRABS?

Thanks, Gert-Jan

[hughcross]

Hi,

You are right @ThomasLuypaert, the docker image is behind. I should be able to get it updated this weekend. Once I do, I will post back here. Thanks, Hugh

[ThomasLuypaert]

Thanks, I appreciate the quick response. Looking forward to trying out the package.

[hughcross]

Hi @ThomasLuypaert and @gjeunen,

Sorry for the delay. The updated Crabs docker image is ready. It is now quay.io/swordfish/crabs:0.1.7. I will change the documentation on the site soon, but just replace the 0.1.7 with the current instructions for now.

[ThomasLuypaert]

Great, thank you! I will give it a try next week.

[ThomasLuypaert]

Hi @gjeunen and @hughcross,

Just tried the updated docker images (version 0.1.7) as you suggested, however I am still getting the same error message as before when downloading the MitoFish data:

singularity exec /MYPATH/crabs_0.1.7.sif crabs db_download -s mitofish -o mitofish_download.fasta -e thomas.luypaert@nmbu.no

downloading sequences from the MitoFish database
complete_partial_mitogenomes.zip        100%[============================================================================>]  85.56M  1.05MB/s    in 84s
Traceback (most recent call last):
  File "/usr/local/bin/crabs", line 1462, in <module>
    main()
  File "/usr/local/bin/crabs", line 1459, in main
    args.func(args)
  File "/usr/local/bin/crabs", line 116, in db_download
    dl_file = mitofish_download(url)
  File "/usr/local/bin/function/module_db_download.py", line 140, in mitofish_download
    results = sp.run(['unzip', 'complete_partial_mitogenomes.zip'], stdout = sp.DEVNULL, stderr = sp.DEVNULL)
  File "/usr/lib/python3.8/subprocess.py", line 493, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'unzip'

Is unzip installed in the new docker image?

Any help is appreciated.

Cheers, Thomas

[hughcross]

Sorry, @ThomasLuypaert, I did not see this before. I did not test the Mitofish download with the updated crabs docker image. I will have to fix that. I will try to get that done soon.

Thanks,

Hugh

[gjeunen]

Hello @cement-head and @ThomasLuypaert,

This issue should now be resolved with the latest version of CRABS (version 1.0.0). It is currently only available via cloning the GitHub repo, but we hope to update the conda package and Docker container shortly.

Best wishes, Gert-Jan