last_common_ancestor.py import module find_betweens error

[qiyunzhu]

This error occurs when I followed the README.md to run with the example dataset. The full screen output is:

(shogun) -bash-4.1$ shogun_bt2_lca -i ./mock_communities -b ./annotated/bt2/test.hmp_species
Traceback (most recent call last):
  File "/home/me/Programs/Miniconda2/envs/shogun/bin/shogun_bt2_lca", line 11, in <module>
    load_entry_point('shogun==0.0.1.dev0', 'console_scripts', 'shogun_bt2_lca')()
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/pkg_resources/__init__.py", line 561, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2631, in load_entry_point
    return ep.load()
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2291, in load
    return self.resolve()
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2297, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/shogun/scripts/shogun_bt2_lca.py", line 7, in <module>
    from shogun.utils.last_common_ancestor import build_lca_map
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/shogun/utils/__init__.py", line 1, in <module>
    from .last_common_ancestor import build_lca_map
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/shogun/utils/last_common_ancestor.py", line 1, in <module>
    from find_betweens import verify_make_dir
ModuleNotFoundError: No module named 'find_betweens'

The computer system is:

(shogun) -bash-4.1$ uname -a
Linux some.addr.edu 2.6.32-504.23.4.el6.x86_64 #1 SMP Tue Jun 9 20:57:37 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Thank you!

[bhillmann]

Unfortunately, this was an artifact of some refactoring I did on the library. I pushed the update that should fix your issue. Please reinstall the library. In the future, we should have CI.

pip install git+https://github.com/knights-lab/SHOGUN.git --no-cache-dir --upgrade

[qiyunzhu]

@bhillmann Thanks! I tried the updated version. Now here is a new error:

(shogun) -bash-4.1$ shogun_bt2_lca -i ./mock_communities -b ./annotated/bt2/test.hmp_species
Traceback (most recent call last):
  File "/home/me/Programs/Miniconda2/envs/shogun/bin/shogun_bt2_lca", line 11, in <module>
    load_entry_point('shogun==0.0.1.dev0', 'console_scripts', 'shogun_bt2_lca')()
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/shogun/scripts/shogun_bt2_lca.py", line 24, in shogun_bt2_lca
    verify_make_dir(output)
NameError: name 'verify_make_dir' is not defined

Thank you!

[bhillmann]

More refactoring bugs. Should be fixed. Are you interested in using the Bowtie2 module? We recommend the use of UTree.

[qiyunzhu]

Hi @bhillmann I installed the update and now the problem is gone! Thanks! I also tried the UTree option, but here is the error message:

(shogun) -bash-4.1$ shogun_utree_db -i ./test.hmp_species.fna -x '>, '
Traceback (most recent call last):
  File "/home/me/Programs/Miniconda2/envs/shogun/bin/shogun_utree_db", line 11, in <module>
    load_entry_point('shogun==0.0.1.dev0', 'console_scripts', 'shogun_utree_db')()
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/shogun/scripts/shogun_utree_db.py", line 71, in shogun_utree_db
    print(utree_build(outf_fasta, outf_map, path_uncompressed_tree, threads=threads))
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/shogun/wrappers/utree.py", line 15, in utree_build
    return run_command(cmd, shell=shell)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/ninja_utils/utils/run_command.py", line 20, in run_command
    cwd=os.getcwd(),
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/subprocess.py", line 403, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/subprocess.py", line 1326, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'utree-build'

[bhillmann]

Are you able to find UTree on your path?

Can you run this commmand?

which utree-build

[qiyunzhu]

@bhillmann No it is not present in my system... How can I get it?

[bhillmann]

You can get the binaries from the release page. Once you download the binary for your system, you can install it by adding it to your path. SHOGUN searches for UTree on your path.

https://github.com/knights-lab/UTree/releases

Full instructions for installation of SHOGUN UTree are available in the BugBase release.

https://github.com/knights-lab/BugBase#bugbase-compatible-otu-tables

[qiyunzhu]

Hi @bhillmann @danknights I would like to let you know that everything worked! Both Bowtie2 and UTree options worked without problem on my side. For UTree I vitnessed a significant performance increase versus Bowtie2 (maybe some dozens of folds) and database size decrease (1:10). Now I am looking at the results.

[qiyunzhu]

Hi @bhillmann I managed to run shogun_utree_lca and got reasonable result. However, later I tried shogun_bt2_lca but encountered with an error. It seemed that the Bowtie2 search was completed successfully, but the subsequent taxonomic assignment was problematic. Below are the error messages.

shogun_bt2_lcaraceback (most recent call last):
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/networkx/classes/digraph.py", line 749, in successors_iter
    return iter(self.succ[n])
KeyError: 1345697

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/me/Programs/Miniconda2/envs/shogun/bin/shogun_bt2_lca", line 11, in <module>
    load_entry_point('shogun==0.0.1.dev0', 'console_scripts', 'shogun_bt2_lca')()
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/shogun/scripts/shogun_bt2_lca.py", line 49, in shogun_bt2_lca
    lca_map = build_lca_map(sam_file, lambda x: int(find_between(x, begin, end)), tree)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/shogun/utils/last_common_ancestor.py", line 19, in build_lca_map
    lca_map[qname] = tree.lowest_common_ancestor(ncbi_tid, current_ncbi_tid)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__ (cytoolz/functoolz.c:3996)
  File "cytoolz/functoolz.pyx", line 461, in cytoolz.functoolz.memoize.__call__ (cytoolz/functoolz.c:8748)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/dojo/taxonomy/ncbi_tree.py", line 177, in lowest_common_ancestor
    path_p = nx.shortest_path(self.tree, p, 1)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/networkx/algorithms/shortest_paths/generic.py", line 136, in shortest_path
    paths=nx.bidirectional_shortest_path(G,source,target)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/networkx/algorithms/shortest_paths/unweighted.py", line 136, in bidirectional_shortest_path
    results=_bidirectional_pred_succ(G,source,target)
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/networkx/algorithms/shortest_paths/unweighted.py", line 186, in _bidirectional_pred_succ
    for w in Gsucc(v):
  File "/home/me/Programs/Miniconda2/envs/shogun/lib/python3.6/site-packages/networkx/classes/digraph.py", line 751, in successors_iter
    raise NetworkXError("The node %s is not in the digraph."%(n,))
networkx.exception.NetworkXError: The node 1345697 is not in the digraph.

Do you have any ideas? Thank you!

[bhillmann]

That ncbi taxonomy identifier is not in the NCBI taxonomy that you have downloaded. A null pointer issue if you will. I should be able to patch this relatively quickly.

[bhillmann]

The fix is in. Re-checkout the helper library DOJO and you should be set.

# DOJO
pip install git+https://github.com/knights-lab/DOJO.git --no-cache-dir --upgrade

[qiyunzhu]

@bhillmann Thank you! I will try and let you know the result.