search

picrust / picrust2

Code, unit tests, and tutorials for running PICRUSt2
GNU General Public License v3.0
328 stars 104 forks source link

Installation Picrust2 #26

Closed yakshiUPR closed 6 years ago

yakshiUPR commented 6 years ago

HI there,

We were following the instructions to install PICRUSt2 (installing it on a Mac with 16Gb of ram). First we got the Clon Repository. Then we started to install the pre-requisites and this was when problems started to happen.

While installing PaPaRa we had issues because we didn't know that we had to install boost first. After installing boost, the installation kept failing.

papara_nt-2.5 yakshiortiz$ sh build_papara2.sh In file included from papara.cpp:30: In file included from ./papara.h:47: In file included from ./pvec.h:41: ivy_mike/src/ivymike/tree_traversal_utils.h:30:10: fatal error: 'boost/tr1/unordered_set.hpp' file not found

include <boost/tr1/unordered_set.hpp>

     ^

1 error generated.

We managed to understand that the problem was because boost latest version removed a folder named tr1 (see here: https://github.com/ros/rospack/pull/80/commits/6fcee1b0d2b1c225c39a5820981166b99f0d8750). We changed the code as it was shown in the link provided. After that, the installation was successful.

While installing epa-ng, another error appeared. First we had to install cmake. Then an error with Tiny_Tree persisted.

[ 97%] Building CXX object src/CMakeFiles/epa_module.dir/tree/Tiny_Tree.cpp.o In file included from /Users/yakshiortiz/picrust2/placement_tools/epa-ng/src/tree/Tiny_Tree.cpp:1: In file included from /Users/yakshiortiz/picrust2/placement_tools/epa-ng/src/tree/Tiny_Tree.hpp:13: /Users/yakshiortiz/picrust2/placement_tools/epa-ng/src/core/Lookup_Store.hpp:166:27: error: implicit instantiation of undefined template 'std::1::array<unsigned long, 128>' std::array<size_t, 128> char_toposish; ^ /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/tuple:116:65: note: template is declared here template <class _Tp, size_t _Size> struct _LIBCPP_TYPE_VIS_ONLY array; ^ 1 error generated. make[3]: [src/CMakeFiles/epa_module.dir/tree/Tiny_Tree.cpp.o] Error 1 make[2]: [src/CMakeFiles/epa_module.dir/all] Error 2 make[1]: [all] Error 2 make: [run_make] Error 2

To solve this, we modified the Tiny_Tree files emacs src/tree/Tiny_Tree.cpp src/tree/Tiny_Tree.hpp to add a line that said Include <Array>.

Finally, while installing GAPPA, we noticed a similar problem.

[ 68%] Building CXX object CMakeFiles/gappa.dir/src/options/file_input.cpp.o /Users/yakshiortiz/picrust2/placement_tools/gappa/src/options/file_input.cpp:38:10: fatal error: 'linux/limits.h' file not found

include <linux/limits.h>

     ^

1 error generated. make[3]: [CMakeFiles/gappa.dir/src/options/file_input.cpp.o] Error 1 make[2]: [CMakeFiles/gappa.dir/all] Error 2 make[1]: [all] Error 2 make: [build] Error 2

To solve this problem, we ran this code: vi +38 src/options/file_input.cppand there we converted this line

include <linux/limits.h>

into a comment. This appeared to solve the problems. But after "successfully" creating PICRUSt environment with conda,

Installing collected packages: PICRUSt2 Running setup.py develop for PICRUSt2 Successfully installed PICRUSt2 You are using pip version 9.0.3, however version 10.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command.

we ran the self test (pytest). This test says that 13 tests failed and 15 passed. I am leaving the error output below. Its very long, but it may help solve the problem. Please let me know if any of you have any insights into what might be happening.

Thanks,

Yakshi Ortiz

(picrust2-dev) ursula:picrust2 yakshiortiz$ pytest
============================= test session starts ==============================
platform darwin -- Python 3.6.5, pytest-3.6.2, py-1.5.4, pluggy-0.6.0
rootdir: /Users/yakshiortiz/picrust2, inifile:
plugins: cov-2.5.1
collected 28 items                                                             

tests/test_hsp.py FFFFFFF                                                [ 25%]
tests/test_metagenome_pipeline.py .......                                [ 50%]
tests/test_place_seqs.py .FFFF.                                          [ 71%]
tests/test_run_minpath.py ..                                             [ 78%]
tests/test_util.py ....                                                  [ 92%]
tests/test_workflow.py FF                                                [100%]

=================================== FAILURES ===================================
________________ castor_hsp_workflow_tests.test_emp_prob_simple ________________

self = <tests.test_hsp.castor_hsp_workflow_tests testMethod=test_emp_prob_simple>

    def test_emp_prob_simple(self):

        predict_out, ci_out = castor_hsp_workflow(tree_path=in_tree1,
                                                 trait_table_path=in_traits1,
                                                 hsp_method="emp_prob",
>                                                ran_seed=10)

tests/test_hsp.py:73: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
picrust2/wrap_hsp.py:65: in castor_hsp_workflow
    for trait_in in file_subsets)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:949: in __call__
    if self.dispatch_one_batch(iterator):
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:791: in dispatch_one_batch
    self._dispatch(tasks)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:748: in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/_parallel_backends.py:183: in apply_async
    result = ImmediateResult(func)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/_parallel_backends.py:543: in __init__
    self.results = batch()
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:224: in __call__
    for func, args, kwargs in self.items]
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:224: in <listcomp>
    for func, args, kwargs in self.items]
picrust2/wrap_hsp.py:126: in castor_hsp_wrapper
    system_call_check(hsp_cmd)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cmd = ['Rscript', '/Users/yakshiortiz/picrust2/picrust2/Rscripts/castor_hsp.R', '/Users/yakshiortiz/picrust2/tests/test_data/hsp/tree.tre', '/var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmp34b217n5/subset_tab_0', 'emp_prob', 'FALSE', ...]
print_out = False, stdout = None, stderr = None

    def system_call_check(cmd, print_out=False, stdout=None, stderr=None):
        """Run system command and throw and error if return is not 0. Input command
        can be a list containing the command or a string."""

        # Print command out if option set.
        if print_out:
            if type(cmd) is list:
                print(" ".join(cmd))
            else:
                print(cmd)

        # Convert command to list if input as string.
        if type(cmd) is str:
            cmd = cmd.split()

        return_value = call(cmd, stdout=stdout, stderr=stderr)

        # Exit with error if command did not finish successfully.
        if return_value != 0:
>           raise SystemExit("Error running this command:\n" + " ".join(cmd))
E           SystemExit: Error running this command:
E           Rscript /Users/yakshiortiz/picrust2/picrust2/Rscripts/castor_hsp.R /Users/yakshiortiz/picrust2/tests/test_data/hsp/tree.tre /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmp34b217n5/subset_tab_0 emp_prob FALSE FALSE /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmp7jma9xbe/predicted_counts.txt /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmp7jma9xbe/predicted_ci.txt 10

picrust2/util.py:161: SystemExit
----------------------------- Captured stderr call -----------------------------

 *** caught segfault ***
address 0x18, cause 'memory not mapped'

Traceback:
 1: dyn.load(file, DLLpath = DLLpath, ...)
 2: library.dynam(lib, package, package.lib)
 3: loadNamespace(package, lib.loc)
 4: doTryCatch(return(expr), name, parentenv, handler)
 5: tryCatchOne(expr, names, parentenv, handlers[[1L]])
 6: tryCatchList(expr, classes, parentenv, handlers)
 7: tryCatch({    attr(package, "LibPath") <- which.lib.loc    ns <- loadNamespace(package, lib.loc)    env <- attachNamespace(ns, pos = pos, deps)}, error = function(e) {    P <- if (!is.null(cc <- conditionCall(e)))         paste(" in", deparse(cc)[1L])    else ""    msg <- gettextf("package or namespace load failed for %s%s:\n %s",         sQuote(package), P, conditionMessage(e))    if (logical.return)         message(paste("Error:", msg), domain = NA)    else stop(msg, call. = FALSE, domain = NA)})
 8: library(pkg, character.only = TRUE, logical.return = TRUE, lib.loc = lib.loc,     quietly = quietly)
 9: .getRequiredPackages2(pkgInfo, quietly = quietly)
10: library(castor, quietly = TRUE)
An irrecoverable exception occurred. R is aborting now ...
_____________________ castor_hsp_workflow_tests.test_mp_ci _____________________

self = <tests.test_hsp.castor_hsp_workflow_tests testMethod=test_mp_ci>

    def test_mp_ci(self):
        '''Test that MP confidence intervals calculated correctly.'''
        predict_out, ci_out = castor_hsp_workflow(tree_path=in_tree1,
                                                 trait_table_path=in_traits1,
                                                 hsp_method="mp",
                                                 ran_seed=10,
>                                                calc_ci=True)

tests/test_hsp.py:126: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
picrust2/wrap_hsp.py:65: in castor_hsp_workflow
    for trait_in in file_subsets)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:949: in __call__
    if self.dispatch_one_batch(iterator):
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:791: in dispatch_one_batch
    self._dispatch(tasks)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:748: in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/_parallel_backends.py:183: in apply_async
    result = ImmediateResult(func)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/_parallel_backends.py:543: in __init__
    self.results = batch()
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:224: in __call__
    for func, args, kwargs in self.items]
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:224: in <listcomp>
    for func, args, kwargs in self.items]
picrust2/wrap_hsp.py:126: in castor_hsp_wrapper
    system_call_check(hsp_cmd)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cmd = ['Rscript', '/Users/yakshiortiz/picrust2/picrust2/Rscripts/castor_hsp.R', '/Users/yakshiortiz/picrust2/tests/test_data/hsp/tree.tre', '/var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmp1fk2yvpl/subset_tab_0', 'mp', 'TRUE', ...]
print_out = False, stdout = None, stderr = None

    def system_call_check(cmd, print_out=False, stdout=None, stderr=None):
        """Run system command and throw and error if return is not 0. Input command
        can be a list containing the command or a string."""

        # Print command out if option set.
        if print_out:
            if type(cmd) is list:
                print(" ".join(cmd))
            else:
                print(cmd)

        # Convert command to list if input as string.
        if type(cmd) is str:
            cmd = cmd.split()

        return_value = call(cmd, stdout=stdout, stderr=stderr)

        # Exit with error if command did not finish successfully.
        if return_value != 0:
>           raise SystemExit("Error running this command:\n" + " ".join(cmd))
E           SystemExit: Error running this command:
E           Rscript /Users/yakshiortiz/picrust2/picrust2/Rscripts/castor_hsp.R /Users/yakshiortiz/picrust2/tests/test_data/hsp/tree.tre /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmp1fk2yvpl/subset_tab_0 mp TRUE FALSE /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpurw96obt/predicted_counts.txt /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpurw96obt/predicted_ci.txt 10

picrust2/util.py:161: SystemExit
----------------------------- Captured stderr call -----------------------------

 *** caught segfault ***
address 0x18, cause 'memory not mapped'

Traceback:
 1: dyn.load(file, DLLpath = DLLpath, ...)
 2: library.dynam(lib, package, package.lib)
 3: loadNamespace(package, lib.loc)
 4: doTryCatch(return(expr), name, parentenv, handler)
 5: tryCatchOne(expr, names, parentenv, handlers[[1L]])
 6: tryCatchList(expr, classes, parentenv, handlers)
 7: tryCatch({    attr(package, "LibPath") <- which.lib.loc    ns <- loadNamespace(package, lib.loc)    env <- attachNamespace(ns, pos = pos, deps)}, error = function(e) {    P <- if (!is.null(cc <- conditionCall(e)))         paste(" in", deparse(cc)[1L])    else ""    msg <- gettextf("package or namespace load failed for %s%s:\n %s",         sQuote(package), P, conditionMessage(e))    if (logical.return)         message(paste("Error:", msg), domain = NA)    else stop(msg, call. = FALSE, domain = NA)})
 8: library(pkg, character.only = TRUE, logical.return = TRUE, lib.loc = lib.loc,     quietly = quietly)
 9: .getRequiredPackages2(pkgInfo, quietly = quietly)
10: library(castor, quietly = TRUE)
An irrecoverable exception occurred. R is aborting now ...
___________________ castor_hsp_workflow_tests.test_mp_simple ___________________

self = <tests.test_hsp.castor_hsp_workflow_tests testMethod=test_mp_simple>

    def test_mp_simple(self):

        predict_out, ci_out = castor_hsp_workflow(tree_path=in_tree1,
                                                 trait_table_path=in_traits1,
                                                 hsp_method="mp",
>                                                ran_seed=10)

tests/test_hsp.py:60: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
picrust2/wrap_hsp.py:65: in castor_hsp_workflow
    for trait_in in file_subsets)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:949: in __call__
    if self.dispatch_one_batch(iterator):
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:791: in dispatch_one_batch
    self._dispatch(tasks)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:748: in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/_parallel_backends.py:183: in apply_async
    result = ImmediateResult(func)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/_parallel_backends.py:543: in __init__
    self.results = batch()
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:224: in __call__
    for func, args, kwargs in self.items]
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:224: in <listcomp>
    for func, args, kwargs in self.items]
picrust2/wrap_hsp.py:126: in castor_hsp_wrapper
    system_call_check(hsp_cmd)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cmd = ['Rscript', '/Users/yakshiortiz/picrust2/picrust2/Rscripts/castor_hsp.R', '/Users/yakshiortiz/picrust2/tests/test_data/hsp/tree.tre', '/var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpt2imkmr_/subset_tab_0', 'mp', 'FALSE', ...]
print_out = False, stdout = None, stderr = None

    def system_call_check(cmd, print_out=False, stdout=None, stderr=None):
        """Run system command and throw and error if return is not 0. Input command
        can be a list containing the command or a string."""

        # Print command out if option set.
        if print_out:
            if type(cmd) is list:
                print(" ".join(cmd))
            else:
                print(cmd)

        # Convert command to list if input as string.
        if type(cmd) is str:
            cmd = cmd.split()

        return_value = call(cmd, stdout=stdout, stderr=stderr)

        # Exit with error if command did not finish successfully.
        if return_value != 0:
>           raise SystemExit("Error running this command:\n" + " ".join(cmd))
E           SystemExit: Error running this command:
E           Rscript /Users/yakshiortiz/picrust2/picrust2/Rscripts/castor_hsp.R /Users/yakshiortiz/picrust2/tests/test_data/hsp/tree.tre /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpt2imkmr_/subset_tab_0 mp FALSE FALSE /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpvtbmcd5n/predicted_counts.txt /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpvtbmcd5n/predicted_ci.txt 10

picrust2/util.py:161: SystemExit
----------------------------- Captured stderr call -----------------------------

 *** caught segfault ***
address 0x18, cause 'memory not mapped'

Traceback:
 1: dyn.load(file, DLLpath = DLLpath, ...)
 2: library.dynam(lib, package, package.lib)
 3: loadNamespace(package, lib.loc)
 4: doTryCatch(return(expr), name, parentenv, handler)
 5: tryCatchOne(expr, names, parentenv, handlers[[1L]])
 6: tryCatchList(expr, classes, parentenv, handlers)
 7: tryCatch({    attr(package, "LibPath") <- which.lib.loc    ns <- loadNamespace(package, lib.loc)    env <- attachNamespace(ns, pos = pos, deps)}, error = function(e) {    P <- if (!is.null(cc <- conditionCall(e)))         paste(" in", deparse(cc)[1L])    else ""    msg <- gettextf("package or namespace load failed for %s%s:\n %s",         sQuote(package), P, conditionMessage(e))    if (logical.return)         message(paste("Error:", msg), domain = NA)    else stop(msg, call. = FALSE, domain = NA)})
 8: library(pkg, character.only = TRUE, logical.return = TRUE, lib.loc = lib.loc,     quietly = quietly)
 9: .getRequiredPackages2(pkgInfo, quietly = quietly)
10: library(castor, quietly = TRUE)
An irrecoverable exception occurred. R is aborting now ...
_____________________ castor_hsp_workflow_tests.test_nsti ______________________

self = <tests.test_hsp.castor_hsp_workflow_tests testMethod=test_nsti>

    def test_nsti(self):
        '''Test that calculated NSTI values match expected.'''

        in_traits1_df = pd.read_csv(in_traits1, sep="\t",
                                    index_col="assembly",
                                    dtype={'assembly' : str})

        nsti_out = castor_nsti(tree_path=in_tree1,
>                              known_tips=in_traits1_df.index.values)

tests/test_hsp.py:142: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
picrust2/wrap_hsp.py:169: in castor_nsti
    nsti_tmp_out]))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cmd = ['Rscript', '/Users/yakshiortiz/picrust2/picrust2/Rscripts/castor_nsti.R', '/Users/yakshiortiz/picrust2/tests/test_dat..._xk300000gr/T/tmpdsgk6pm2/known_tips.txt', '/var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpdsgk6pm2/nsti_out.txt']
print_out = False, stdout = None, stderr = None

    def system_call_check(cmd, print_out=False, stdout=None, stderr=None):
        """Run system command and throw and error if return is not 0. Input command
        can be a list containing the command or a string."""

        # Print command out if option set.
        if print_out:
            if type(cmd) is list:
                print(" ".join(cmd))
            else:
                print(cmd)

        # Convert command to list if input as string.
        if type(cmd) is str:
            cmd = cmd.split()

        return_value = call(cmd, stdout=stdout, stderr=stderr)

        # Exit with error if command did not finish successfully.
        if return_value != 0:
>           raise SystemExit("Error running this command:\n" + " ".join(cmd))
E           SystemExit: Error running this command:
E           Rscript /Users/yakshiortiz/picrust2/picrust2/Rscripts/castor_nsti.R /Users/yakshiortiz/picrust2/tests/test_data/hsp/tree.tre /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpdsgk6pm2/known_tips.txt /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpdsgk6pm2/nsti_out.txt

picrust2/util.py:161: SystemExit
----------------------------- Captured stderr call -----------------------------

 *** caught segfault ***
address 0x18, cause 'memory not mapped'

Traceback:
 1: dyn.load(file, DLLpath = DLLpath, ...)
 2: library.dynam(lib, package, package.lib)
 3: loadNamespace(package, lib.loc)
 4: doTryCatch(return(expr), name, parentenv, handler)
 5: tryCatchOne(expr, names, parentenv, handlers[[1L]])
 6: tryCatchList(expr, classes, parentenv, handlers)
 7: tryCatch({    attr(package, "LibPath") <- which.lib.loc    ns <- loadNamespace(package, lib.loc)    env <- attachNamespace(ns, pos = pos, deps)}, error = function(e) {    P <- if (!is.null(cc <- conditionCall(e)))         paste(" in", deparse(cc)[1L])    else ""    msg <- gettextf("package or namespace load failed for %s%s:\n %s",         sQuote(package), P, conditionMessage(e))    if (logical.return)         message(paste("Error:", msg), domain = NA)    else stop(msg, call. = FALSE, domain = NA)})
 8: library(pkg, character.only = TRUE, logical.return = TRUE, lib.loc = lib.loc,     quietly = quietly)
 9: .getRequiredPackages2(pkgInfo, quietly = quietly)
10: library(castor, quietly = TRUE)
An irrecoverable exception occurred. R is aborting now ...
__________________ castor_hsp_workflow_tests.test_pic_simple ___________________

self = <tests.test_hsp.castor_hsp_workflow_tests testMethod=test_pic_simple>

    def test_pic_simple(self):

        predict_out, ci_out = castor_hsp_workflow(tree_path=in_tree1,
                                                 trait_table_path=in_traits1,
                                                 hsp_method="pic",
>                                                ran_seed=10)

tests/test_hsp.py:86: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
picrust2/wrap_hsp.py:65: in castor_hsp_workflow
    for trait_in in file_subsets)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:949: in __call__
    if self.dispatch_one_batch(iterator):
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:791: in dispatch_one_batch
    self._dispatch(tasks)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:748: in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/_parallel_backends.py:183: in apply_async
    result = ImmediateResult(func)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/_parallel_backends.py:543: in __init__
    self.results = batch()
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:224: in __call__
    for func, args, kwargs in self.items]
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:224: in <listcomp>
    for func, args, kwargs in self.items]
picrust2/wrap_hsp.py:126: in castor_hsp_wrapper
    system_call_check(hsp_cmd)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cmd = ['Rscript', '/Users/yakshiortiz/picrust2/picrust2/Rscripts/castor_hsp.R', '/Users/yakshiortiz/picrust2/tests/test_data/hsp/tree.tre', '/var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpp5w68hjg/subset_tab_0', 'pic', 'FALSE', ...]
print_out = False, stdout = None, stderr = None

    def system_call_check(cmd, print_out=False, stdout=None, stderr=None):
        """Run system command and throw and error if return is not 0. Input command
        can be a list containing the command or a string."""

        # Print command out if option set.
        if print_out:
            if type(cmd) is list:
                print(" ".join(cmd))
            else:
                print(cmd)

        # Convert command to list if input as string.
        if type(cmd) is str:
            cmd = cmd.split()

        return_value = call(cmd, stdout=stdout, stderr=stderr)

        # Exit with error if command did not finish successfully.
        if return_value != 0:
>           raise SystemExit("Error running this command:\n" + " ".join(cmd))
E           SystemExit: Error running this command:
E           Rscript /Users/yakshiortiz/picrust2/picrust2/Rscripts/castor_hsp.R /Users/yakshiortiz/picrust2/tests/test_data/hsp/tree.tre /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpp5w68hjg/subset_tab_0 pic FALSE FALSE /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpgy9_zgq2/predicted_counts.txt /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpgy9_zgq2/predicted_ci.txt 10

picrust2/util.py:161: SystemExit
----------------------------- Captured stderr call -----------------------------

 *** caught segfault ***
address 0x18, cause 'memory not mapped'

Traceback:
 1: dyn.load(file, DLLpath = DLLpath, ...)
 2: library.dynam(lib, package, package.lib)
 3: loadNamespace(package, lib.loc)
 4: doTryCatch(return(expr), name, parentenv, handler)
 5: tryCatchOne(expr, names, parentenv, handlers[[1L]])
 6: tryCatchList(expr, classes, parentenv, handlers)
 7: tryCatch({    attr(package, "LibPath") <- which.lib.loc    ns <- loadNamespace(package, lib.loc)    env <- attachNamespace(ns, pos = pos, deps)}, error = function(e) {    P <- if (!is.null(cc <- conditionCall(e)))         paste(" in", deparse(cc)[1L])    else ""    msg <- gettextf("package or namespace load failed for %s%s:\n %s",         sQuote(package), P, conditionMessage(e))    if (logical.return)         message(paste("Error:", msg), domain = NA)    else stop(msg, call. = FALSE, domain = NA)})
 8: library(pkg, character.only = TRUE, logical.return = TRUE, lib.loc = lib.loc,     quietly = quietly)
 9: .getRequiredPackages2(pkgInfo, quietly = quietly)
10: library(castor, quietly = TRUE)
An irrecoverable exception occurred. R is aborting now ...
__________________ castor_hsp_workflow_tests.test_scp_simple ___________________

self = <tests.test_hsp.castor_hsp_workflow_tests testMethod=test_scp_simple>

    def test_scp_simple(self):

        predict_out, ci_out = castor_hsp_workflow(tree_path=in_tree1,
                                                 trait_table_path=in_traits1,
                                                 hsp_method="scp",
>                                                ran_seed=10)

tests/test_hsp.py:99: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
picrust2/wrap_hsp.py:65: in castor_hsp_workflow
    for trait_in in file_subsets)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:949: in __call__
    if self.dispatch_one_batch(iterator):
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:791: in dispatch_one_batch
    self._dispatch(tasks)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:748: in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/_parallel_backends.py:183: in apply_async
    result = ImmediateResult(func)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/_parallel_backends.py:543: in __init__
    self.results = batch()
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:224: in __call__
    for func, args, kwargs in self.items]
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:224: in <listcomp>
    for func, args, kwargs in self.items]
picrust2/wrap_hsp.py:126: in castor_hsp_wrapper
    system_call_check(hsp_cmd)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cmd = ['Rscript', '/Users/yakshiortiz/picrust2/picrust2/Rscripts/castor_hsp.R', '/Users/yakshiortiz/picrust2/tests/test_data/hsp/tree.tre', '/var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmp1d6j9yi3/subset_tab_0', 'scp', 'FALSE', ...]
print_out = False, stdout = None, stderr = None

    def system_call_check(cmd, print_out=False, stdout=None, stderr=None):
        """Run system command and throw and error if return is not 0. Input command
        can be a list containing the command or a string."""

        # Print command out if option set.
        if print_out:
            if type(cmd) is list:
                print(" ".join(cmd))
            else:
                print(cmd)

        # Convert command to list if input as string.
        if type(cmd) is str:
            cmd = cmd.split()

        return_value = call(cmd, stdout=stdout, stderr=stderr)

        # Exit with error if command did not finish successfully.
        if return_value != 0:
>           raise SystemExit("Error running this command:\n" + " ".join(cmd))
E           SystemExit: Error running this command:
E           Rscript /Users/yakshiortiz/picrust2/picrust2/Rscripts/castor_hsp.R /Users/yakshiortiz/picrust2/tests/test_data/hsp/tree.tre /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmp1d6j9yi3/subset_tab_0 scp FALSE FALSE /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpkup1ddgv/predicted_counts.txt /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpkup1ddgv/predicted_ci.txt 10

picrust2/util.py:161: SystemExit
----------------------------- Captured stderr call -----------------------------

 *** caught segfault ***
address 0x18, cause 'memory not mapped'

Traceback:
 1: dyn.load(file, DLLpath = DLLpath, ...)
 2: library.dynam(lib, package, package.lib)
 3: loadNamespace(package, lib.loc)
 4: doTryCatch(return(expr), name, parentenv, handler)
 5: tryCatchOne(expr, names, parentenv, handlers[[1L]])
 6: tryCatchList(expr, classes, parentenv, handlers)
 7: tryCatch({    attr(package, "LibPath") <- which.lib.loc    ns <- loadNamespace(package, lib.loc)    env <- attachNamespace(ns, pos = pos, deps)}, error = function(e) {    P <- if (!is.null(cc <- conditionCall(e)))         paste(" in", deparse(cc)[1L])    else ""    msg <- gettextf("package or namespace load failed for %s%s:\n %s",         sQuote(package), P, conditionMessage(e))    if (logical.return)         message(paste("Error:", msg), domain = NA)    else stop(msg, call. = FALSE, domain = NA)})
 8: library(pkg, character.only = TRUE, logical.return = TRUE, lib.loc = lib.loc,     quietly = quietly)
 9: .getRequiredPackages2(pkgInfo, quietly = quietly)
10: library(castor, quietly = TRUE)
An irrecoverable exception occurred. R is aborting now ...
____________ castor_hsp_workflow_tests.test_subtree_average_simple _____________

self = <tests.test_hsp.castor_hsp_workflow_tests testMethod=test_subtree_average_simple>

    def test_subtree_average_simple(self):

        predict_out, ci_out = castor_hsp_workflow(tree_path=in_tree1,
                                                 trait_table_path=in_traits1,
                                                 hsp_method="subtree_average",
>                                                ran_seed=10)

tests/test_hsp.py:112: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
picrust2/wrap_hsp.py:65: in castor_hsp_workflow
    for trait_in in file_subsets)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:949: in __call__
    if self.dispatch_one_batch(iterator):
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:791: in dispatch_one_batch
    self._dispatch(tasks)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:748: in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/_parallel_backends.py:183: in apply_async
    result = ImmediateResult(func)
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/_parallel_backends.py:543: in __init__
    self.results = batch()
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:224: in __call__
    for func, args, kwargs in self.items]
../miniconda3/envs/picrust2-dev/lib/python3.6/site-packages/joblib/parallel.py:224: in <listcomp>
    for func, args, kwargs in self.items]
picrust2/wrap_hsp.py:126: in castor_hsp_wrapper
    system_call_check(hsp_cmd)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cmd = ['Rscript', '/Users/yakshiortiz/picrust2/picrust2/Rscripts/castor_hsp.R', '/Users/yakshiortiz/picrust2/tests/test_data...ree.tre', '/var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpxirgx95y/subset_tab_0', 'subtree_average', 'FALSE', ...]
print_out = False, stdout = None, stderr = None

    def system_call_check(cmd, print_out=False, stdout=None, stderr=None):
        """Run system command and throw and error if return is not 0. Input command
        can be a list containing the command or a string."""

        # Print command out if option set.
        if print_out:
            if type(cmd) is list:
                print(" ".join(cmd))
            else:
                print(cmd)

        # Convert command to list if input as string.
        if type(cmd) is str:
            cmd = cmd.split()

        return_value = call(cmd, stdout=stdout, stderr=stderr)

        # Exit with error if command did not finish successfully.
        if return_value != 0:
>           raise SystemExit("Error running this command:\n" + " ".join(cmd))
E           SystemExit: Error running this command:
E           Rscript /Users/yakshiortiz/picrust2/picrust2/Rscripts/castor_hsp.R /Users/yakshiortiz/picrust2/tests/test_data/hsp/tree.tre /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpxirgx95y/subset_tab_0 subtree_average FALSE FALSE /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmp_bxftajt/predicted_counts.txt /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmp_bxftajt/predicted_ci.txt 10

picrust2/util.py:161: SystemExit
----------------------------- Captured stderr call -----------------------------

 *** caught segfault ***
address 0x18, cause 'memory not mapped'

Traceback:
 1: dyn.load(file, DLLpath = DLLpath, ...)
 2: library.dynam(lib, package, package.lib)
 3: loadNamespace(package, lib.loc)
 4: doTryCatch(return(expr), name, parentenv, handler)
 5: tryCatchOne(expr, names, parentenv, handlers[[1L]])
 6: tryCatchList(expr, classes, parentenv, handlers)
 7: tryCatch({    attr(package, "LibPath") <- which.lib.loc    ns <- loadNamespace(package, lib.loc)    env <- attachNamespace(ns, pos = pos, deps)}, error = function(e) {    P <- if (!is.null(cc <- conditionCall(e)))         paste(" in", deparse(cc)[1L])    else ""    msg <- gettextf("package or namespace load failed for %s%s:\n %s",         sQuote(package), P, conditionMessage(e))    if (logical.return)         message(paste("Error:", msg), domain = NA)    else stop(msg, call. = FALSE, domain = NA)})
 8: library(pkg, character.only = TRUE, logical.return = TRUE, lib.loc = lib.loc,     quietly = quietly)
 9: .getRequiredPackages2(pkgInfo, quietly = quietly)
10: library(castor, quietly = TRUE)
An irrecoverable exception occurred. R is aborting now ...
_________________ place_seqs_tests.test_gappa_jplace_to_newick _________________

self = <tests.test_place_seqs.place_seqs_tests testMethod=test_gappa_jplace_to_newick>

    def test_gappa_jplace_to_newick(self):
        '''Basic test for gappa_jplace_to_newick function.'''

        # Read in expected newick output.
        exp_newick_in = open(exp_newick).read()

        with TemporaryDirectory() as temp_dir:
            newick_out = path.join(temp_dir, "out.tre")

>           gappa_jplace_to_newick(jplace_file=exp_jplace, outfile=newick_out)

tests/test_place_seqs.py:120: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
picrust2/place_seqs.py:137: in gappa_jplace_to_newick
    print_out=print_cmds)
picrust2/util.py:157: in system_call_check
    return_value = call(cmd, stdout=stdout, stderr=stderr)
../miniconda3/envs/picrust2-dev/lib/python3.6/subprocess.py:267: in call
    with Popen(*popenargs, **kwargs) as p:
../miniconda3/envs/picrust2-dev/lib/python3.6/subprocess.py:709: in __init__
    restore_signals, start_new_session)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <subprocess.Popen object at 0x1197df278>
args = ['gappa', 'analyze', 'graft', '--jplace-path', '/Users/yakshiortiz/picrust2/tests/test_data/place_seqs/place_seqs_output/place_seqs_working/epa_out/epa_result.jplace', '--fully-resolve', ...]
executable = b'gappa', preexec_fn = None, close_fds = True, pass_fds = ()
cwd = None, env = None, startupinfo = None, creationflags = 0, shell = False
p2cread = -1, p2cwrite = -1, c2pread = -1, c2pwrite = -1, errread = -1
errwrite = -1, restore_signals = True, start_new_session = False

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       restore_signals, start_new_session):
        """Execute program (POSIX version)"""

        if isinstance(args, (str, bytes)):
            args = [args]
        else:
            args = list(args)

        if shell:
            args = ["/bin/sh", "-c"] + args
            if executable:
                args[0] = executable

        if executable is None:
            executable = args[0]
        orig_executable = executable

        # For transferring possible exec failure from child to parent.
        # Data format: "exception name:hex errno:description"
        # Pickle is not used; it is complex and involves memory allocation.
        errpipe_read, errpipe_write = os.pipe()
        # errpipe_write must not be in the standard io 0, 1, or 2 fd range.
        low_fds_to_close = []
        while errpipe_write < 3:
            low_fds_to_close.append(errpipe_write)
            errpipe_write = os.dup(errpipe_write)
        for low_fd in low_fds_to_close:
            os.close(low_fd)
        try:
            try:
                # We must avoid complex work that could involve
                # malloc or free in the child process to avoid
                # potential deadlocks, thus we do all this here.
                # and pass it to fork_exec()

                if env is not None:
                    env_list = []
                    for k, v in env.items():
                        k = os.fsencode(k)
                        if b'=' in k:
                            raise ValueError("illegal environment variable name")
                        env_list.append(k + b'=' + os.fsencode(v))
                else:
                    env_list = None  # Use execv instead of execve.
                executable = os.fsencode(executable)
                if os.path.dirname(executable):
                    executable_list = (executable,)
                else:
                    # This matches the behavior of os._execvpe().
                    executable_list = tuple(
                        os.path.join(os.fsencode(dir), executable)
                        for dir in os.get_exec_path(env))
                fds_to_keep = set(pass_fds)
                fds_to_keep.add(errpipe_write)
                self.pid = _posixsubprocess.fork_exec(
                        args, executable_list,
                        close_fds, tuple(sorted(map(int, fds_to_keep))),
                        cwd, env_list,
                        p2cread, p2cwrite, c2pread, c2pwrite,
                        errread, errwrite,
                        errpipe_read, errpipe_write,
                        restore_signals, start_new_session, preexec_fn)
                self._child_created = True
            finally:
                # be sure the FD is closed no matter what
                os.close(errpipe_write)

            # self._devnull is not always defined.
            devnull_fd = getattr(self, '_devnull', None)
            if p2cread != -1 and p2cwrite != -1 and p2cread != devnull_fd:
                os.close(p2cread)
            if c2pwrite != -1 and c2pread != -1 and c2pwrite != devnull_fd:
                os.close(c2pwrite)
            if errwrite != -1 and errread != -1 and errwrite != devnull_fd:
                os.close(errwrite)
            if devnull_fd is not None:
                os.close(devnull_fd)
            # Prevent a double close of these fds from __init__ on error.
            self._closed_child_pipe_fds = True

            # Wait for exec to fail or succeed; possibly raising an
            # exception (limited in size)
            errpipe_data = bytearray()
            while True:
                part = os.read(errpipe_read, 50000)
                errpipe_data += part
                if not part or len(errpipe_data) > 50000:
                    break
        finally:
            # be sure the FD is closed no matter what
            os.close(errpipe_read)

        if errpipe_data:
            try:
                pid, sts = os.waitpid(self.pid, 0)
                if pid == self.pid:
                    self._handle_exitstatus(sts)
                else:
                    self.returncode = sys.maxsize
            except ChildProcessError:
                pass

            try:
                exception_name, hex_errno, err_msg = (
                        errpipe_data.split(b':', 2))
                # The encoding here should match the encoding
                # written in by the subprocess implementations
                # like _posixsubprocess
                err_msg = err_msg.decode()
            except ValueError:
                exception_name = b'SubprocessError'
                hex_errno = b'0'
                err_msg = 'Bad exception data from child: {!r}'.format(
                              bytes(errpipe_data))
            child_exception_type = getattr(
                    builtins, exception_name.decode('ascii'),
                    SubprocessError)
            if issubclass(child_exception_type, OSError) and hex_errno:
                errno_num = int(hex_errno, 16)
                child_exec_never_called = (err_msg == "noexec")
                if child_exec_never_called:
                    err_msg = ""
                    # The error must be from chdir(cwd).
                    err_filename = cwd
                else:
                    err_filename = orig_executable
                if errno_num != 0:
                    err_msg = os.strerror(errno_num)
                    if errno_num == errno.ENOENT:
                        err_msg += ': ' + repr(err_filename)
>               raise child_exception_type(errno_num, err_msg, err_filename)
E               FileNotFoundError: [Errno 2] No such file or directory: 'gappa': 'gappa'

../miniconda3/envs/picrust2-dev/lib/python3.6/subprocess.py:1344: FileNotFoundError
_______________________ place_seqs_tests.test_run_epa_ng _______________________

self = <tests.test_place_seqs.place_seqs_tests testMethod=test_run_epa_ng>

    def test_run_epa_ng(self):
        '''Basic test to check whether EPA-NG wrapper can be run. Exact
            matches to a treefile are not checked since slight differences
            are expected depending on different versions.'''

        with TemporaryDirectory() as temp_dir:
            run_epa_ng(tree=test_tree,
                       ref_msa_fastafile=exp_ref_fasta,
                       study_msa_fastafile=exp_study_fasta,
>                      out_dir=temp_dir)

tests/test_place_seqs.py:135: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
picrust2/place_seqs.py:125: in run_epa_ng
    str(threads) + " -w " + out_dir, print_out=print_cmds)
picrust2/util.py:157: in system_call_check
    return_value = call(cmd, stdout=stdout, stderr=stderr)
../miniconda3/envs/picrust2-dev/lib/python3.6/subprocess.py:267: in call
    with Popen(*popenargs, **kwargs) as p:
../miniconda3/envs/picrust2-dev/lib/python3.6/subprocess.py:709: in __init__
    restore_signals, start_new_session)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <subprocess.Popen object at 0x1196ed7b8>
args = ['epa-ng', '--tree', '/Users/yakshiortiz/picrust2/tests/test_data/place_seqs/img_centroid_16S_aligned_head30.tre', '--...iortiz/picrust2/tests/test_data/place_seqs/place_seqs_output/place_seqs_working/ref_seqs_papara.fasta', '--query', ...]
executable = b'epa-ng', preexec_fn = None, close_fds = True, pass_fds = ()
cwd = None, env = None, startupinfo = None, creationflags = 0, shell = False
p2cread = -1, p2cwrite = -1, c2pread = -1, c2pwrite = -1, errread = -1
errwrite = -1, restore_signals = True, start_new_session = False

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       restore_signals, start_new_session):
        """Execute program (POSIX version)"""

        if isinstance(args, (str, bytes)):
            args = [args]
        else:
            args = list(args)

        if shell:
            args = ["/bin/sh", "-c"] + args
            if executable:
                args[0] = executable

        if executable is None:
            executable = args[0]
        orig_executable = executable

        # For transferring possible exec failure from child to parent.
        # Data format: "exception name:hex errno:description"
        # Pickle is not used; it is complex and involves memory allocation.
        errpipe_read, errpipe_write = os.pipe()
        # errpipe_write must not be in the standard io 0, 1, or 2 fd range.
        low_fds_to_close = []
        while errpipe_write < 3:
            low_fds_to_close.append(errpipe_write)
            errpipe_write = os.dup(errpipe_write)
        for low_fd in low_fds_to_close:
            os.close(low_fd)
        try:
            try:
                # We must avoid complex work that could involve
                # malloc or free in the child process to avoid
                # potential deadlocks, thus we do all this here.
                # and pass it to fork_exec()

                if env is not None:
                    env_list = []
                    for k, v in env.items():
                        k = os.fsencode(k)
                        if b'=' in k:
                            raise ValueError("illegal environment variable name")
                        env_list.append(k + b'=' + os.fsencode(v))
                else:
                    env_list = None  # Use execv instead of execve.
                executable = os.fsencode(executable)
                if os.path.dirname(executable):
                    executable_list = (executable,)
                else:
                    # This matches the behavior of os._execvpe().
                    executable_list = tuple(
                        os.path.join(os.fsencode(dir), executable)
                        for dir in os.get_exec_path(env))
                fds_to_keep = set(pass_fds)
                fds_to_keep.add(errpipe_write)
                self.pid = _posixsubprocess.fork_exec(
                        args, executable_list,
                        close_fds, tuple(sorted(map(int, fds_to_keep))),
                        cwd, env_list,
                        p2cread, p2cwrite, c2pread, c2pwrite,
                        errread, errwrite,
                        errpipe_read, errpipe_write,
                        restore_signals, start_new_session, preexec_fn)
                self._child_created = True
            finally:
                # be sure the FD is closed no matter what
                os.close(errpipe_write)

            # self._devnull is not always defined.
            devnull_fd = getattr(self, '_devnull', None)
            if p2cread != -1 and p2cwrite != -1 and p2cread != devnull_fd:
                os.close(p2cread)
            if c2pwrite != -1 and c2pread != -1 and c2pwrite != devnull_fd:
                os.close(c2pwrite)
            if errwrite != -1 and errread != -1 and errwrite != devnull_fd:
                os.close(errwrite)
            if devnull_fd is not None:
                os.close(devnull_fd)
            # Prevent a double close of these fds from __init__ on error.
            self._closed_child_pipe_fds = True

            # Wait for exec to fail or succeed; possibly raising an
            # exception (limited in size)
            errpipe_data = bytearray()
            while True:
                part = os.read(errpipe_read, 50000)
                errpipe_data += part
                if not part or len(errpipe_data) > 50000:
                    break
        finally:
            # be sure the FD is closed no matter what
            os.close(errpipe_read)

        if errpipe_data:
            try:
                pid, sts = os.waitpid(self.pid, 0)
                if pid == self.pid:
                    self._handle_exitstatus(sts)
                else:
                    self.returncode = sys.maxsize
            except ChildProcessError:
                pass

            try:
                exception_name, hex_errno, err_msg = (
                        errpipe_data.split(b':', 2))
                # The encoding here should match the encoding
                # written in by the subprocess implementations
                # like _posixsubprocess
                err_msg = err_msg.decode()
            except ValueError:
                exception_name = b'SubprocessError'
                hex_errno = b'0'
                err_msg = 'Bad exception data from child: {!r}'.format(
                              bytes(errpipe_data))
            child_exception_type = getattr(
                    builtins, exception_name.decode('ascii'),
                    SubprocessError)
            if issubclass(child_exception_type, OSError) and hex_errno:
                errno_num = int(hex_errno, 16)
                child_exec_never_called = (err_msg == "noexec")
                if child_exec_never_called:
                    err_msg = ""
                    # The error must be from chdir(cwd).
                    err_filename = cwd
                else:
                    err_filename = orig_executable
                if errno_num != 0:
                    err_msg = os.strerror(errno_num)
                    if errno_num == errno.ENOENT:
                        err_msg += ': ' + repr(err_filename)
>               raise child_exception_type(errno_num, err_msg, err_filename)
E               FileNotFoundError: [Errno 2] No such file or directory: 'epa-ng': 'epa-ng'

../miniconda3/envs/picrust2-dev/lib/python3.6/subprocess.py:1344: FileNotFoundError
_______________________ place_seqs_tests.test_run_papara _______________________

self = <tests.test_place_seqs.place_seqs_tests testMethod=test_run_papara>

    def test_run_papara(self):
        '''Basic test for run_papara function.'''

        exp_phylip = read_phylip(exp_papara_phylip)
        in_msa = read_fasta(test_msa)

        with TemporaryDirectory() as temp_dir:
            obs_phylip = run_papara(tree=test_tree,
                                    ref_msa=in_msa,
                                    out_dir=temp_dir,
>                                   study_fasta=test_study_seqs)

/Users/yakshiortiz/picrust2/tests/test_place_seqs.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/Users/yakshiortiz/picrust2/picrust2/place_seqs.py:85: in run_papara
    " -n out", print_out=print_cmds)
/Users/yakshiortiz/picrust2/picrust2/util.py:157: in system_call_check
    return_value = call(cmd, stdout=stdout, stderr=stderr)
/Users/yakshiortiz/miniconda3/envs/picrust2-dev/lib/python3.6/subprocess.py:267: in call
    with Popen(*popenargs, **kwargs) as p:
/Users/yakshiortiz/miniconda3/envs/picrust2-dev/lib/python3.6/subprocess.py:709: in __init__
    restore_signals, start_new_session)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <subprocess.Popen object at 0x1196fa898>
args = ['papara', '-t', '/Users/yakshiortiz/picrust2/tests/test_data/place_seqs/img_centroid_16S_aligned_head30.tre', '-s', 'ref_seqs.phylip', '-q', ...]
executable = b'papara', preexec_fn = None, close_fds = True, pass_fds = ()
cwd = None, env = None, startupinfo = None, creationflags = 0, shell = False
p2cread = -1, p2cwrite = -1, c2pread = -1, c2pwrite = -1, errread = -1
errwrite = -1, restore_signals = True, start_new_session = False

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       restore_signals, start_new_session):
        """Execute program (POSIX version)"""

        if isinstance(args, (str, bytes)):
            args = [args]
        else:
            args = list(args)

        if shell:
            args = ["/bin/sh", "-c"] + args
            if executable:
                args[0] = executable

        if executable is None:
            executable = args[0]
        orig_executable = executable

        # For transferring possible exec failure from child to parent.
        # Data format: "exception name:hex errno:description"
        # Pickle is not used; it is complex and involves memory allocation.
        errpipe_read, errpipe_write = os.pipe()
        # errpipe_write must not be in the standard io 0, 1, or 2 fd range.
        low_fds_to_close = []
        while errpipe_write < 3:
            low_fds_to_close.append(errpipe_write)
            errpipe_write = os.dup(errpipe_write)
        for low_fd in low_fds_to_close:
            os.close(low_fd)
        try:
            try:
                # We must avoid complex work that could involve
                # malloc or free in the child process to avoid
                # potential deadlocks, thus we do all this here.
                # and pass it to fork_exec()

                if env is not None:
                    env_list = []
                    for k, v in env.items():
                        k = os.fsencode(k)
                        if b'=' in k:
                            raise ValueError("illegal environment variable name")
                        env_list.append(k + b'=' + os.fsencode(v))
                else:
                    env_list = None  # Use execv instead of execve.
                executable = os.fsencode(executable)
                if os.path.dirname(executable):
                    executable_list = (executable,)
                else:
                    # This matches the behavior of os._execvpe().
                    executable_list = tuple(
                        os.path.join(os.fsencode(dir), executable)
                        for dir in os.get_exec_path(env))
                fds_to_keep = set(pass_fds)
                fds_to_keep.add(errpipe_write)
                self.pid = _posixsubprocess.fork_exec(
                        args, executable_list,
                        close_fds, tuple(sorted(map(int, fds_to_keep))),
                        cwd, env_list,
                        p2cread, p2cwrite, c2pread, c2pwrite,
                        errread, errwrite,
                        errpipe_read, errpipe_write,
                        restore_signals, start_new_session, preexec_fn)
                self._child_created = True
            finally:
                # be sure the FD is closed no matter what
                os.close(errpipe_write)

            # self._devnull is not always defined.
            devnull_fd = getattr(self, '_devnull', None)
            if p2cread != -1 and p2cwrite != -1 and p2cread != devnull_fd:
                os.close(p2cread)
            if c2pwrite != -1 and c2pread != -1 and c2pwrite != devnull_fd:
                os.close(c2pwrite)
            if errwrite != -1 and errread != -1 and errwrite != devnull_fd:
                os.close(errwrite)
            if devnull_fd is not None:
                os.close(devnull_fd)
            # Prevent a double close of these fds from __init__ on error.
            self._closed_child_pipe_fds = True

            # Wait for exec to fail or succeed; possibly raising an
            # exception (limited in size)
            errpipe_data = bytearray()
            while True:
                part = os.read(errpipe_read, 50000)
                errpipe_data += part
                if not part or len(errpipe_data) > 50000:
                    break
        finally:
            # be sure the FD is closed no matter what
            os.close(errpipe_read)

        if errpipe_data:
            try:
                pid, sts = os.waitpid(self.pid, 0)
                if pid == self.pid:
                    self._handle_exitstatus(sts)
                else:
                    self.returncode = sys.maxsize
            except ChildProcessError:
                pass

            try:
                exception_name, hex_errno, err_msg = (
                        errpipe_data.split(b':', 2))
                # The encoding here should match the encoding
                # written in by the subprocess implementations
                # like _posixsubprocess
                err_msg = err_msg.decode()
            except ValueError:
                exception_name = b'SubprocessError'
                hex_errno = b'0'
                err_msg = 'Bad exception data from child: {!r}'.format(
                              bytes(errpipe_data))
            child_exception_type = getattr(
                    builtins, exception_name.decode('ascii'),
                    SubprocessError)
            if issubclass(child_exception_type, OSError) and hex_errno:
                errno_num = int(hex_errno, 16)
                child_exec_never_called = (err_msg == "noexec")
                if child_exec_never_called:
                    err_msg = ""
                    # The error must be from chdir(cwd).
                    err_filename = cwd
                else:
                    err_filename = orig_executable
                if errno_num != 0:
                    err_msg = os.strerror(errno_num)
                    if errno_num == errno.ENOENT:
                        err_msg += ': ' + repr(err_filename)
>               raise child_exception_type(errno_num, err_msg, err_filename)
E               FileNotFoundError: [Errno 2] No such file or directory: 'papara': 'papara'

/Users/yakshiortiz/miniconda3/envs/picrust2-dev/lib/python3.6/subprocess.py:1344: FileNotFoundError
________________ place_seqs_tests.test_run_place_seqs_pipeline _________________

self = <tests.test_place_seqs.place_seqs_tests testMethod=test_run_place_seqs_pipeline>

    def test_run_place_seqs_pipeline(self):
        '''Basic test of full place seqs pipeline. As for EPA-NG, exact
            matches to a treefile are not checked since slight differences
            are expected depending on different versions.'''

        with TemporaryDirectory() as temp_dir:
            tmp_tree = path.join(temp_dir, "out.tre")

            place_seqs_pipeline(study_fasta=test_study_seqs,
                                ref_msa=test_msa,
                                tree=test_tree,
                                out_tree=tmp_tree,
                                threads=1,
                                papara_output=None,
                                out_dir=temp_dir,
                                chunk_size=5000,
>                               print_cmds=False)

/Users/yakshiortiz/picrust2/tests/test_place_seqs.py:153: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/Users/yakshiortiz/picrust2/picrust2/place_seqs.py:35: in place_seqs_pipeline
    threads=threads, print_cmds=print_cmds)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

tree = '/Users/yakshiortiz/picrust2/tests/test_data/place_seqs/img_centroid_16S_aligned_head30.tre'
ref_msa = {'2501846300_cluster': 'ATTGAACGCTGGCGGCAGGCCTAACACATGCAAGTCGAGCGGAGCGGCGGACGGGTGAGTAATGCCTAGGA-ATCTGCCTGGTAGTGGGGGATA...CGTTCCCGGGCC-TGTACACACCGCCCGTCACACCATGGGAGTTTGTTGCACCAGAAGTA--TAC-CACGGTGTGGCCGATGACTGGGGTGAAGTCGTAACAAGGTAGCCGT', ...}
study_fasta = '/Users/yakshiortiz/picrust2/tests/test_data/place_seqs/study_seqs_test.fasta'
out_dir = '/var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpixi1dk16'
threads = 1, print_cmds = False

    def run_papara(tree: str, ref_msa: dict, study_fasta: str, out_dir: str,
                   threads=1, print_cmds=False):
        '''Run PaPaRa to place study sequences into reference multiple-sequence
        alignment (MSA). Will return dictionary of the the output MSA (sequence ids
        as keys). Expects path to tree and study FASTA as strings. Expects
        reference MSA as a dictionary output by read_fasta. This MSA will be
        converted to phylip format before running PaPaRa.'''

        # Get absolute paths to input files.
        tree = path.abspath(tree)
        study_fasta = path.abspath(study_fasta)

        # Change working directory to out directory (but keep track of original).
        # This is necessary because PaPaRa outputs into the current working
        # directory.
>       orig_wd = getcwd()
E       FileNotFoundError: [Errno 2] No such file or directory

/Users/yakshiortiz/picrust2/picrust2/place_seqs.py:76: FileNotFoundError
____________________ workflow_test.test_full_pipeline_biom _____________________

self = <tests.test_workflow.workflow_test testMethod=test_full_pipeline_biom>

    def test_full_pipeline_biom(self):
        '''Test that full pipeline can be run without error with
            BIOM sequence abundance table.'''

        with TemporaryDirectory() as temp_dir:

            out_tree = path.join(temp_dir, "out.tre")

            system_call_check("place_seqs.py -s " + test_study_seqs + " -r " +\
                              test_msa + " -t " + test_tree + " -o " +\
>                             out_tree)

/Users/yakshiortiz/picrust2/tests/test_workflow.py:97: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cmd = ['place_seqs.py', '-s', '/Users/yakshiortiz/picrust2/tests/test_data/place_seqs/study_seqs_test.fasta', '-r', '/Users/yakshiortiz/picrust2/tests/test_data/place_seqs/img_centroid_16S_aligned_head30.fna', '-t', ...]
print_out = False, stdout = None, stderr = None

    def system_call_check(cmd, print_out=False, stdout=None, stderr=None):
        """Run system command and throw and error if return is not 0. Input command
        can be a list containing the command or a string."""

        # Print command out if option set.
        if print_out:
            if type(cmd) is list:
                print(" ".join(cmd))
            else:
                print(cmd)

        # Convert command to list if input as string.
        if type(cmd) is str:
            cmd = cmd.split()

        return_value = call(cmd, stdout=stdout, stderr=stderr)

        # Exit with error if command did not finish successfully.
        if return_value != 0:
>           raise SystemExit("Error running this command:\n" + " ".join(cmd))
E           SystemExit: Error running this command:
E           place_seqs.py -s /Users/yakshiortiz/picrust2/tests/test_data/place_seqs/study_seqs_test.fasta -r /Users/yakshiortiz/picrust2/tests/test_data/place_seqs/img_centroid_16S_aligned_head30.fna -t /Users/yakshiortiz/picrust2/tests/test_data/place_seqs/img_centroid_16S_aligned_head30.tre -o /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpuy2ivted/out.tre

/Users/yakshiortiz/picrust2/picrust2/util.py:161: SystemExit
----------------------------- Captured stderr call -----------------------------
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
Traceback (most recent call last):
  File "/Users/yakshiortiz/miniconda3/envs/picrust2-dev/bin/place_seqs.py", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/Users/yakshiortiz/picrust2/scripts/place_seqs.py", line 97, in <module>
    main()
  File "/Users/yakshiortiz/picrust2/scripts/place_seqs.py", line 93, in main
    print_cmds=args.print_cmds)
  File "/Users/yakshiortiz/picrust2/picrust2/place_seqs.py", line 35, in place_seqs_pipeline
    threads=threads, print_cmds=print_cmds)
  File "/Users/yakshiortiz/picrust2/picrust2/place_seqs.py", line 76, in run_papara
    orig_wd = getcwd()
FileNotFoundError: [Errno 2] No such file or directory
_____________________ workflow_test.test_full_pipeline_tsv _____________________

self = <tests.test_workflow.workflow_test testMethod=test_full_pipeline_tsv>

    def test_full_pipeline_tsv(self):
        '''Test that full pipeline can be run without error with
            TSV sequence abundance table.'''

        with TemporaryDirectory() as temp_dir:

            out_tree = path.join(temp_dir, "out.tre")

            system_call_check("place_seqs.py -s " + test_study_seqs + " -r " +\
                              test_msa + " -t " + test_tree + " -o " +\
>                             out_tree)

/Users/yakshiortiz/picrust2/tests/test_workflow.py:54: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cmd = ['place_seqs.py', '-s', '/Users/yakshiortiz/picrust2/tests/test_data/place_seqs/study_seqs_test.fasta', '-r', '/Users/yakshiortiz/picrust2/tests/test_data/place_seqs/img_centroid_16S_aligned_head30.fna', '-t', ...]
print_out = False, stdout = None, stderr = None

    def system_call_check(cmd, print_out=False, stdout=None, stderr=None):
        """Run system command and throw and error if return is not 0. Input command
        can be a list containing the command or a string."""

        # Print command out if option set.
        if print_out:
            if type(cmd) is list:
                print(" ".join(cmd))
            else:
                print(cmd)

        # Convert command to list if input as string.
        if type(cmd) is str:
            cmd = cmd.split()

        return_value = call(cmd, stdout=stdout, stderr=stderr)

        # Exit with error if command did not finish successfully.
        if return_value != 0:
>           raise SystemExit("Error running this command:\n" + " ".join(cmd))
E           SystemExit: Error running this command:
E           place_seqs.py -s /Users/yakshiortiz/picrust2/tests/test_data/place_seqs/study_seqs_test.fasta -r /Users/yakshiortiz/picrust2/tests/test_data/place_seqs/img_centroid_16S_aligned_head30.fna -t /Users/yakshiortiz/picrust2/tests/test_data/place_seqs/img_centroid_16S_aligned_head30.tre -o /var/folders/_2/bs2w2q6n6n555b_bj3__xk300000gr/T/tmpzh3vt4vu/out.tre

/Users/yakshiortiz/picrust2/picrust2/util.py:161: SystemExit
----------------------------- Captured stderr call -----------------------------
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
Traceback (most recent call last):
  File "/Users/yakshiortiz/miniconda3/envs/picrust2-dev/bin/place_seqs.py", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/Users/yakshiortiz/picrust2/scripts/place_seqs.py", line 97, in <module>
    main()
  File "/Users/yakshiortiz/picrust2/scripts/place_seqs.py", line 93, in main
    print_cmds=args.print_cmds)
  File "/Users/yakshiortiz/picrust2/picrust2/place_seqs.py", line 35, in place_seqs_pipeline
    threads=threads, print_cmds=print_cmds)
  File "/Users/yakshiortiz/picrust2/picrust2/place_seqs.py", line 76, in run_papara
    orig_wd = getcwd()
FileNotFoundError: [Errno 2] No such file or directory
===================== 13 failed, 15 passed in 7.44 seconds =====================
gavinmdouglas commented 6 years ago

Hi there,

I have only tested the PICRUSt2 beta on Linux environments and it seems you're running into incompatibility issues with the placement tools on Mac OS X. I'm sorry for the confusion - I'll clarify on the installation page that PICRUSt2 is Linux only at least for now.

Best,

Gavin

yakshiUPR commented 6 years ago

Gavin,

Thanks for your reply! This explains the errors I was getting. I will try it on a Linux cluster.

Yakshi