OS-X support

[laserson]

I just built partis from scratch on my mac. I can successfully run the annotate command from manual.md. However, when I run the partition command

./bin/partis partition --infname test/example.fa --outfname _output/example.csv

I get the following output

$ ./bin/partis partition --infname test/example.fa --outfname _output/example.csv
  parameter dir '_output/test_example' does not exist, so caching a new set of parameters before running action 'partition'
caching parameters
smith-waterman   (removing less-likely alleles) 
        sequences    n_procs     ig-sw time    processing time
             250        1           11.9           0.4
               9        1            0.4           0.0
               9        1            0.4           0.0
      info for 250 / 250 = 1.000   (0 failed)
      kept 144 (0.576) unproductive
  removing least likely alleles (250 total counts)
     genes to keep           counts (snps)      removed counts     removed genes 
       hv3-23D01                  71      
       hv3-2302                    4  (2) 
       hv3-NL101                  45      
       hv3-1104                   24      
       hv3-701                    18      
       hv3-6404                   14      
       hv3-6603                   12      
       hv3-5302                    4  (4)                1            hv3-5301 
       hv3-4905                    9      
       hv3-4904                    8  (2) 
       hv1-301                     8      
       hv4-407                     7      
       hv3-6601                    7      
       hv3-7403                    3      
       hv1-6901                    3      
       hv1-6913                    1  (2) 
       hv1-802                     2      
       hv4-6102                    2      
       hv4-6108                    2      
       hv5-5103                    2      
       hv1-4603                    2      
       hv3-4803                    1      
    keeping 22 v genes
    removing 265 v genes: 264 with no matches, 1 with unconvincing matches (1 / 250 queries had their best match removed)
      removed 4 / 250 = 0.02 duplicate sequences after trimming framework insertions (leaving 246)
        water time: 13.3
smith-waterman   (writing parameters) 
        sequences    n_procs     ig-sw time    processing time
             250        1            1.4           0.3
               9        1            0.1           0.0
               9        1            0.1           0.0
      info for 250 / 250 = 1.000   (0 failed)
      kept 144 (0.576) unproductive
      removed 4 / 250 = 0.02 duplicate sequences after trimming framework insertions (leaving 246)
        writing sw results to _output/test_example/sw-cache-5421323308298323221
    writing parameters (2.7 sec)
        water time: 4.7
  writing hmms (13.4 sec)
hmm
    running 1 proc
    read output
   warning best-supported gene hd3-302 not same as viterbi gene hd3-301
    writing parameters (2.0 sec)
        processed 246 hmm output lines with 246 sequences in 246 events  (0 failures)
         infra time: 2.5
      hmm step time: 6.1
  writing hmms (12.0 sec)
      total time: 49.5
partitioning
smith-waterman 
        reading sw results from _output/test_example/sw-cache-5421323308298323221
      info for 246 / 250 = 0.984   (4 failed)
      kept 142 (0.568) unproductive
        water time: 0.2
hmm
--> caching all 246 naive sequences
    running 1 proc
    proc 0 try 1 failed with -6 (output is missing)
        out tail:           (/tmp/laserson/hmms/554706/out)
                    read-cache:  logprobs 0   naive-seqs 0
                  caching all naive sequences

        err tail:           (/tmp/laserson/hmms/554706/err)
            Assertion failed: (naive_seqs_.count(queries) == 0), function CalculateNaiveSeq, file _build/glomerator.cc, line 702.

    restarting proc 0
    proc 0 try 2 failed with -6 (output is missing)
        out tail:           (/tmp/laserson/hmms/554706/out)
                    read-cache:  logprobs 0   naive-seqs 0
                  caching all naive sequences

        err tail:           (/tmp/laserson/hmms/554706/err)
            Assertion failed: (naive_seqs_.count(queries) == 0), function CalculateNaiveSeq, file _build/glomerator.cc, line 702.

    restarting proc 0
    proc 0 try 3 failed with -6 (output is missing)
        out tail:           (/tmp/laserson/hmms/554706/out)
                    read-cache:  logprobs 0   naive-seqs 0
                  caching all naive sequences

        err tail:           (/tmp/laserson/hmms/554706/err)
            Assertion failed: (naive_seqs_.count(queries) == 0), function CalculateNaiveSeq, file _build/glomerator.cc, line 702.

    restarting proc 0
    proc 0 try 4 failed with -6 (output is missing)
        out tail:           (/tmp/laserson/hmms/554706/out)
                    read-cache:  logprobs 0   naive-seqs 0
                  caching all naive sequences

        err tail:           (/tmp/laserson/hmms/554706/err)
            Assertion failed: (naive_seqs_.count(queries) == 0), function CalculateNaiveSeq, file _build/glomerator.cc, line 702.

    restarting proc 0
    proc 0 try 5 failed with -6 (output is missing)
        out tail:           (/tmp/laserson/hmms/554706/out)
                    read-cache:  logprobs 0   naive-seqs 0
                  caching all naive sequences

        err tail:           (/tmp/laserson/hmms/554706/err)
            Assertion failed: (naive_seqs_.count(queries) == 0), function CalculateNaiveSeq, file _build/glomerator.cc, line 702.

    restarting proc 0
exceeded max number of tries for cmd
    /Users/laserson/repos/partis/packages/ham/bcrham --algorithm viterbi --hmmdir /Users/laserson/repos/partis/_output/test_example/hmm/hmms --datadir /tmp/laserson/hmms/554706/germline-sets --infile /tmp/laserson/hmms/554706/hmm_input.csv --outfile /tmp/laserson/hmms/554706/hmm_output.csv --locus igh --random-seed 1495511391 --output-cachefname /tmp/laserson/hmms/554706/hmm_cached_info.csv --cache-naive-seqs --ambig-base N
look for output in /tmp/laserson/hmms/554706 and /tmp/laserson/hmms/554706
Traceback (most recent call last):
  File "./bin/partis", line 581, in <module>
    args.func(args)
  File "./bin/partis", line 206, in run_partitiondriver
    parter.partition()
  File "/Users/laserson/repos/partis/python/partitiondriver.py", line 338, in partition
    self.run_hmm('viterbi', self.sub_param_dir, n_procs=self.auto_nprocs(len(self.sw_info['queries'])), precache_all_naive_seqs=True)
  File "/Users/laserson/repos/partis/python/partitiondriver.py", line 837, in run_hmm
    self.execute(cmd_str, n_procs)
  File "/Users/laserson/repos/partis/python/partitiondriver.py", line 808, in execute
    utils.run_cmds(cmdfos, batch_system=self.args.batch_system, batch_options=self.args.batch_options, batch_config_fname=self.args.batch_config_fname, debug='print' if self.args.debug else None)
  File "/Users/laserson/repos/partis/python/utils.py", line 1785, in run_cmds
    finish_process(iproc, procs, n_tries, cmdfos[iproc], dbgfo=cmdfos[iproc]['dbgfo'], batch_system=batch_system, batch_options=batch_options, debug=debug, ignore_stderr=ignore_stderr)
  File "/Users/laserson/repos/partis/python/utils.py", line 1812, in finish_process
    raise Exception(failstr)
Exception: exceeded max number of tries for cmd
    /Users/laserson/repos/partis/packages/ham/bcrham --algorithm viterbi --hmmdir /Users/laserson/repos/partis/_output/test_example/hmm/hmms --datadir /tmp/laserson/hmms/554706/germline-sets --infile /tmp/laserson/hmms/554706/hmm_input.csv --outfile /tmp/laserson/hmms/554706/hmm_output.csv --locus igh --random-seed 1495511391 --output-cachefname /tmp/laserson/hmms/554706/hmm_cached_info.csv --cache-naive-seqs --ambig-base N
look for output in /tmp/laserson/hmms/554706 and /tmp/laserson/hmms/554706

The /tmp/... directory contains a file called err with contents

Assertion failed: (naive_seqs_.count(queries) == 0), function CalculateNaiveSeq, file _build/glomerator.cc, line 702.

[psathyrella]

hm, sorry about that. It runs fine on my installs, but the mac thing is certainly suspicous -- I don't know for a fact that anyone's used it on osx. There can't possibly be much different, but I'm not familiar enough to know what would need to be changed on osx.

I wonder if you could try the test suite,

./test/test.py

[psathyrella]

I wonder if a different compiler could be defining size_type here to something differerent? That seems unlikely, but I don't have a lot of other ideas yet.

[laserson]

Good call, as I see the build script runs a smaller number of tests. The main test script fails like so:

$ test/test.py 
annotate-ref-simu                ./bin/partis annotate --plot-performance --plotdir test/new-results/simu-ref-performance --only-csv-plots --is-simu --sw-cachefname test/reference-results/test/parameters/simu/sw-cache.csv --infname test/reference-results/test/simu.csv --parameter-dir test/r[...]
partition-ref-simu               ./bin/partis partition --n-max-queries 500 --n-precache-procs 10 --biggest-logprob-cluster-to-calculate 5 --biggest-naive-seq-cluster-to-calculate 5 --persistent-cachefname test/new-results/cache-ref-partition.csv --is-simu --sw-cachefname test/reference-res[...]
  log tail:
          File "/Users/laserson/repos/partis/python/utils.py", line 1785, in run_cmds
            finish_process(iproc, procs, n_tries, cmdfos[iproc], dbgfo=cmdfos[iproc]['dbgfo'], batch_system=batch_system, batch_options=batch_options, debug=debug, ignore_stderr=ignore_stderr)
          File "/Users/laserson/repos/partis/python/utils.py", line 1812, in finish_process
            raise Exception(failstr)
        Exception: exceeded max number of tries for cmd
            /Users/laserson/repos/partis/packages/ham/bcrham --algorithm viterbi --hmmdir /Users/laserson/repos/partis/test/reference-results/test/parameters/simu/hmm/hmms --datadir /tmp/laserson/hmms/771957/germline-sets --infile /tmp/laserson/hmms/771957/hmm-0/hmm_input.csv --outfile /tmp/laserson/hmms/771957/hmm-0/hmm_output.csv --locus igh --random-seed 1 --only-cache-new-vals --output-cachefname /tmp/laserson/hmms/771957/hmm-0/hmm_cached_info.csv --cache-naive-seqs --ambig-base N
        look for output in /tmp/laserson/hmms/771957/hmm-0 and /tmp/laserson/hmms/771957/hmm-0
        exceeded max number of tries for cmd
            /Users/laserson/repos/partis/packages/ham/bcrham --algorithm viterbi --hmmdir /Users/laserson/repos/partis/test/reference-results/test/parameters/simu/hmm/hmms --datadir /tmp/laserson/hmms/771957/germline-sets --infile /tmp/laserson/hmms/771957/hmm-0/hmm_input.csv --outfile /tmp/laserson/hmms/771957/hmm-0/hmm_output.csv --locus igh --random-seed 1 --only-cache-new-vals --output-cachefname /tmp/laserson/hmms/771957/hmm-0/hmm_cached_info.csv --cache-naive-seqs --ambig-base N
        look for output in /tmp/laserson/hmms/771957/hmm-0 and /tmp/laserson/hmms/771957/hmm-0

[laserson]

And same error

Assertion failed: (naive_seqs_.count(queries) == 0), function CalculateNaiveSeq, file _build/glomerator.cc, line 702.

[psathyrella]

ok, I'll try to round up an osx machine to test it on and figure out what the deal is. I'd been meaning to add osx-specific directions anyway. In the meantime you're probably best off running in docker, though.

[psathyrella]

ok I still haven't managed to try this on an os-x machine, but someone else just successfully installed/built it on their os-x laptop. They pretty much took the apt-get/pip installs and tried them first with brew, then with pip if that didn't work.

[laserson]

Did they run the same test suite? I had no problem with the build, but test/test.py failed.

[psathyrella]

oh, darn, you're right it was only the partition that was crashing for you, just had him go back and run partitioning and it crashes with the same error.

Yays, it's repeatable. Not-yays, that is so weird. Well, I'll dig through and work it out. Maybe the compiler's treating the nullptrkeyword differently? Sorry for the trouble.

[psathyrella]

ok this seems to fix it on Branden's install. Let me know if it doesn't fix it for you.

the issue was a line like this:

map[key] = getval();

where getval() checks to make sure that key isn't already in map. The os-x compiler apparently evaluates the LHS before the RHS, i.e. adds key to map before calling getval(), which causes the check in getval() to fail.

[psathyrella]

oh, also: I added a little os-x specific installation info to the manual here

https://github.com/psathyrella/partis/blob/master/manual.md#installation-from-scratch

it'd be lovely if you have anything to add to it if you could let me know.

[laserson]

Now I get

$ test/test.py 
annotate-ref-simu                ./bin/partis annotate --plot-performance --plotdir test/new-results/simu-ref-performance --only-csv-plots --is-simu --sw-cachefname test/reference-results/test/parameters/simu/sw-cache.csv --infname test/reference-results/test/simu.csv --parameter-dir test/r[...]
partition-ref-simu               ./bin/partis partition --n-max-queries 500 --n-precache-procs 10 --biggest-logprob-cluster-to-calculate 5 --biggest-naive-seq-cluster-to-calculate 5 --persistent-cachefname test/new-results/cache-ref-partition.csv --is-simu --sw-cachefname test/reference-res[...]
seed-partition-ref-simu          ./bin/partis partition --n-max-queries 500 --persistent-cachefname test/new-results/cache-ref-partition.csv --dont-precache-naive-seqs --is-simu --sw-cachefname test/reference-results/test/parameters/simu/sw-cache.csv --infname test/reference-results/test/si[...]
vsearch-partition-ref-simu       ./bin/partis partition --naive-vsearch --n-max-queries 500 --persistent-cachefname test/new-results/cache-ref-partition.csv --dont-precache-naive-seqs --is-simu --sw-cachefname test/reference-results/test/parameters/simu/sw-cache.csv --infname test/reference[...]
  log tail:
            errread, errwrite)
          File "/Users/laserson/miniconda3/envs/py27/lib/python2.7/subprocess.py", line 1024, in _execute_child
            raise child_exception
        OSError: [Errno 8] Exec format error
                water time: 0.7
        hmm
               naive hfrac bounds: 0.036 0.036   (0.131 mean mutation in parameter dir test/reference-results/test/parameters/simu/hmm)
                collapsed 500 sequences into 296 unique naive sequences
            using hfrac bound for vsearch 0.036
            running vsearch 20 times (once for each cdr3 length class):

[psathyrella]

ah, that must be the result of me just having the vsearch binary in the repo. I'll probably add the os-x version and look up how to test at run time what os I'm running on? That might not scale, though, I'll have to read a bit.

[psathyrella]

https://github.com/psathyrella/partis/commit/87ac0d54e921f5240d2be7b9ea1c6726d9c4d59c

ok, this is fixed (added darwin vsearch binary and started checking for os). Let me know if you have any other problems.