Problem compiling weave.inline code

[inti]

hi Matt, I am trying to run this on our computer cluster. I am getting an error with the compilation of the weave.inline function on the negative binomial sampling. The code works find on my desktop machine scipy 0.11.0 numpy 1.6.2

on the cluster produces the error below scipy 0.12.0 numpy 1.7.1 both the latest

For some reason your other weave code compiles and runs but the one on the negative binomial does not. I wonder if you can see a reason for this an a possible solution.

Thanks a lot in advance, BW, Inti Pedroso

$ ~/.local/bin/ipython cnv_calling.py data/NA12878.mapped.ILLUMINA.bwa.CEU.low_coverage.20120522.10kb.21.normalized.bed data/NA12878.mapped.ILLUMINA.bwa.CEU.low_coverage.20120522.10kb.21.normalized.bed
WARNING: IPython History requires SQLite, your history will not be saved
Fri Apr 12 15:04:08 2013 Started
Fri Apr 12 15:04:08 2013 1  files to load
Fri Apr 12 15:04:08 2013    done loading data...
Fri Apr 12 15:04:08 2013 Loading data
Fri Apr 12 15:04:08 2013 Sampling model using Gaussian approximation
repairing catalog by removing key
In file included from /home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1732,
                 from /home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from /home/pedrosoi/.python27_compiled/sc_e5c8f7d8772826cfbd12c6b9b9277fc82.cpp:22:
/home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
/home/pedrosoi/.python27_compiled/sc_e5c8f7d8772826cfbd12c6b9b9277fc82.cpp: In function ‘PyObject* compiled_func(PyObject*, PyObject*)’:
/home/pedrosoi/.python27_compiled/sc_e5c8f7d8772826cfbd12c6b9b9277fc82.cpp:676: warning: unused variable ‘Nweighted_cols’
/home/pedrosoi/.python27_compiled/sc_e5c8f7d8772826cfbd12c6b9b9277fc82.cpp:677: warning: unused variable ‘Sweighted_cols’
/home/pedrosoi/.python27_compiled/sc_e5c8f7d8772826cfbd12c6b9b9277fc82.cpp:678: warning: unused variable ‘Dweighted_cols’
/home/pedrosoi/.python27_compiled/sc_e5c8f7d8772826cfbd12c6b9b9277fc82.cpp:693: warning: unused variable ‘Nmsum’
/home/pedrosoi/.python27_compiled/sc_e5c8f7d8772826cfbd12c6b9b9277fc82.cpp:694: warning: unused variable ‘Smsum’
/home/pedrosoi/.python27_compiled/sc_e5c8f7d8772826cfbd12c6b9b9277fc82.cpp:695: warning: unused variable ‘Dmsum’
/home/pedrosoi/.python27_compiled/sc_e5c8f7d8772826cfbd12c6b9b9277fc82.cpp:704: warning: unused variable ‘Ncounts’
/home/pedrosoi/.python27_compiled/sc_e5c8f7d8772826cfbd12c6b9b9277fc82.cpp:705: warning: unused variable ‘Scounts’
/home/pedrosoi/.python27_compiled/sc_e5c8f7d8772826cfbd12c6b9b9277fc82.cpp:706: warning: unused variable ‘Dcounts’
repairing catalog by removing key
In file included from /home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1732,
                 from /home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from /home/pedrosoi/.python27_compiled/sc_64393f72e1f0a6d5a91c617a1c339d347.cpp:22:
/home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
/home/pedrosoi/.python27_compiled/sc_64393f72e1f0a6d5a91c617a1c339d347.cpp: In function ‘PyObject* compiled_func(PyObject*, PyObject*)’:
/home/pedrosoi/.python27_compiled/sc_64393f72e1f0a6d5a91c617a1c339d347.cpp:677: warning: unused variable ‘NAT’
/home/pedrosoi/.python27_compiled/sc_64393f72e1f0a6d5a91c617a1c339d347.cpp:678: warning: unused variable ‘SAT’
/home/pedrosoi/.python27_compiled/sc_64393f72e1f0a6d5a91c617a1c339d347.cpp:679: warning: unused variable ‘DAT’
/home/pedrosoi/.python27_compiled/sc_64393f72e1f0a6d5a91c617a1c339d347.cpp:688: warning: unused variable ‘Nbetal’
/home/pedrosoi/.python27_compiled/sc_64393f72e1f0a6d5a91c617a1c339d347.cpp:689: warning: unused variable ‘Sbetal’
/home/pedrosoi/.python27_compiled/sc_64393f72e1f0a6d5a91c617a1c339d347.cpp:690: warning: unused variable ‘Dbetal’
/home/pedrosoi/.python27_compiled/sc_64393f72e1f0a6d5a91c617a1c339d347.cpp:699: warning: unused variable ‘NaBl’
/home/pedrosoi/.python27_compiled/sc_64393f72e1f0a6d5a91c617a1c339d347.cpp:700: warning: unused variable ‘SaBl’
/home/pedrosoi/.python27_compiled/sc_64393f72e1f0a6d5a91c617a1c339d347.cpp:701: warning: unused variable ‘DaBl’
repairing catalog by removing key
In file included from /home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1732,
                 from /home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from /home/pedrosoi/.python27_compiled/sc_7ae1c26ac5e5e56eaa7306322a49ecfd2.cpp:22:
/home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
/home/pedrosoi/.python27_compiled/sc_7ae1c26ac5e5e56eaa7306322a49ecfd2.cpp: In function ‘PyObject* compiled_func(PyObject*, PyObject*)’:
/home/pedrosoi/.python27_compiled/sc_7ae1c26ac5e5e56eaa7306322a49ecfd2.cpp:678: warning: unused variable ‘NA’
/home/pedrosoi/.python27_compiled/sc_7ae1c26ac5e5e56eaa7306322a49ecfd2.cpp:679: warning: unused variable ‘SA’
/home/pedrosoi/.python27_compiled/sc_7ae1c26ac5e5e56eaa7306322a49ecfd2.cpp:680: warning: unused variable ‘DA’
/home/pedrosoi/.python27_compiled/sc_7ae1c26ac5e5e56eaa7306322a49ecfd2.cpp:691: warning: unused variable ‘Npi0’
/home/pedrosoi/.python27_compiled/sc_7ae1c26ac5e5e56eaa7306322a49ecfd2.cpp:692: warning: unused variable ‘Spi0’
/home/pedrosoi/.python27_compiled/sc_7ae1c26ac5e5e56eaa7306322a49ecfd2.cpp:693: warning: unused variable ‘Dpi0’
/home/pedrosoi/.python27_compiled/sc_7ae1c26ac5e5e56eaa7306322a49ecfd2.cpp:702: warning: unused variable ‘Nstateseq’
/home/pedrosoi/.python27_compiled/sc_7ae1c26ac5e5e56eaa7306322a49ecfd2.cpp:703: warning: unused variable ‘Sstateseq’
/home/pedrosoi/.python27_compiled/sc_7ae1c26ac5e5e56eaa7306322a49ecfd2.cpp:704: warning: unused variable ‘Dstateseq’
/home/pedrosoi/.python27_compiled/sc_7ae1c26ac5e5e56eaa7306322a49ecfd2.cpp:713: warning: unused variable ‘NaBl’
/home/pedrosoi/.python27_compiled/sc_7ae1c26ac5e5e56eaa7306322a49ecfd2.cpp:714: warning: unused variable ‘SaBl’
/home/pedrosoi/.python27_compiled/sc_7ae1c26ac5e5e56eaa7306322a49ecfd2.cpp:715: warning: unused variable ‘DaBl’
/home/pedrosoi/.python27_compiled/sc_7ae1c26ac5e5e56eaa7306322a49ecfd2.cpp:724: warning: unused variable ‘Nbetal’
/home/pedrosoi/.python27_compiled/sc_7ae1c26ac5e5e56eaa7306322a49ecfd2.cpp:725: warning: unused variable ‘Sbetal’
/home/pedrosoi/.python27_compiled/sc_7ae1c26ac5e5e56eaa7306322a49ecfd2.cpp:726: warning: unused variable ‘Dbetal’
.....
   1.73sec avg,    8.67sec total

Fri Apr 12 15:04:17 2013 Loading data
Fri Apr 12 15:04:17 2013 Sampling model using Negative Binomial approximation
In file included from /home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1732,
                 from /home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from /home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.cpp:22:
/home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
/home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.cpp: In function ‘PyObject* compiled_func(PyObject*, PyObject*)’:
/home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.cpp:691: error: ambiguous overload for ‘operator<’ in ‘j < py::object::operator[](int)(i)’
/home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.cpp:691: note: candidates are: operator<(int, int) <built-in>
/home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.cpp:691: note:                 operator<(int, float) <built-in>
/home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.cpp:691: note:                 operator<(int, double) <built-in>
/home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.cpp:682: warning: unused variable ‘Nmsum’
/home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.cpp:683: warning: unused variable ‘Smsum’
/home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.cpp:684: warning: unused variable ‘Dmsum’
In file included from /home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1732,
                 from /home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from /home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.cpp:22:
/home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
/home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.cpp: In function ‘PyObject* compiled_func(PyObject*, PyObject*)’:
/home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.cpp:691: error: ambiguous overload for ‘operator<’ in ‘j < py::object::operator[](int)(i)’
/home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.cpp:691: note: candidates are: operator<(int, int) <built-in>
/home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.cpp:691: note:                 operator<(int, float) <built-in>
/home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.cpp:691: note:                 operator<(int, double) <built-in>
/home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.cpp:682: warning: unused variable ‘Nmsum’
/home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.cpp:683: warning: unused variable ‘Smsum’
/home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.cpp:684: warning: unused variable ‘Dmsum’
---------------------------------------------------------------------------
CompileError                              Traceback (most recent call last)
/home/pedrosoi/.local/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where)
    176             else:
    177                 filename = fname
--> 178             __builtin__.execfile(filename, *where)

/gpfs/home/pedrosoi/My_Soft/cnv-hdp-hmm/cnv_calling.py in <module>()
    157 cp = {} # sequence of states
    158 for itr in progprint_xrange(niter_negative_binomial):
--> 159     negative_binomial_model.resample_model()
    160     ll[itr] = negative_binomial_model.log_likelihood(negative_binomial_model.states_list[0].data)
    161     cp[itr] = negative_binomial_model.states_list[0].stateseq

/home/pedrosoi/APP/pyhsmm/models.pyc in resample_model(self)
    112         # resample obsparams
    113         for state, distn in enumerate(self.obs_distns):
--> 114             distn.resample([s.data[s.stateseq == state] for s in self.states_list])
    115 
    116         # resample transitions

/home/pedrosoi/APP/pyhsmm/basic/pybasicbayes/distributions.pyc in resample(self, data, niter)
   1061                         ''',
   1062                         ['N','data','r','msum'],
-> 1063                         extra_compile_args=['-O3'])
   1064                 self.r = np.random.gamma(self.k_0 + msum, 1/(1/self.theta_0 - N*np.log(1-self.p)))
   1065                 ### resample p

/home/pedrosoi/.local/lib/python2.7/site-packages/scipy/weave/inline_tools.pyc in inline(code, arg_names, local_dict, global_dict, force, compiler, verbose, support_code, headers, customize, type_converters, auto_downcast, newarr_converter, **kw)
    355                                     type_converters = type_converters,
    356                                     auto_downcast = auto_downcast,
--> 357                                     **kw)
    358 
    359             function_catalog.add_function(code,func,module_dir)

/home/pedrosoi/.local/lib/python2.7/site-packages/scipy/weave/inline_tools.pyc in compile_function(code, arg_names, local_dict, global_dict, module_dir, compiler, verbose, support_code, headers, customize, type_converters, auto_downcast, **kw)
    482     # setting.  All input keywords are passed through to distutils
    483     mod.compile(location=storage_dir,compiler=compiler,
--> 484                 verbose=verbose, **kw)
    485 
    486     # import the module and return the function.  Make sure

/home/pedrosoi/.local/lib/python2.7/site-packages/scipy/weave/ext_tools.pyc in compile(self, location, compiler, verbose, **kw)
    367         success = build_tools.build_extension(file, temp_dir = temp,
    368                                               compiler_name = compiler,
--> 369                                               verbose = verbose, **kw)
    370         if not success:
    371             raise SystemError('Compilation failed')

/home/pedrosoi/.local/lib/python2.7/site-packages/scipy/weave/build_tools.pyc in build_extension(module_path, compiler_name, build_dir, temp_dir, verbose, **kw)
    271         environ = copy.deepcopy(os.environ)
    272         try:
--> 273             setup(name = module_name, ext_modules = [ext],verbose=verb)
    274         finally:
    275             # restore state

/home/pedrosoi/.local/lib/python2.7/site-packages/numpy/distutils/core.pyc in setup(**attr)
    167     new_attr['distclass'] = NumpyDistribution
    168 
--> 169     return old_setup(**new_attr)
    170 
    171 def _check_append_library(libraries, item):

/home/pedrosoi/.local/lib/python2.7/distutils/core.pyc in setup(**attrs)
    167                 raise
    168             else:
--> 169                 raise SystemExit, "error: " + str(msg)
    170 
    171     return dist

CompileError: error: Command "g++ -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/pedrosoi/.local/lib/python2.7/site-packages/scipy/weave -I/home/pedrosoi/.local/lib/python2.7/site-packages/scipy/weave/scxx -I/home/pedrosoi/.local/lib/python2.7/site-packages/numpy/core/include -I/home/pedrosoi/.local/include/python2.7 -c /home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.cpp -o /tmp/pedrosoi/python27_intermediate/compiler_670789304ca6cb1709cb60244151fa66/home/pedrosoi/.python27_compiled/sc_72dc99e2266d31e07e15531a470f8a369.o -O3" failed with exit status 1

[inti]

I think found the potential culprit, casting the variables sort this out Inti

scipy.weave.inline(
                        '''
                        int tot = 0;
                        for (int i=0; i < int(N); i++) {
                            for (int j=0; j < int(data[i]); j++) {
                                tot += ((float) rand()) / RAND_MAX < ((float) r)/(j+r);
                            }
                        }
                        *msum = tot;
                        ''',
                        ['N','data','r','msum'],
                        extra_compile_args=['-O3'])

[mattjj]

I think I introduced that bug recently and I have it fixed on my local code; I'll push a change today in the Python code, though your casting fix may work as well.

Somehow I got to a version of flattendata that would call np.ma.concatenate, which cast things to MaskedArrays that weave can't automatically convert as it can with numpy arrays. (If you debug that line in the Python interpeter, you'll see data is an instance of np.ma.MaskedArray instead of an ndarray.)

Here's the updated flattendata in util.stats:

def flattendata(data):
    # data is either an array or a list of arrays
    if isinstance(data,np.ndarray):
        return data
    elif isinstance(data,list):
        if any(isinstance(d,np.ma.MaskedArray) for d in data):
            return np.ma.concatenate(data)
        else:
            return np.concatenate(data)
    else:
        assert isinstance(data,int) or isinstance(data,float)
        return np.array(data,ndmin=1)

[mattjj]

Fixed in ca24b3e