search

GeoscienceAustralia / eqrm

Automatically exported from code.google.com/p/eqrm
Other
5 stars 4 forks source link

g++ errors when running from a cleaned python environment #15

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. run python clean_all.py
2. run python check_scenarios.py

What is the expected output? What do you see instead?
check_scenarios runs successfully. A g++ error is instead seen:

u78240@tornado:/nas/gemd/georisk_models/earthquake/sandpits/ben/eqrm/trunk$ 
python check_scenarios.py 
C extension polygon_ext.c cannot be opened
NOTICE: C-extension polygon_ext.c not used
Pypar (version 2.1.4) initialised MPI OK with 1 processors
Running scenario TS_risk65.py
*******952:954**********
In file included from 
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/array-impl.h:37,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/array.h:26,
                 from /nas/users/u78240/unix/.python25_compiled/sc_7d69cda7fe7d1c8e8e05b7b896c9f8db0.cpp:11:
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/range.h: In 
member function 'bool blitz::Range::isAscendingContiguous() const':
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/range.h:120: 
warning: suggest parentheses around '&&' within '||'
In file included from 
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/applics.h:394,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/vecexpr.h:26,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/vecpick.cc:16,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/vecpick.h:287,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/vector.h:443,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/tinyvec.h:424,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/array-impl.h:38,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/array.h:26,
                 from /nas/users/u78240/unix/.python25_compiled/sc_7d69cda7fe7d1c8e8e05b7b896c9f8db0.cpp:11:
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/mathfunc.h: In 
static member function 'static long int blitz::_bz_abs<long int>::apply(long 
int)':
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/mathfunc.h:45: 
error: 'labs' is not a member of 'std'
In file included from 
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/array/funcs.h:23,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/array/newet.h:23,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/array/et.h:21,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/array-impl.h:2509,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/array.h:26,
                 from /nas/users/u78240/unix/.python25_compiled/sc_7d69cda7fe7d1c8e8e05b7b896c9f8db0.cpp:11:
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/funcs.h: In 
static member function 'static int blitz::Fn_abs<int>::apply(int)':
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/funcs.h:503: 
error: call of overloaded 'abs(int&)' is ambiguous
/usr/local/lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/
cmath:94: note: candidates are: double std::abs(double)
/usr/local/lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/
cmath:98: note:                 float std::abs(float)
/usr/local/lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/
cmath:102: note:                 long double std::abs(long double)
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/funcs.h: In 
static member function 'static long int blitz::Fn_abs<long int>::apply(long 
int)':
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/funcs.h:524: 
error: 'labs' is not a member of 'std'
/nas/users/u78240/unix/.python25_compiled/sc_7d69cda7fe7d1c8e8e05b7b896c9f8db0.c
pp: In function 'PyObject* compiled_func(PyObject*, PyObject*)':
/nas/users/u78240/unix/.python25_compiled/sc_7d69cda7fe7d1c8e8e05b7b896c9f8db0.c
pp:745: error: no match for call to '(py::object) (int&)'
In file included from 
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/array-impl.h:37,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/array.h:26,
                 from /nas/users/u78240/unix/.python25_compiled/sc_7d69cda7fe7d1c8e8e05b7b896c9f8db0.cpp:11:
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/range.h: In 
member function 'bool blitz::Range::isAscendingContiguous() const':
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/range.h:120: 
warning: suggest parentheses around '&&' within '||'
In file included from 
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/applics.h:394,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/vecexpr.h:26,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/vecpick.cc:16,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/vecpick.h:287,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/vector.h:443,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/tinyvec.h:424,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/array-impl.h:38,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/array.h:26,
                 from /nas/users/u78240/unix/.python25_compiled/sc_7d69cda7fe7d1c8e8e05b7b896c9f8db0.cpp:11:
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/mathfunc.h: In 
static member function 'static long int blitz::_bz_abs<long int>::apply(long 
int)':
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/mathfunc.h:45: 
error: 'labs' is not a member of 'std'
In file included from 
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/array/funcs.h:23,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/array/newet.h:23,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/array/et.h:21,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/array-impl.h:2509,
                 from /usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/array.h:26,
                 from /nas/users/u78240/unix/.python25_compiled/sc_7d69cda7fe7d1c8e8e05b7b896c9f8db0.cpp:11:
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/funcs.h: In 
static member function 'static int blitz::Fn_abs<int>::apply(int)':
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/funcs.h:503: 
error: call of overloaded 'abs(int&)' is ambiguous
/usr/local/lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/
cmath:94: note: candidates are: double std::abs(double)
/usr/local/lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/
cmath:98: note:                 float std::abs(float)
/usr/local/lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/
cmath:102: note:                 long double std::abs(long double)
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/funcs.h: In 
static member function 'static long int blitz::Fn_abs<long int>::apply(long 
int)':
/usr/local/lib/python2.5/site-packages/scipy/weave/blitz/blitz/funcs.h:524: 
error: 'labs' is not a member of 'std'
/nas/users/u78240/unix/.python25_compiled/sc_7d69cda7fe7d1c8e8e05b7b896c9f8db0.c
pp: In function 'PyObject* compiled_func(PyObject*, PyObject*)':
/nas/users/u78240/unix/.python25_compiled/sc_7d69cda7fe7d1c8e8e05b7b896c9f8db0.c
pp:745: error: no match for call to '(py::object) (int&)'

Traceback (most recent call last):
  File "check_scenarios.py", line 39, in <module>
    check_scenarios_main()
  File "/nas/gemd/georisk_models/earthquake/sandpits/ben/eqrm/trunk/eqrm_code/check_scenarios.py", line 539, in check_scenarios_main
    files=files)
  File "/nas/gemd/georisk_models/earthquake/sandpits/ben/eqrm/trunk/eqrm_code/check_scenarios.py", line 171, in run_scenarios
    analysis.main(pull_path, True)
  File "/nas/gemd/georisk_models/earthquake/sandpits/ben/eqrm/trunk/eqrm_code/analysis.py", line 453, in main
    bridge_SA_indices)
  File "/nas/gemd/georisk_models/earthquake/sandpits/ben/eqrm/trunk/eqrm_code/damage_model.py", line 405, in calc_total_loss
    loss_aus_contents=eqrm_flags.loss_aus_contents)
  File "/nas/gemd/georisk_models/earthquake/sandpits/ben/eqrm/trunk/eqrm_code/damage_model.py", line 184, in aggregated_building_loss
    self.building_loss(ci=ci, loss_aus_contents=loss_aus_contents)
  File "/nas/gemd/georisk_models/earthquake/sandpits/ben/eqrm/trunk/eqrm_code/damage_model.py", line 150, in building_loss
    damage_states = self.get_building_states()
  File "/nas/gemd/georisk_models/earthquake/sandpits/ben/eqrm/trunk/eqrm_code/damage_model.py", line 111, in get_building_states
    (SA, SD) = self.get_building_displacement()
  File "/nas/gemd/georisk_models/earthquake/sandpits/ben/eqrm/trunk/eqrm_code/damage_model.py", line 145, in get_building_displacement
    point = self.capacity_spectrum_model.building_response(self.SA)
  File "/nas/gemd/georisk_models/earthquake/sandpits/ben/eqrm/trunk/eqrm_code/capacity_spectrum_model.py", line 128, in building_response
    SA,SD,SAcap=self._damped_response(non_linear_damping=0)
  File "/nas/gemd/georisk_models/earthquake/sandpits/ben/eqrm/trunk/eqrm_code/capacity_spectrum_model.py", line 220, in _damped_response
    csm_damping_use_smoothing=self.csm_damping_use_smoothing)
  File "/nas/gemd/georisk_models/earthquake/sandpits/ben/eqrm/trunk/eqrm_code/capacity_spectrum_functions.py", line 492, in calculate_updated_demand
    compiler='gcc')
  File "/usr/local/lib/python2.5/site-packages/scipy/weave/inline_tools.py", line 335, in inline
    **kw)
  File "/usr/local/lib/python2.5/site-packages/scipy/weave/inline_tools.py", line 462, in compile_function
    verbose=verbose, **kw)
  File "/usr/local/lib/python2.5/site-packages/scipy/weave/ext_tools.py", line 365, in compile
    verbose = verbose, **kw)
  File "/usr/local/lib/python2.5/site-packages/scipy/weave/build_tools.py", line 272, in build_extension
    setup(name = module_name, ext_modules = [ext],verbose=verb)
  File "/usr/local/lib/python2.5/site-packages/numpy/distutils/core.py", line 184, in setup
    return old_setup(**new_attr)
  File "/usr/local/lib/python2.5/distutils/core.py", line 168, in setup
    raise SystemExit, "error: " + str(msg)
CompileError: error: Command "g++ -pthread -fno-strict-aliasing -DNDEBUG -g 
-fwrapv -O3 -Wall -fPIC -I/usr/local/lib/python2.5/site-packages/scipy/weave 
-I/usr/local/lib/python2.5/site-packages/scipy/weave/scxx 
-I/usr/local/lib/python2.5/site-packages/scipy/weave/blitz 
-I/usr/local/lib/python2.5/site-packages/numpy/core/include 
-I/usr/local/include/python2.5 -c 
/nas/users/u78240/unix/.python25_compiled/sc_7d69cda7fe7d1c8e8e05b7b896c9f8db0.c
pp -o 
/tmp/u78240/python25_intermediate/compiler_6bebbd2a17b4e26234087e3b8e378cf0/nas/
users/u78240/unix/.python25_compiled/sc_7d69cda7fe7d1c8e8e05b7b896c9f8db0.o" 
failed with exit status 1

Original issue reported on code.google.com by b...@girorosso.com on 24 Feb 2012 at 5:28

GoogleCodeExporter commented 9 years ago 
If I run the g++ command:
g++ -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -fPIC 
-I/usr/local/lib/python2.5/site-packages/scipy/weave 
-I/usr/local/lib/python2.5/site-packages/scipy/weave/scxx 
-I/usr/local/lib/python2.5/site-packages/scipy/weave/blitz 
-I/usr/local/lib/python2.5/site-packages/numpy/core/include 
-I/usr/local/include/python2.5 -c 
/nas/users/u78240/unix/.python25_compiled/sc_7d69cda7fe7d1c8e8e05b7b896c9f8db0.c
pp -o 
/tmp/u78240/python25_intermediate/compiler_eea14bca698e73bcb1037b044675db15/nas/
users/u78240/unix/.python25_compiled/sc_7d69cda7fe7d1c8e8e05b7b896c9f8db0.o

I get the following error:

/nas/users/u78240/unix/.python25_compiled/sc_7d69cda7fe7d1c8e8e05b7b896c9f8db0.c
pp: In function 'PyObject* compiled_func(PyObject*, PyObject*)':
/nas/users/u78240/unix/.python25_compiled/sc_7d69cda7fe7d1c8e8e05b7b896c9f8db0.c
pp:745: error: no match for call to '(py::object) (int&)'

Original comment by b...@girorosso.com on 24 Feb 2012 at 5:59

GoogleCodeExporter commented 9 years ago 
If I switch to revision 886 check_scenarios runs fine. If I then switch to the 
current revision, without cleaning the environment, check_scenarios also runs 
fine. Investigating further.

Original comment by b...@girorosso.com on 24 Feb 2012 at 6:08

GoogleCodeExporter commented 9 years ago 
Tracked down source of failure to revision 915. Looking to see what changed 
here to cause this.

Original comment by b...@girorosso.com on 26 Feb 2012 at 11:00

GoogleCodeExporter commented 9 years ago 
As an update on comment 3, this is specifically caused by opening the file 
based array as an memmap object. i.e.

    def _get_numpy_binary_array(self, name):
        """Return the an memmap object as represented by the .npy file"""
        filename = self._array_files.get(name)  
        if filename is not None:
            #return load(filename)
            return open_memmap(filename)

If I comment out the call to open_memmap and uncomment the call to load the 
problem goes away.

The issue here is that we want to use a memmap object as slice assignment won't 
work otherwise. Looking into what we can do to resolve this.

Original comment by b...@girorosso.com on 26 Feb 2012 at 11:12

GoogleCodeExporter commented 9 years ago 
The theory is that weave functions do not cope well with memmap objects. 

For this particular one, called in 
capacity_spectrum_functions.calculate_updated_demand, an memmap object 'TVD' is 
derived from the magnitudes vector in the event set. 

If I turn this into an in memory array by using asarray, e.g.

TVD = asarray(TVD)

this passes the function successfully (and fails at a different weave function 
called later).

This was successful until clean_all.py called as the compiled weave functions 
get cached, so the old version was being called. This worked as 'TVD' is a 
ndarray-like object.

Original comment by b...@girorosso.com on 27 Feb 2012 at 12:31

GoogleCodeExporter commented 9 years ago 
All uses of weave.inline use type_converters=converters.blitz. This does not 
contain a converter for numpy.memmap so it defaults to the standard python 
object converter. A python object cannot be used in the same way so the 
compiler throws the error.

This is resolved by adding to the converters list a custom numpy.memmap 
converter that simply subclasses the blitz array converter and assigns 
numpy.memmap to the matching types, like so:

class memmap_converter(blitz_spec.array_converter):
    def init_info(self):
        blitz_spec.array_converter.init_info(self)
        self.matching_types = [numpy.memmap]

# Use the blitz converters already used for weave inlines
# and add to start of list so we don't get the catchall
eqrm_converters = [memmap_converter()] + converters.blitz

and in use:

            weave.inline(code,
                         ['num_sites','num_events','num_periods',
                          'Ra','Rv','Rd','R','periods',
                          'TAV','TVD'],
                         type_converters=weave_converters.eqrm_converters,
                         compiler='gcc')

Original comment by b...@girorosso.com on 27 Feb 2012 at 3:36

GoogleCodeExporter commented 9 years ago 
The fix described above is implemented in revision 963.

Original comment by b...@girorosso.com on 27 Feb 2012 at 3:57