BUG/NotImplemented datashape conversion from void?

[jreback]

In [1]: from datashape import dshape

In [2]: from dynd import ndt

In [3]: dshape(ndt.type('?int64').dshape)
Out[3]: dshape("?int64")

Is this just not implemented?

In [4]: dshape(ndt.type('?void').dshape)

  File <nofile>, line 1
    ?void
     ^

DataShapeSyntaxError: Invalid datashape

---------------------------------------------------------------------------
DataShapeSyntaxError                      Traceback (most recent call last)
<ipython-input-4-43248ddfb0e7> in <module>()
----> 1 dshape(ndt.type('?void').dshape)

/Users/jreback/anaconda/envs/dynd/lib/python2.7/site-packages/datashape/util.pyc in dshape(o)
     47         return o
     48     if isinstance(o, py2help._strtypes):
---> 49         ds = parser.parse(o, type_symbol_table.sym)
     50     elif isinstance(o, (coretypes.CType, coretypes.String,
     51                         coretypes.Record, coretypes.JSON,

/Users/jreback/anaconda/envs/dynd/lib/python2.7/site-packages/datashape/parser.pyc in parse(ds_str, sym)
    573     # If no datashape could be found
    574     if ds is None:
--> 575         dsp.raise_error('Invalid datashape')
    576 
    577     # Make sure there's no garbage at the end

/Users/jreback/anaconda/envs/dynd/lib/python2.7/site-packages/datashape/parser.pyc in raise_error(self, errmsg)
     55     def raise_error(self, errmsg):
     56         raise error.DataShapeSyntaxError(self.tok.span[0], '<nofile>',
---> 57                                          self.ds_str, errmsg)
     58 
     59     def parse_homogeneous_list(self, parse_item, sep_tok_id, errmsg,

DataShapeSyntaxError: 

  File <nofile>, line 1
    ?void
     ^

DataShapeSyntaxError: Invalid datashape

cc @cpcloud cc @mrocklin

[jreback]

I am just trying to avoid special casing things..

In [6]: from dynd import nd

In [7]: t = nd.array([None,None])
Out[7]: 
nd.array([NA, NA],
         type="2 * ?void")

I am ultimately going to coerce this to something else, but I have to figure out the base dtype

so I do

ds = dshape(t).measure
ds = getattr(ds,'ty',ds)
ds.to_numpy_dtype()

which is working of all dtypes so far (except void)

[cpcloud]

Can you show how you arrive at a void dshape?

[cpcloud]

I'd also like to know what void is used for. In numpy it's used as the type of a row in a structured array

In [6]: x = np.array([(1, 2.0), (3, 4.0)], dtype=[('a', 'i8'), ('b', 'f8')])

In [7]: x
Out[7]:
array([(1, 2.0), (3, 4.0)],
      dtype=[('a', '<i8'), ('b', '<f8')])

In [8]: x[0]
Out[8]: (1, 2.0)

In [9]: type(x[0])
Out[9]: numpy.void

but this isn't necessary in datashape because we have Record types.

[jreback]

In [7]: t = nd.array([None,None])
Out[7]: 
nd.array([NA, NA],
         type="2 * ?void")

ultimately I am going to coerce this to numpy object, but its acting differently than the other dtypes.

[cpcloud]

What version of dynd-python are you using? I get an error when trying to construct that array

[jreback]

I am a little behind master, but not that much (these from about 1 week ago)

dynd: 0.6.6.post148.g7e2c94e [0.6.6.post673.gf089f7c] 2nd is libdynd

these are built by me :) as osx not up on binstar :<

[izaid]

The void type in DyND is used for functions that don't return anything. The option void "?void" type is used for optional arguments in functions whose types are unable to resolve themselves. I don't think we use either for anything concrete.

This is probably another case of DyND leaping ahead of datashape...

[cpcloud]

@jreback What case does this arise in? When there's an array of all missing values?

[jreback]

its possible to construct an array full of None values (see above). In numpy this would be object dtype.

In [4]: from datashape import discover

In [5]: discover(np.array([None,None]))
Out[5]: dshape("2 * object")

the dynd equiv seem to be 'void' type.

[jreback]

side issue, just rebuilt libdynd from master - looks good, but

(dynd)bash-3.2$ python setup.py build
running build
running build_py
running build_ext
cmake -DPYTHON_EXECUTABLE=/Users/jreback/anaconda/envs/dynd/bin/python -DDYND_INSTALL_LIB=ON   /Users/jreback/dynd-python
CUDA_TOOLKIT_ROOT_DIR not found or specified
-- Could NOT find CUDA (missing:  CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) (Required is at least version "6.5")
-- Getting git information at: /Users/jreback/dynd-python
-- Getting git information at: /Users/jreback/dynd-python
-- Arguments to /usr/bin/git: --dirty;--always;--match;v*
-- DyND-Python version: v0.6.6-179-g3735673
-- NOT adding relative rpath to _pydynd for linking libdynd
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7
make
[  3%] Building CXX object CMakeFiles/dynd._pydynd.dir/dynd._pydynd.cxx.o
In file included from /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/dynd._pydynd.cxx:249:
In file included from /Users/jreback/dynd-python/dynd/include/numpy_interop.hpp:43:
In file included from /usr/local/include/dynd/type.hpp:11:
In file included from /usr/local/include/dynd/types/base_type.hpp:10:
/usr/local/include/dynd/config.hpp:14:10: fatal error: 'initializer_list' file not found
#include <initializer_list>
         ^
1 error generated.
make[2]: *** [CMakeFiles/dynd._pydynd.dir/dynd._pydynd.cxx.o] Error 1
make[1]: *** [CMakeFiles/dynd._pydynd.dir/all] Error 2
make: *** [all] Error 2
error: command 'make' failed with exit status 2

[cpcloud]

If void is the type used for functions that don't return anything, then how does one read ?void? "Not-anything-or-missing"? Shouldn't the dshape of nd.array([None, None]) be object here?

[izaid]

Ah, okay. I don't believe we've implemented the equivalent of a type for arbitrary objects. The usage of void there is a temporary stand-in until we actually do! So object types are unsupported.

[izaid]

What exactly is it that you need to do? What you want may exist in another form...

[jreback]

@izaid

I have a routine which takes a nd (or a numpy dtype) and gives me back the numpy base dtype that I need, e.g. ?int64 -> int64 and such. I need to compare using certain routines which are numpy based at this point (and it works well enough).

So I was creating nd.array([None,None]) which I want to return object from this, but the coercion routine was bombing.

I can work-around for now.

[izaid]

As it currently stands, "void" is read as "no type". "?void" is akin to "unresolvable type" (or "missing type").

When we implemented unresolvable types, "?void" seemed like the most sensible way to do it. The other possibility was optional symbolic types, which mean symbolic types had to become instantiable, which seemed really weird.

[cpcloud]

@izaid cool! just wanted to clarify. thanks for clearing it up

[izaid]

@jreback Your issue is strange. Doing a rebuild from master on my Mac now...

[jreback]

@izaid thanks, lmk (btw the new python setup.py build def way to go!)

[izaid]

@jreback Everything worked flawless for me. Can you give me details of your setup? Are you sure you're working with master?

[jreback]

can you show me your build commands for both libdynd / dynd-python?

[izaid]

libdynd: cmake .. then make dynd-python: sudo python setup.py install

[izaid]

Oh, and also make sure dynd-python can find your libdynd. Can just install it with "sudo make install".

[jreback]

libdynd looks good and looks like it installs. how does it know where to find libdynd though?

[jreback]

(dynd)bash-3.2$ libdynd-config -libpath
/usr/local/lib/libdynd.dylib
(dynd)bash-3.2$ libdynd-config -version
v0.6.6-717-gf2d6937

[izaid]

Can you give dynd-python a shot now and tell me how it goes?

[izaid]

Should be found automatically.

[jreback]

still same problem. my libdynd looks good (see above)

[izaid]

Which compiler? Is this on Mac or Linux?

[jreback]

mac

just noticed that dynd-python HEAD is a bit behind libdydn (though I just updated both from master)

[jreback]

/Users/jreback/dynd-python
(dynd)bash-3.2$ git log
commit 373567342dca8e08a8b4ad23e11179e97abbd03f
Author: Mark Wiebe <mwwiebe@gmail.com>
Date:   Mon Mar 2 03:37:06 2015 -0800

    Remove the checkout depth limit, so the version # is visible

commit d35715c799fa4eb4a0b3491c67f85be0ac59b722

[jreback]

for libdydn

commit f2d6937c0f1d73ae7dce89f96f911cd31175073e
Merge: 1d6ce54 0ae7d9c
Author: Irwin Zaid <izaid@users.noreply.github.com>
Date:   Sun Mar 8 18:25:58 2015 +0000

    Merge pull request #330 from izaid/kernels

    CMake

[jreback]

possible you have local commits that I am not seeing?

[mwiebe]

@jreback the header is a C++11 header, could it be that the standard library your C++ compiler is using is too old? What compiler version are you running?

[mwiebe]

Or perhaps dynd-python is somehow seeing a different compiler than libdynd?

[izaid]

Which compiler on Mac? Clang?

[jreback]

(dynd)bash-3.2$ cc --version
Apple LLVM version 6.0 (clang-600.0.56) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin13.4.0
Thread model: posix

[izaid]

What's disturbing is that I have the same...

Apple LLVM version 6.0 (clang-600.0.56) (based on LLVM 3.5svn) Target: x86_64-apple-darwin14.0.0 Thread model: posix

[jreback]

the commits appear to differ, no?

e.g. mar 2 for dynd-python but mar 8 for libpython, are they in sync?

[izaid]

That's okay, we update them separately.

[izaid]

If I had to guess, the problem is something like your Clang isn't getting the "-std=c++11" flag. Can you look at the compile commands when make fails and tell me if this is the case? (Or just send me the failed build commands.)

[jreback]

the only output I had posted above. Is there a way to make it show the actual build command?

[izaid]

Yep. For the purposes of this debugging, don't use setup.py. Instead do "cmake -DCMAKE_VERBOSE_MAKEFILE=TRUE .." and then "make".

[jreback]

This worked beautifully (lots of output of course); this is what I was doing before you had setup.py (well was specifying the -DCMAKE_BUILD_TYPE=RelWithDebInfo but same idea)

(dynd)bash-3.2$ cmake -DCMAKE_VERBOSE_MAKEFILE=TRUE .. -DPYTHON_PACKAGE_INSTALL_PREFIX=/Users/jreback/anaconda/envs/dynd/lib/python2.7/site-packages

dynd: v0.6.6-179-g3735673
libdynd: v0.6.6-717-gf2d6937

[izaid]

Okay, I really don't understand what's going on here -- setup.py actually just calls our CMake script. Any ideas, @mwiebe

[mwiebe]

Maybe there's some difference in how setup.py is calling cmake? @jreback maybe if you add a debug print to the setup.py so that it prints out the cmake command it's using, so you can try it in exactly the same form outside of that context, but with the verbose option added?

[jreback]

(Pdb) p self.spawn(['pwd'])
pwd
/Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7
None

(Pdb) p ' '.join(['cmake', pyexe_option, install_lib_option,static_lib_option, cuda_option, source])
'cmake -DPYTHON_EXECUTABLE=/Users/jreback/anaconda/envs/dynd/bin/python -DDYND_INSTALL_LIB=ON   /Users/jreback/dynd-python'

[jreback]

(dynd)bash-3.2$ cmake -DPYTHON_EXECUTABLE=/Users/jreback/anaconda/envs/dynd/bin/python -DDYND_INSTALL_LIB=ON   /Users/jreback/dynd-python -DCMAKE_VERBOSE_MAKEFILE=TRUE 
CUDA_TOOLKIT_ROOT_DIR not found or specified
-- Could NOT find CUDA (missing:  CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) (Required is at least version "6.5")
-- Getting git information at: /Users/jreback/dynd-python
-- Getting git information at: /Users/jreback/dynd-python
-- Arguments to /usr/bin/git: --dirty;--always;--match;v*
-- DyND-Python version: v0.6.6-179-g3735673-dirty
-- NOT adding relative rpath to _pydynd for linking libdynd
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7
(dynd)bash-3.2$ make
/usr/local/Cellar/cmake/3.0.2/bin/cmake -H/Users/jreback/dynd-python -B/Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7 --check-build-system CMakeFiles/Makefile.cmake 0
/usr/local/Cellar/cmake/3.0.2/bin/cmake -E cmake_progress_start /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/CMakeFiles /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/CMakeFiles/progress.marks
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/Makefile2 all
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/dynd._pydynd.dir/build.make CMakeFiles/dynd._pydynd.dir/depend
cd /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7 && /usr/local/Cellar/cmake/3.0.2/bin/cmake -E cmake_depends "Unix Makefiles" /Users/jreback/dynd-python /Users/jreback/dynd-python /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7 /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7 /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/CMakeFiles/dynd._pydynd.dir/DependInfo.cmake --color=
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/dynd._pydynd.dir/build.make CMakeFiles/dynd._pydynd.dir/build
/usr/local/Cellar/cmake/3.0.2/bin/cmake -E cmake_progress_report /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/CMakeFiles 2
[  3%] Building CXX object CMakeFiles/dynd._pydynd.dir/dynd._pydynd.cxx.o
/usr/bin/c++   -Ddynd__pydynd_EXPORTS -std=c++11 -g -fomit-frame-pointer -fstrict-aliasing -fPIC -Wall -Wextra -Werror -Wno-missing-field-initializers -ferror-limit=20 -Wdocumentation -O2 -g -DNDEBUG -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk -mmacosx-version-min=10.5 -fPIC -I/Users/jreback/anaconda/envs/dynd/lib/python2.7/site-packages/numpy/core/include -I/Users/jreback/anaconda/envs/dynd/include/python2.7 -I/usr/local/include -I/Users/jreback/dynd-python/dynd/include -I/Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/dynd    -Wno-unused-parameter -Wno-unused-function -Wno-error -o CMakeFiles/dynd._pydynd.dir/dynd._pydynd.cxx.o -c /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/dynd._pydynd.cxx
In file included from /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/dynd._pydynd.cxx:249:
In file included from /Users/jreback/dynd-python/dynd/include/numpy_interop.hpp:43:
In file included from /usr/local/include/dynd/type.hpp:11:
In file included from /usr/local/include/dynd/types/base_type.hpp:10:
/usr/local/include/dynd/config.hpp:14:10: fatal error: 'initializer_list' file not found
#include <initializer_list>
         ^
1 error generated.
make[2]: *** [CMakeFiles/dynd._pydynd.dir/dynd._pydynd.cxx.o] Error 1
make[1]: *** [CMakeFiles/dynd._pydynd.dir/all] Error 2
make: *** [all] Error 2

[izaid]

Well, the -std=c++11 flag seems to be there...

Another guess: I wonder if you need the -stdlib=libc++ flag as well... Can you modify the dynd-python CMakeLists.txt to pass -stdlib=libc++?

[jreback]

odd error with that addtl flag

(dynd)bash-3.2$ python setup.py build
running build
running build_py
running build_ext
cmake -DPYTHON_EXECUTABLE=/Users/jreback/anaconda/envs/dynd/bin/python -DDYND_INSTALL_LIB=ON   /Users/jreback/dynd-python
CUDA_TOOLKIT_ROOT_DIR not found or specified
-- Could NOT find CUDA (missing:  CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) (Required is at least version "6.5")
-- Getting git information at: /Users/jreback/dynd-python
-- Getting git information at: /Users/jreback/dynd-python
-- Arguments to /usr/bin/git: --dirty;--always;--match;v*
-- DyND-Python version: v0.6.6-179-g3735673-dirty
-- NOT adding relative rpath to _pydynd for linking libdynd
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7
make
/usr/local/Cellar/cmake/3.0.2/bin/cmake -H/Users/jreback/dynd-python -B/Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7 --check-build-system CMakeFiles/Makefile.cmake 0
/usr/local/Cellar/cmake/3.0.2/bin/cmake -E cmake_progress_start /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/CMakeFiles /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/CMakeFiles/progress.marks
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/Makefile2 all
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/dynd._pydynd.dir/build.make CMakeFiles/dynd._pydynd.dir/depend
cd /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7 && /usr/local/Cellar/cmake/3.0.2/bin/cmake -E cmake_depends "Unix Makefiles" /Users/jreback/dynd-python /Users/jreback/dynd-python /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7 /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7 /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/CMakeFiles/dynd._pydynd.dir/DependInfo.cmake --color=
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/dynd._pydynd.dir/build.make CMakeFiles/dynd._pydynd.dir/build
/usr/local/Cellar/cmake/3.0.2/bin/cmake -E cmake_progress_report /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/CMakeFiles 2
[  3%] Building CXX object CMakeFiles/dynd._pydynd.dir/dynd._pydynd.cxx.o
/usr/bin/c++   -Ddynd__pydynd_EXPORTS -std=c++11 -stdlib=libc++ -g -fomit-frame-pointer -fstrict-aliasing -fPIC -Wall -Wextra -Werror -Wno-missing-field-initializers -ferror-limit=20 -Wdocumentation -O2 -g -DNDEBUG -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk -mmacosx-version-min=10.5 -fPIC -I/Users/jreback/anaconda/envs/dynd/lib/python2.7/site-packages/numpy/core/include -I/Users/jreback/anaconda/envs/dynd/include/python2.7 -I/usr/local/include -I/Users/jreback/dynd-python/dynd/include -I/Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/dynd    -Wno-unused-parameter -Wno-unused-function -Wno-error -o CMakeFiles/dynd._pydynd.dir/dynd._pydynd.cxx.o -c /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/dynd._pydynd.cxx
clang: error: invalid deployment target for -stdlib=libc++ (requires OS X 10.7 or later)
make[2]: *** [CMakeFiles/dynd._pydynd.dir/dynd._pydynd.cxx.o] Error 1
make[1]: *** [CMakeFiles/dynd._pydynd.dir/all] Error 2
make: *** [all] Error 2
error: command 'make' failed with exit status 2

[izaid]

Are you on OS X 10.5?