Open jreback opened 9 years ago
I am just trying to avoid special casing things..
In [6]: from dynd import nd
In [7]: t = nd.array([None,None])
Out[7]:
nd.array([NA, NA],
type="2 * ?void")
I am ultimately going to coerce this to something else, but I have to figure out the base dtype
so I do
ds = dshape(t).measure
ds = getattr(ds,'ty',ds)
ds.to_numpy_dtype()
which is working of all dtypes so far (except void)
Can you show how you arrive at a void
dshape?
I'd also like to know what void
is used for. In numpy
it's used as the type of a row in a structured array
In [6]: x = np.array([(1, 2.0), (3, 4.0)], dtype=[('a', 'i8'), ('b', 'f8')])
In [7]: x
Out[7]:
array([(1, 2.0), (3, 4.0)],
dtype=[('a', '<i8'), ('b', '<f8')])
In [8]: x[0]
Out[8]: (1, 2.0)
In [9]: type(x[0])
Out[9]: numpy.void
but this isn't necessary in datashape
because we have Record
types.
In [7]: t = nd.array([None,None])
Out[7]:
nd.array([NA, NA],
type="2 * ?void")
ultimately I am going to coerce this to numpy object
, but its acting differently than the other dtypes.
What version of dynd-python
are you using? I get an error when trying to construct that array
I am a little behind master, but not that much (these from about 1 week ago)
dynd: 0.6.6.post148.g7e2c94e [0.6.6.post673.gf089f7c]
2nd is libdynd
these are built by me :) as osx not up on binstar :<
The void type in DyND is used for functions that don't return anything. The option void "?void" type is used for optional arguments in functions whose types are unable to resolve themselves. I don't think we use either for anything concrete.
This is probably another case of DyND leaping ahead of datashape...
@jreback What case does this arise in? When there's an array of all missing values?
its possible to construct an array full of None
values (see above). In numpy this would be object
dtype.
In [4]: from datashape import discover
In [5]: discover(np.array([None,None]))
Out[5]: dshape("2 * object")
the dynd equiv seem to be 'void' type.
side issue, just rebuilt libdynd from master - looks good, but
(dynd)bash-3.2$ python setup.py build
running build
running build_py
running build_ext
cmake -DPYTHON_EXECUTABLE=/Users/jreback/anaconda/envs/dynd/bin/python -DDYND_INSTALL_LIB=ON /Users/jreback/dynd-python
CUDA_TOOLKIT_ROOT_DIR not found or specified
-- Could NOT find CUDA (missing: CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) (Required is at least version "6.5")
-- Getting git information at: /Users/jreback/dynd-python
-- Getting git information at: /Users/jreback/dynd-python
-- Arguments to /usr/bin/git: --dirty;--always;--match;v*
-- DyND-Python version: v0.6.6-179-g3735673
-- NOT adding relative rpath to _pydynd for linking libdynd
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7
make
[ 3%] Building CXX object CMakeFiles/dynd._pydynd.dir/dynd._pydynd.cxx.o
In file included from /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/dynd._pydynd.cxx:249:
In file included from /Users/jreback/dynd-python/dynd/include/numpy_interop.hpp:43:
In file included from /usr/local/include/dynd/type.hpp:11:
In file included from /usr/local/include/dynd/types/base_type.hpp:10:
/usr/local/include/dynd/config.hpp:14:10: fatal error: 'initializer_list' file not found
#include <initializer_list>
^
1 error generated.
make[2]: *** [CMakeFiles/dynd._pydynd.dir/dynd._pydynd.cxx.o] Error 1
make[1]: *** [CMakeFiles/dynd._pydynd.dir/all] Error 2
make: *** [all] Error 2
error: command 'make' failed with exit status 2
If void
is the type used for functions that don't return anything, then how does one read ?void
? "Not-anything-or-missing"? Shouldn't the dshape of nd.array([None, None])
be object
here?
Ah, okay. I don't believe we've implemented the equivalent of a type for arbitrary objects. The usage of void there is a temporary stand-in until we actually do! So object types are unsupported.
What exactly is it that you need to do? What you want may exist in another form...
@izaid
I have a routine which takes a nd (or a numpy dtype) and gives me back the numpy base dtype that I need, e.g. ?int64
-> int64
and such. I need to compare using certain routines which are numpy based at this point (and it works well enough).
So I was creating nd.array([None,None])
which I want to return object
from this, but the coercion routine was bombing.
I can work-around for now.
As it currently stands, "void" is read as "no type". "?void" is akin to "unresolvable type" (or "missing type").
When we implemented unresolvable types, "?void" seemed like the most sensible way to do it. The other possibility was optional symbolic types, which mean symbolic types had to become instantiable, which seemed really weird.
@izaid cool! just wanted to clarify. thanks for clearing it up
@jreback Your issue is strange. Doing a rebuild from master on my Mac now...
@izaid thanks, lmk (btw the new python setup.py build
def way to go!)
@jreback Everything worked flawless for me. Can you give me details of your setup? Are you sure you're working with master?
can you show me your build commands for both libdynd / dynd-python?
libdynd: cmake .. then make dynd-python: sudo python setup.py install
Oh, and also make sure dynd-python can find your libdynd. Can just install it with "sudo make install".
libdynd looks good and looks like it installs. how does it know where to find libdynd though?
(dynd)bash-3.2$ libdynd-config -libpath
/usr/local/lib/libdynd.dylib
(dynd)bash-3.2$ libdynd-config -version
v0.6.6-717-gf2d6937
Can you give dynd-python a shot now and tell me how it goes?
Should be found automatically.
still same problem. my libdynd looks good (see above)
Which compiler? Is this on Mac or Linux?
mac
just noticed that dynd-python HEAD is a bit behind libdydn (though I just updated both from master)
/Users/jreback/dynd-python
(dynd)bash-3.2$ git log
commit 373567342dca8e08a8b4ad23e11179e97abbd03f
Author: Mark Wiebe <mwwiebe@gmail.com>
Date: Mon Mar 2 03:37:06 2015 -0800
Remove the checkout depth limit, so the version # is visible
commit d35715c799fa4eb4a0b3491c67f85be0ac59b722
for libdydn
commit f2d6937c0f1d73ae7dce89f96f911cd31175073e
Merge: 1d6ce54 0ae7d9c
Author: Irwin Zaid <izaid@users.noreply.github.com>
Date: Sun Mar 8 18:25:58 2015 +0000
Merge pull request #330 from izaid/kernels
CMake
possible you have local commits that I am not seeing?
@jreback the
Or perhaps dynd-python is somehow seeing a different compiler than libdynd?
Which compiler on Mac? Clang?
(dynd)bash-3.2$ cc --version
Apple LLVM version 6.0 (clang-600.0.56) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin13.4.0
Thread model: posix
What's disturbing is that I have the same...
Apple LLVM version 6.0 (clang-600.0.56) (based on LLVM 3.5svn) Target: x86_64-apple-darwin14.0.0 Thread model: posix
the commits appear to differ, no?
e.g. mar 2 for dynd-python but mar 8 for libpython, are they in sync?
That's okay, we update them separately.
If I had to guess, the problem is something like your Clang isn't getting the "-std=c++11" flag. Can you look at the compile commands when make fails and tell me if this is the case? (Or just send me the failed build commands.)
the only output I had posted above. Is there a way to make it show the actual build command?
Yep. For the purposes of this debugging, don't use setup.py. Instead do "cmake -DCMAKE_VERBOSE_MAKEFILE=TRUE .." and then "make".
This worked beautifully (lots of output of course); this is what I was doing before you had setup.py (well was specifying the -DCMAKE_BUILD_TYPE=RelWithDebInfo but same idea)
(dynd)bash-3.2$ cmake -DCMAKE_VERBOSE_MAKEFILE=TRUE .. -DPYTHON_PACKAGE_INSTALL_PREFIX=/Users/jreback/anaconda/envs/dynd/lib/python2.7/site-packages
dynd: v0.6.6-179-g3735673
libdynd: v0.6.6-717-gf2d6937
Okay, I really don't understand what's going on here -- setup.py actually just calls our CMake script. Any ideas, @mwiebe
Maybe there's some difference in how setup.py is calling cmake? @jreback maybe if you add a debug print to the setup.py so that it prints out the cmake command it's using, so you can try it in exactly the same form outside of that context, but with the verbose option added?
(Pdb) p self.spawn(['pwd'])
pwd
/Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7
None
(Pdb) p ' '.join(['cmake', pyexe_option, install_lib_option,static_lib_option, cuda_option, source])
'cmake -DPYTHON_EXECUTABLE=/Users/jreback/anaconda/envs/dynd/bin/python -DDYND_INSTALL_LIB=ON /Users/jreback/dynd-python'
(dynd)bash-3.2$ cmake -DPYTHON_EXECUTABLE=/Users/jreback/anaconda/envs/dynd/bin/python -DDYND_INSTALL_LIB=ON /Users/jreback/dynd-python -DCMAKE_VERBOSE_MAKEFILE=TRUE
CUDA_TOOLKIT_ROOT_DIR not found or specified
-- Could NOT find CUDA (missing: CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) (Required is at least version "6.5")
-- Getting git information at: /Users/jreback/dynd-python
-- Getting git information at: /Users/jreback/dynd-python
-- Arguments to /usr/bin/git: --dirty;--always;--match;v*
-- DyND-Python version: v0.6.6-179-g3735673-dirty
-- NOT adding relative rpath to _pydynd for linking libdynd
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7
(dynd)bash-3.2$ make
/usr/local/Cellar/cmake/3.0.2/bin/cmake -H/Users/jreback/dynd-python -B/Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7 --check-build-system CMakeFiles/Makefile.cmake 0
/usr/local/Cellar/cmake/3.0.2/bin/cmake -E cmake_progress_start /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/CMakeFiles /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/CMakeFiles/progress.marks
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/Makefile2 all
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/dynd._pydynd.dir/build.make CMakeFiles/dynd._pydynd.dir/depend
cd /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7 && /usr/local/Cellar/cmake/3.0.2/bin/cmake -E cmake_depends "Unix Makefiles" /Users/jreback/dynd-python /Users/jreback/dynd-python /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7 /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7 /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/CMakeFiles/dynd._pydynd.dir/DependInfo.cmake --color=
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/dynd._pydynd.dir/build.make CMakeFiles/dynd._pydynd.dir/build
/usr/local/Cellar/cmake/3.0.2/bin/cmake -E cmake_progress_report /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/CMakeFiles 2
[ 3%] Building CXX object CMakeFiles/dynd._pydynd.dir/dynd._pydynd.cxx.o
/usr/bin/c++ -Ddynd__pydynd_EXPORTS -std=c++11 -g -fomit-frame-pointer -fstrict-aliasing -fPIC -Wall -Wextra -Werror -Wno-missing-field-initializers -ferror-limit=20 -Wdocumentation -O2 -g -DNDEBUG -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk -mmacosx-version-min=10.5 -fPIC -I/Users/jreback/anaconda/envs/dynd/lib/python2.7/site-packages/numpy/core/include -I/Users/jreback/anaconda/envs/dynd/include/python2.7 -I/usr/local/include -I/Users/jreback/dynd-python/dynd/include -I/Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/dynd -Wno-unused-parameter -Wno-unused-function -Wno-error -o CMakeFiles/dynd._pydynd.dir/dynd._pydynd.cxx.o -c /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/dynd._pydynd.cxx
In file included from /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/dynd._pydynd.cxx:249:
In file included from /Users/jreback/dynd-python/dynd/include/numpy_interop.hpp:43:
In file included from /usr/local/include/dynd/type.hpp:11:
In file included from /usr/local/include/dynd/types/base_type.hpp:10:
/usr/local/include/dynd/config.hpp:14:10: fatal error: 'initializer_list' file not found
#include <initializer_list>
^
1 error generated.
make[2]: *** [CMakeFiles/dynd._pydynd.dir/dynd._pydynd.cxx.o] Error 1
make[1]: *** [CMakeFiles/dynd._pydynd.dir/all] Error 2
make: *** [all] Error 2
Well, the -std=c++11 flag seems to be there...
Another guess: I wonder if you need the -stdlib=libc++ flag as well... Can you modify the dynd-python CMakeLists.txt to pass -stdlib=libc++?
odd error with that addtl flag
(dynd)bash-3.2$ python setup.py build
running build
running build_py
running build_ext
cmake -DPYTHON_EXECUTABLE=/Users/jreback/anaconda/envs/dynd/bin/python -DDYND_INSTALL_LIB=ON /Users/jreback/dynd-python
CUDA_TOOLKIT_ROOT_DIR not found or specified
-- Could NOT find CUDA (missing: CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) (Required is at least version "6.5")
-- Getting git information at: /Users/jreback/dynd-python
-- Getting git information at: /Users/jreback/dynd-python
-- Arguments to /usr/bin/git: --dirty;--always;--match;v*
-- DyND-Python version: v0.6.6-179-g3735673-dirty
-- NOT adding relative rpath to _pydynd for linking libdynd
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7
make
/usr/local/Cellar/cmake/3.0.2/bin/cmake -H/Users/jreback/dynd-python -B/Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7 --check-build-system CMakeFiles/Makefile.cmake 0
/usr/local/Cellar/cmake/3.0.2/bin/cmake -E cmake_progress_start /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/CMakeFiles /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/CMakeFiles/progress.marks
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/Makefile2 all
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/dynd._pydynd.dir/build.make CMakeFiles/dynd._pydynd.dir/depend
cd /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7 && /usr/local/Cellar/cmake/3.0.2/bin/cmake -E cmake_depends "Unix Makefiles" /Users/jreback/dynd-python /Users/jreback/dynd-python /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7 /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7 /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/CMakeFiles/dynd._pydynd.dir/DependInfo.cmake --color=
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/dynd._pydynd.dir/build.make CMakeFiles/dynd._pydynd.dir/build
/usr/local/Cellar/cmake/3.0.2/bin/cmake -E cmake_progress_report /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/CMakeFiles 2
[ 3%] Building CXX object CMakeFiles/dynd._pydynd.dir/dynd._pydynd.cxx.o
/usr/bin/c++ -Ddynd__pydynd_EXPORTS -std=c++11 -stdlib=libc++ -g -fomit-frame-pointer -fstrict-aliasing -fPIC -Wall -Wextra -Werror -Wno-missing-field-initializers -ferror-limit=20 -Wdocumentation -O2 -g -DNDEBUG -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk -mmacosx-version-min=10.5 -fPIC -I/Users/jreback/anaconda/envs/dynd/lib/python2.7/site-packages/numpy/core/include -I/Users/jreback/anaconda/envs/dynd/include/python2.7 -I/usr/local/include -I/Users/jreback/dynd-python/dynd/include -I/Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/dynd -Wno-unused-parameter -Wno-unused-function -Wno-error -o CMakeFiles/dynd._pydynd.dir/dynd._pydynd.cxx.o -c /Users/jreback/dynd-python/build/temp.macosx-10.5-x86_64-2.7/dynd._pydynd.cxx
clang: error: invalid deployment target for -stdlib=libc++ (requires OS X 10.7 or later)
make[2]: *** [CMakeFiles/dynd._pydynd.dir/dynd._pydynd.cxx.o] Error 1
make[1]: *** [CMakeFiles/dynd._pydynd.dir/all] Error 2
make: *** [all] Error 2
error: command 'make' failed with exit status 2
Are you on OS X 10.5?
Is this just not implemented?
cc @cpcloud cc @mrocklin