Windows support

[s3shat]

Hey,

Do you think you can make it work with Python 3.8?

I tried to compile myself with small changes for 3.8, but very hard to do in Windows unfortunately and I am no C++ expert.

[s3shat]

Actually tried with 3.9 and pip install also fails, problem seems to be windows, not the python version

Let me know if I can help, I already tried but it gives about 138 errors when compiling the arrow libraries, probably I am doing something wrong

[0x26res]

@s3shat this hasn't been tested on windows and would probably require some work.

Can you provide more details about the error you get when you try to install? You're probably missing some dependencies that are required to build the C++ code.

[s3shat]

I was also expecting some dependency error but first error is a strange one: setuptools\_distutils\util.py", line 141, in convert_path raise ValueError("path '%s' cannot end with '/'" % pathname) ValueError: path './' cannot end with '/'

I think because of this line "package_dir={"": "./"}," in setup.py

I would do the work to make in compatible with windows but I would need some direction since I dont have much experience with C++ compilation with python etc

[0x26res]

I think the other issues you are seeing are because since pyarrow 10.0.0 it is using C++17 and protarrow is using C++11 (https://github.com/0x26res/cassarrow/blob/master/setup.py#L27).

It's probably something I should update.

[s3shat]

I did manage to get around that path issue by removing that from the setup.py, seems to go further but gives lots of problems with the pyarrow header files, might be an easy fix but it is just beyond me :) :

"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DVERSION_INFO=0.1.2 -I\Lib\site-packages\pybind11\include -IC:\AppData\Local\Programs\Python\Python311\include -IC:\AppData\Local\Programs\Python\Python311\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" /EHsc /Tpcpp\src\cassarrow\bindings.cpp /Fobuild\temp.win-amd64-cpython-311\Release\cpp\src\cassarrow\bindings.obj /EHsc /bigobj /std:c++14 -D_GLIBCXX_USE_CXX11_ABI=0 -Icpp\src -I\Lib\site-packages\pyarrow\include bindings.cpp The contents of <string_view> are available only with C++17 or later. \Lib\site-packages\pyarrow\include\arrow/util/bytes_view.h(26): error C2039: 'basic_string_view': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\valarray(21): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/util/bytes_view.h(26): error C2061: syntax error: identifier 'basic_string_view' \Lib\site-packages\pyarrow\include\arrow/buffer.h(81): error C2039: 'string_view': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\valarray(21): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/buffer.h(81): error C2061: syntax error: identifier 'string_view' \Lib\site-packages\pyarrow\include\arrow/buffer.h(165): error C2039: 'string_view': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\valarray(21): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/buffer.h(165): error C2833: 'operator string_view' is not a recognized operator or type \Lib\site-packages\pyarrow\include\arrow/buffer.h(165): error C2059: syntax error: 'newline' \Lib\site-packages\pyarrow\include\arrow/buffer.h(165): error C2334: unexpected token(s) preceding '{'; skipping apparent function body \Lib\site-packages\pyarrow\include\arrow/buffer.h(171): error C2039: 'bytes_view': is not a member of 'arrow::util' \Lib\site-packages\pyarrow\include\arrow/util/bytes_view.h(24): note: see declaration of 'arrow::util' \Lib\site-packages\pyarrow\include\arrow/buffer.h(171): error C2833: 'operator bytes_view' is not a recognized operator or type \Lib\site-packages\pyarrow\include\arrow/buffer.h(171): error C2059: syntax error: 'newline' \Lib\site-packages\pyarrow\include\arrow/buffer.h(171): error C2334: unexpected token(s) preceding '{'; skipping apparent function body \Lib\site-packages\pyarrow\include\arrow/buffer.h(311): error C2535: 'arrow::Buffer::Buffer(void)': member function already defined or declared \Lib\site-packages\pyarrow\include\arrow/buffer.h(81): note: see declaration of 'arrow::Buffer::Buffer' \Lib\site-packages\pyarrow\include\arrow/buffer.h(82): error C7535: 'arrow::Buffer::Buffer': delegating constructor calls itself The contents of <variant> are available only with C++17 or later. \Lib\site-packages\pyarrow\include\arrow/type.h(1849): error C2039: 'variant': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\atomic(134): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/type.h(1849): error C2143: syntax error: missing ';' before '<' \Lib\site-packages\pyarrow\include\arrow/type.h(1849): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int \Lib\site-packages\pyarrow\include\arrow/type.h(1849): error C2238: unexpected token(s) preceding ';' \Lib\site-packages\pyarrow\include\arrow/type.h(1688): error C2614: 'arrow::FieldRef': illegal member initialization: 'impl_' is not a base or member \Lib\site-packages\pyarrow\include\arrow/type.h(1689): error C2614: 'arrow::FieldRef': illegal member initialization: 'impl_' is not a base or member \Lib\site-packages\pyarrow\include\arrow/type.h(1692): error C2614: 'arrow::FieldRef': illegal member initialization: 'impl_' is not a base or member \Lib\site-packages\pyarrow\include\arrow/type.h(1729): error C2065: 'impl_': undeclared identifier \Lib\site-packages\pyarrow\include\arrow/type.h(1729): error C2039: 'impl_': is not a member of 'arrow::FieldRef' \Lib\site-packages\pyarrow\include\arrow/type.h(1676): note: see declaration of 'arrow::FieldRef' \Lib\site-packages\pyarrow\include\arrow/type.h(1741): error C2039: 'holds_alternative': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\atomic(134): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/type.h(1741): error C2065: 'holds_alternative': undeclared identifier \Lib\site-packages\pyarrow\include\arrow/type.h(1741): error C2275: 'arrow::FieldPath': illegal use of this type as an expression \Lib\site-packages\pyarrow\include\arrow/type.h(1597): note: see declaration of 'arrow::FieldPath' \Lib\site-packages\pyarrow\include\arrow/type.h(1741): error C2065: 'impl_': undeclared identifier \Lib\site-packages\pyarrow\include\arrow/type.h(1742): error C2039: 'holds_alternative': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\atomic(134): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/type.h(1742): error C2065: 'holds_alternative': undeclared identifier \Lib\site-packages\pyarrow\include\arrow/type.h(1742): error C2275: 'std::string': illegal use of this type as an expression \Lib\site-packages\pyarrow\include\arrow/type.h(1742): note: see declaration of 'std::string' \Lib\site-packages\pyarrow\include\arrow/type.h(1742): error C2065: 'impl_': undeclared identifier \Lib\site-packages\pyarrow\include\arrow/type.h(1745): error C2065: 'impl_': undeclared identifier \Lib\site-packages\pyarrow\include\arrow/type.h(1750): error C2065: 'impl_': undeclared identifier \Lib\site-packages\pyarrow\include\arrow/type.h(1753): error C2065: 'impl_': undeclared identifier \Lib\site-packages\pyarrow\include\arrow/type.h(1756): error C2039: 'holds_alternative': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\atomic(134): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/type.h(1756): error C2065: 'holds_alternative': undeclared identifier \Lib\site-packages\pyarrow\include\arrow/type.h(1756): error C2275: 'std::vector<arrow::FieldRef,std::allocator<arrow::FieldRef>>': illegal use of this type as an expression \Lib\site-packages\pyarrow\include\arrow/type.h(1756): note: see declaration of 'std::vector<arrow::FieldRef,std::allocator<arrow::FieldRef>>' \Lib\site-packages\pyarrow\include\arrow/type.h(1756): error C2065: 'impl_': undeclared identifier \Lib\site-packages\pyarrow\include\arrow/type.h(1757): error C2065: 'impl_': undeclared identifier The contents of <optional> are available only with C++17 or later. \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(52): error C2061: syntax error: identifier 'optional' \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(135): note: see reference to class template instantiation 'arrow::stl::ArrayIterator<ArrayType,ValueAccessor>' being compiled \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(54): error C2061: syntax error: identifier 'value_type' \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(55): error C2061: syntax error: identifier 'value_type' \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(65): error C2143: syntax error: missing ';' before '*' \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(65): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(70): error C2143: syntax error: missing ';' before '[' \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(70): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(141): error C2061: syntax error: identifier 'optional' \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(247): note: see reference to class template instantiation 'arrow::stl::ChunkedArrayIterator<ArrayType,ValueAccessor>' being compiled \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(143): error C2061: syntax error: identifier 'value_type' \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(144): error C2061: syntax error: identifier 'value_type' \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(155): error C2143: syntax error: missing ';' before '*' \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(155): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(163): error C2143: syntax error: missing ';' before '[' \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(163): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int \Lib\site-packages\pyarrow\include\arrow/array/array_primitive.h(57): error C2039: 'optional': is not a member of 'std' \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(282): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/array/array_primitive.h(57): error C2143: syntax error: missing ';' before '<' \Lib\site-packages\pyarrow\include\arrow/array/array_primitive.h(57): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int \Lib\site-packages\pyarrow\include\arrow/array/array_primitive.h(57): error C2334: unexpected token(s) preceding '{'; skipping apparent function body \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(52): error C2039: 'optional': is not a member of 'std' \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(282): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/array/array_primitive.h(67): note: see reference to class template instantiation 'arrow::stl::ArrayIterator<arrow::BooleanArray,arrow::stl::detail::DefaultValueAccessor<ArrayType>>' being compiled with [ ArrayType=arrow::BooleanArray ] \Lib\site-packages\pyarrow\include\arrow/array/array_primitive.h(114): error C2143: syntax error: missing ';' before '<' \Lib\site-packages\pyarrow\include\arrow/array/array_primitive.h(124): note: see reference to class template instantiation 'arrow::NumericArray<TYPE>' being compiled \Lib\site-packages\pyarrow\include\arrow/array/array_primitive.h(114): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int \Lib\site-packages\pyarrow\include\arrow/array/array_primitive.h(114): error C2334: unexpected token(s) preceding '{'; skipping apparent function body \Lib\site-packages\pyarrow\include\arrow/array/array_primitive.h(155): error C2039: 'optional': is not a member of 'std' \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(282): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/array/array_primitive.h(155): error C2143: syntax error: missing ';' before '<' \Lib\site-packages\pyarrow\include\arrow/array/array_primitive.h(155): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int \Lib\site-packages\pyarrow\include\arrow/array/array_primitive.h(155): error C2334: unexpected token(s) preceding '{'; skipping apparent function body \Lib\site-packages\pyarrow\include\arrow/array/array_primitive.h(191): error C2039: 'optional': is not a member of 'std' \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(282): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/array/array_primitive.h(191): error C2143: syntax error: missing ';' before '<' \Lib\site-packages\pyarrow\include\arrow/array/array_primitive.h(191): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int \Lib\site-packages\pyarrow\include\arrow/array/array_primitive.h(191): error C2334: unexpected token(s) preceding '{'; skipping apparent function body \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(70): error C3646: 'GetView': unknown override specifier \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(152): note: see reference to class template instantiation 'arrow::BaseBinaryArray<TYPE>' being compiled \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(70): error C2059: syntax error: '(' \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(70): error C2334: unexpected token(s) preceding '{'; skipping apparent function body \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(78): error C2143: syntax error: missing ';' before '<' \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(78): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(78): error C2334: unexpected token(s) preceding '{'; skipping apparent function body \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(87): error C3646: 'Value': unknown override specifier \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(87): error C2059: syntax error: '(' \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(87): error C2334: unexpected token(s) preceding '{'; skipping apparent function body \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(70): error C2039: 'string_view': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\numeric(23): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(155): note: see reference to class template instantiation 'arrow::BaseBinaryArray<arrow::BinaryType>' being compiled \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(78): error C2039: 'optional': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\numeric(23): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(78): error C2039: 'string_view': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\numeric(23): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(87): error C2039: 'string_view': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\numeric(23): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(239): error C2039: 'string_view': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\numeric(23): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(239): error C3646: 'GetView': unknown override specifier \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(239): error C2059: syntax error: '(' \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(239): error C2334: unexpected token(s) preceding '{'; skipping apparent function body \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(243): error C2039: 'optional': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\numeric(23): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(243): error C2143: syntax error: missing ';' before '<' \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(243): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(243): error C2039: 'string_view': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\numeric(23): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(243): error C2334: unexpected token(s) preceding '{'; skipping apparent function body \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(247): error C3861: 'GetView': identifier not found \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(39): error C2039: 'GetView': is not a member of 'arrow::FixedSizeBinaryArray' \Lib\site-packages\pyarrow\include\arrow/array/array_binary.h(224): note: see declaration of 'arrow::FixedSizeBinaryArray' \Lib\site-packages\pyarrow\include\arrow/stl_iterator.h(52): note: see reference to class template instantiation 'arrow::stl::detail::DefaultValueAccessor<ArrayType>' being compiled with [ ArrayType=arrow::FixedSizeBinaryArray ] \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(80): error C2061: syntax error: identifier 'string_view' \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(397): note: see reference to class template instantiation 'arrow::BaseBinaryBuilder<TYPE>' being compiled \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(96): error C2061: syntax error: identifier 'string_view' \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(153): error C2061: syntax error: identifier 'string_view' \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(162): error C2061: syntax error: identifier 'string_view' \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(373): error C3646: 'GetView': unknown override specifier \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(373): error C2059: syntax error: '(' \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(373): error C2334: unexpected token(s) preceding '{'; skipping apparent function body \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(80): error C2039: 'string_view': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\numeric(23): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(401): note: see reference to class template instantiation 'arrow::BaseBinaryBuilder<arrow::BinaryType>' being compiled \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(96): error C2039: 'string_view': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\numeric(23): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(153): error C2039: 'string_view': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\numeric(23): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(162): error C2039: 'string_view': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\numeric(23): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(373): error C2039: 'string_view': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\numeric(23): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(479): error C2039: 'string_view': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\numeric(23): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(479): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(479): error C2143: syntax error: missing ',' before '&' \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(537): error C2039: 'string_view': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\numeric(23): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(537): error C2061: syntax error: identifier 'string_view' \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(593): error C2039: 'string_view': is not a member of 'std' C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\numeric(23): note: see declaration of 'std' \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(593): error C3646: 'GetView': unknown override specifier \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(593): error C2059: syntax error: '(' \Lib\site-packages\pyarrow\include\arrow/array/builder_binary.h(593): fatal error C1003: error count exceeds 100; stopping compilation error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.26.28801\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2 [end of output]

[s3shat]

I tried changing that to 17 and gave a lot of unresolved externals, probably not that easy to do :) :

cassarrow.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __cdecl arrow::Status::Status(class arrow::Status &&)" (__imp_??0Status@arrow@@QEAA@$$QEAV01@@Z) cassarrow.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __cdecl arrow::Status::Status(class arrow::Status const &)" (__imp_??0Status@arrow@@QEAA@AEBV01@@Z) cassarrow.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: static class arrow::Status __cdecl arrow::Status::OK(void)" (__imp_?OK@Status@arrow@@SA?AV12@XZ) cassarrow.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: class std::basic_ostream<char,struct std::char_traits<char> > & __cdecl arrow::util::detail::StringStreamWrapper::stream(void)" (__imp_?stream@StringStreamWrapper@detail@util@arrow@@QEAAAEAV?$basic_ostream@DU?$char_traits@D@std@@@std@@XZ) build\lib.win-amd64-cpython-311\_cassarrow.cp311-win_amd64.pyd : fatal error LNK1120: 132 unresolved externals

[0x26res]

It looks like either:

• It's not linking against the arrow cpp library
• the symbols are not compatible (so arrow has been linked with a different compiler or version)

For the first one, it's worth checking that these arguments https://github.com/0x26res/cassarrow/blob/master/setup.py#L33 (library and library_dir) are passing correct values.

[0x26res]

I've just pushed some code in master. I've made setup.py less platform specific. Please give it a try.

[s3shat]

that problem seems like fixed, but now it gives this error: cassarrow.cpp cpp\src\cassarrow\cassarrow.cpp(96): warning C4267: 'initializing': conversion from 'size_t' to 'int', possible loss of data warning: I don't know what to do with 'runtime_library_dirs': ['\\Lib\\site-packages\\pyarrow'] error: don't know how to set runtime library search path for MSVC

[0x26res]

cassarrow.cpp cpp\src\cassarrow\cassarrow.cpp(96): warning C4267: 'initializing': conversion from 'size_t' to 'int', possible loss of data

That's safe to ignore

error: don't know how to set runtime library search path for MSVC

You need to comment out the line with runtime_library_dirs, it looks like windows doesn't support it.

[s3shat]

Yes, I did commented that line and it did work, I mean it did compile: Building wheels for collected packages: cassarrow Building wheel for cassarrow (pyproject.toml) ... done Created wheel for cassarrow: filename=cassarrow-0.1.2-cp38-cp38-win_amd64.whl size=87614 sha256=51931f243d118a3a2c8635abdfdb1a659aa29bfc328cdd3e4706067e7cb1b8e2 Stored in directory: ...\wheels\ed\51\51\d41e6b8c5484ef4d31d56fec42440ab4898f29b123ac358d6a Successfully built cassarrow Installing collected packages: geomet, cassarrow Attempting uninstall: geomet Found existing installation: geomet 0.3.0 Uninstalling geomet-0.3.0: Successfully uninstalled geomet-0.3.0 Successfully installed cassarrow-0.1.2 geomet-0.2.1.post1 But now when I import the library it gives this error: `

import cassarrow Traceback (most recent call last): File "", line 1, in File "...\lib\site-packages\cassarrow__init__.py", line 1, in from cassarrow.impl import install_cassarrow, metadata_to_schema, result_set_to_table File "...\lib\site-packages\cassarrow\impl.py", line 5, in import _cassarrow ImportError: DLL load failed while importing _cassarrow: The specified module could not be found. `

[0x26res]

Which command are you using to build?

Typically I use this to test that it works:

python setup.py build_ext --inplace &&  python -c "import cassarrow"

[s3shat]

git cloned to cassarow directory and

pip install ./cassarrow

[s3shat]

I do have the _cassarrow.cp38-win_amd64.pyd in site-packages directory, not sure why it is not finding it or I don't what DLL is he talking about

[0x26res]

pip install ./cassarrow works for me on unix.

So in your pip virtual environment you should see a cassarrow dll. For example I have this:

test-venv/lib/python3.9/site-packages/_cassarrow.cpython-39-darwin.so

Do you see any cassarrow dll in your site-package?

[s3shat]

searched the whole pc but no cassarrow.dll

[0x26res]

I don't expect it to be called cassarow.dll (but something like cassarrow.dll)

[s3shat]

I searched the whole pc for anything that contains cassarrow, these came out:

cassarrow-0.1.2.dist-info cassarrow-0.1.2 (1).dist-info cassarrow.egg-info cassarrow (1).egg-info _cassarrow.cp311-win_amd64.pyd _cassarrow.cp38-win_amd64.pyd _cassarrow.cp38-win_amd64 (1).pyd _cassarrow.cp311-win_amd64 (1).pyd _cassarrow.cp38-win_amd64 (2).pyd _cassarrow.cp311-win_amd64.exp _cassarrow.cp311-win_amd64.lib cassarrow.obj _cassarrow.cp38-win_amd64.exp _cassarrow.cp38-win_amd64.lib cassarrow (1).obj cassarrow.cpp cassarrow.h test_cassarrow.cpp _cassarrow.cp38-win_amd64 (1).exp _cassarrow.cp38-win_amd64 (1).lib cassarrow (2).obj cassarrow (1).cpp cassarrow (1).h test_cassarrow (1).cpp cassarrow-master.zip

[0x26res]

Not sure what's going on. I'm not an expert on windows, and I don't have a development environment set up on windows to test anything.

There are some resources available if you want to dig a bit deeper: https://stackoverflow.com/questions/61369141/importerror-dll-load-failed-with-pybind11-and-pcl/61371168#61371168

Also you may want to double check pip install is doing the right thing (pip install --verbose ./cassarrow)

[s3shat]

--verbose does not give any error as well, I did research a bit, thanks for the link, I do have a pyd file generated by the build process and python doc says: https://docs.python.org/3/faq/windows.html#is-a-pyd-file-the-same-as-a-dll , pyd is basically a DLL, and some guy from that SO article says the message might be misleading, so probably cassarow's DLL is not missing, some other dependency is missing, need to find that somehow,

[s3shat]

Made it work, thanks to that SO, found Dependencies and found out _cassarrow cannot find pyarrow.dll, pyarrow_python.dll and parquet.dll, so a workaround would be copy them to site-packages where the pyd is or add pyarrow folder into path, and MAGIC, it works, I mean import worked but I had to modify cassarrow.impl a bit since it uses safe typing stuff from Python 3.9 which does not work with 3.8, just removed those since they are optional and imports, I will test it with a cassandra installation and let you know.

[0x26res]

Good to hear.

For the type hints, I'll try to make them compatible with 3.8 (or feel free to submit an MR).

I guess the dll issue you were having comes from the fact that runtime_library_dirs doesn't work on windows. I wonder if there's a work around.

[s3shat]

Yes, there should be something for that, which would fix that DLL problem, I was testing with an actual cassandra and getting this error now, do you think it might be related:

File "cass.py", line 298, in format_resultset return pl.from_arrow(cassarow.result_set_to_table(result)) File "site-packages\cassarrow\impl.py", line 85, in result_set_to_table return pa.Table.from_batches(result_set, schema=schema) File "pyarrow\table.pxi", line 3761, in pyarrow.lib.Table.from_batches TypeError: Cannot convert list to pyarrow.lib.RecordBatch

I did print the result_set just before the line 85 and it is a cassandra.cluster.ResultSet

I think I am missing something, I will investigate more but any input is welcome

[s3shat]

ok, made that work as well, my bad there was another python 3.9 thing thing that I converted wrong, this: message_types_by_opcode = _ProtocolHandler.message_types_by_opcode | {ArrowResultMessage.opcode: ArrowResultMessage }

should be like this for 3.8: message_types_by_opcode = {**_ProtocolHandler.message_types_by_opcode, **{ArrowResultMessage.opcode: ArrowResultMessage} }

now it works fine and quite fast as well.

[0x26res]

Hey, so looking at this issue:

https://stackoverflow.com/questions/66591019/python-extension-modules-on-windows-controlling-the-dll-path-for-dependencies

It might be the case that for windows, we can add something like:

import pa
import os

for dir in pa.get_library_dirs():
  os.add_dll_directory(dir)

At the top of the cassarrow/init.py file. And it may solve the issue of the missing dll.

Could you give it a try?

[s3shat]

I think you changed the build? I wanted to try a clean install and now getting this:

[s3shat]

Also, pyarrow takes care of that DLLs locations when you import it, I was questioning why you have this: import pyarrow as pa # Must be imported before _cassarrow But it is after the import of _cassarrow

If you move the import pyarrow before _cassarrow, works like a charm :) no OS fiddling required :)

[0x26res]

I think you changed the build? I wanted to try a clean install and now getting this:

Yes I moved to poetry so it's easier to manager build. But it looks like it can't install locally. Not sure yet how to fix that. But I would suggest you use the latest version from pypi which should work for you:

pip install cassarrow==0.2.0rc2

If you move the import pyarrow before _cassarrow, works like a charm :) no OS fiddling required :)

OK, the import order is fixed and is now enforced correctly by the linter (isort).

[s3shat]

pip install cassarrow==0.2.0rc2 is giving me this now (someone said to upgrade the build package which I did but still the same problem, I did successfully installed build-0.9.0 pep517-0.13.0 )

` >pip install cassarrow==0.2.0rc2 Collecting cassarrow==0.2.0rc2 Using cached cassarrow-0.2.0rc2.tar.gz (8.4 kB) Installing build dependencies ... done Getting requirements to build wheel ... error error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> [15 lines of output] Traceback (most recent call last): File "\Local\Programs\Python\Python38\lib\site-packages\pip_vendor\pep517\in_process_in_process.py", line 351, in main() File "\Local\Programs\Python\Python38\lib\site-packages\pip_vendor\pep517\in_process_in_process.py", line 333, in main json_out['return_val'] = hook(**hook_input['kwargs']) File "\Local\Programs\Python\Python38\lib\site-packages\pip_vendor\pep517\in_process_in_process.py", line 118, in get_requires_for_build_wheel return hook(config_settings) File "\Local\Temp\pip-build-env-8sib0tak\overlay\Lib\site-packages\setuptools\build_meta.py", line 338, in get_requires_for_build_wheel return self._get_build_requires(config_settings, requirements=['wheel']) File "\Local\Temp\pip-build-env-8sib0tak\overlay\Lib\site-packages\setuptools\build_meta.py", line 320, in _get_build_requires self.run_setup() File "\Local\Temp\pip-build-env-8sib0tak\overlay\Lib\site-packages\setuptools\build_meta.py", line 335, in run_setup exec(code, locals()) File "", line 31, in ModuleNotFoundError: No module named 'build' [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> See above for output. `

[0x26res]

OK, so I think I've fixed the local install (pip install ./cassarrow) and I should have a fix for the pypi install soon.

[s3shat]

now I get this (local install): overlay\Lib\site-packages\dunamai\__init__.py", line 308, in _detect_vcs raise RuntimeError("Unable to detect version control system.") RuntimeError: Unable to detect version control system.

[0x26res]

I suspect dunami can't detect git in your windows environment. You've clone the repo right?

Otherwise can you try pip install 0.2.0rc4 I think it's working now.

[s3shat]

yes, moved along a bit but getting this now: c1xx: fatal error C1081: 'C:\Users\username\AppData\Local\Temp\pip-install-52nmfnq9\cassarrow_aca48ffe1d1d48a1a9ce89db18097915\build\temp.win-amd64-cpython-38\Release\Users\username\AppData\Local\Temp\pip-install-52nmfnq9\cassarrow_aca48ffe1d1d48a1a9ce89db18097915\cpp\src\cassarrow\bindings.obj': file name too long not sure what is deciding that filename with a hash of something

[0x26res]

Maybe try this https://github.com/pypa/pip/issues/3055#issuecomment-323794950