search

mcaceresb / stata-parquet

Read and write parquet files from Stata
MIT License
22 stars 6 forks source link

Installing #1

Closed kylebarron closed 6 years ago

kylebarron commented 6 years ago

I'm trying out Conda. To install the libraries I did:

conda create -n stata-parquet -c conda-forge arrow-cpp parquet-cpp -y

Then the lib and include paths are:

/home/kyle/local/anaconda3/envs/stata-parquet/lib
/home/kyle/local/anaconda3/envs/stata-parquet/include

But I'm still getting a linking error... It's been a while since I've tried to compile C/C++ code. Is it clear what I'm doing wrong?

> make INCLUDE=/home/kyle/local/anaconda3/envs/stata-parquet/include/arrow LIBS=/home/kyle/local/anaconda3/envs/stata-parquet/lib

rm -f build/parquet_unix.plugin
rm -f  src/plugin/lib
rm -f  src/plugin/spi
ln -sf ../../lib          src/plugin/lib
ln -sf lib/spi-2.0 src/plugin/spi
mkdir -p ./build
g++ -Wall -O3 -shared -fPIC -DSYSTEM=OPUNIX -o build/parquet_unix.plugin src/plugin/spi/stplugin.cpp src/plugin/parquet.cpp /home/kyle/local/anaconda3/envs/stata-parquet/include/arrow /home/kyle/local/anaconda3/envs/stata-parquet/lib -larrow -lparquet
In file included from src/plugin/parquet.cpp:41:0:
src/plugin/reader_writer.h:20:10: fatal error: arrow/api.h: No such file or directory
 #include <arrow/api.h>
          ^~~~~~~~~~~~~
compilation terminated.
Makefile:40: recipe for target 'parquet' failed
make: *** [parquet] Error
mcaceresb commented 6 years ago 
make INCLUDE=-I/home/kyle/local/anaconda3/envs/stata-parquet/include/arrow LIBS=-L/home/kyle/local/anaconda3/envs/stata-parquet/lib
kylebarron commented 6 years ago

You're right, I did miss the -I and -L, but it still didn't work...

> make INCLUDE=-I/home/kyle/local/anaconda3/envs/stata-parquet/include/arrow LIBS=-L/home/kyle/local/anaconda3/envs/stata-parquet/lib

rm -f build/parquet_unix.plugin
rm -f  src/plugin/lib
rm -f  src/plugin/spi
ln -sf ../../lib          src/plugin/lib
ln -sf lib/spi-2.0 src/plugin/spi
mkdir -p ./build
g++ -Wall -O3 -shared -fPIC -DSYSTEM=OPUNIX -o build/parquet_unix.plugin src/plugin/spi/stplugin.cpp src/plugin/parquet.cpp -I/home/kyle/local/anaconda3/envs/stata-parquet/include/arrow -L/home/kyle/local/anaconda3/envs/stata-parquet/lib -larrow -lparquet
In file included from src/plugin/parquet.cpp:41:0:
src/plugin/reader_writer.h:20:10: fatal error: arrow/api.h: No such file or directory
 #include <arrow/api.h>
          ^~~~~~~~~~~~~
compilation terminated.
Makefile:40: recipe for target 'parquet' failed
make: *** [parquet] Error
mcaceresb commented 6 years ago

Let me try

mcaceresb commented 6 years ago 
make INCLUDE=-I/home/kyle/local/anaconda3/envs/stata-parquet/include LIBS=-L/home/kyle/local/anaconda3/envs/stata-parquet/lib

gets it to compile. It didn't quite work on the server (undefined symbol error), but let me know if the above does the trick in terms of making the plugin.

kylebarron commented 6 years ago

Ah sorry, I tried that first but missed the -I and -L

kylebarron commented 6 years ago

I get a different error (on my personal linux computer):

> make INCLUDE=-I/home/kyle/local/anaconda3/envs/stata-parquet/include LIBS=-L/home/kyle/local/anaconda3/envs/stata-parquet/lib

rm -f build/parquet_unix.plugin
rm -f  src/plugin/lib
rm -f  src/plugin/spi
ln -sf ../../lib          src/plugin/lib
ln -sf lib/spi-2.0 src/plugin/spi
mkdir -p ./build
g++ -Wall -O3 -shared -fPIC -DSYSTEM=OPUNIX -o build/parquet_unix.plugin src/plugin/spi/stplugin.cpp src/plugin/parquet.cpp -I/home/kyle/local/anaconda3/envs/stata-parquet/include -L/home/kyle/local/anaconda3/envs/stata-parquet/lib -larrow -lparquet
In file included from src/plugin/parquet.cpp:42:0:
src/plugin/parquet.h: In function ‘void sf_printf_debug(int, const char*, ...)’:
src/plugin/parquet.h:14:31: warning: format not a string literal and no format arguments [-Wformat-security]
     if (debug > 1) printf (buf);
                               ^
cp build/*plugin lib/plugin/
mkdir -p               ./build/parquet/
cp changelog.md        ./build/parquet/
cp src/parquet.pkg     ./build/parquet/
cp src/stata.toc       ./build/parquet/
cp src/ado/parquet.ado ./build/parquet/
cp docs/parquet.sthlp  ./build/parquet/
cp build/*plugin       ./build/parquet/
cp ./src/test/parquet_tests.do ./build/
cd build && ls parquet/*.{ado,sthlp,plugin} | zip -@ parquet-ssc.zip
ls: cannot access 'parquet/*.{ado,sthlp,plugin}': No such file or directory

zip error: Nothing to do! (parquet-ssc.zip)
Makefile:55: recipe for target 'zip' failed
make: *** [zip] Error 12
mcaceresb commented 6 years ago

I think you can ignore that for now. Did it make the plugin?

kylebarron commented 6 years ago 
kyle at desktop in ~/local/anaconda3/envs/stata-parquet/lib                                20:21
> export LD_LIBRARY_PATH=`pwd -P`:$LD_LIBRARY_PATH
kyle at desktop in ~/local/anaconda3/envs/stata-parquet/lib                                20:21
> xstata
/home/kyle/local/stata/xstata: symbol lookup error: /usr/lib/x86_64-linux-gnu/libatk-1.0.so.0: undefined symbol: g_log_structured_standard

Is that what you got on the servers?

kylebarron commented 6 years ago

But I think it installed correctly?

kyle at desktop in ~/github/stata/stata-parquet on master ✗                      [0aee3f3]  20:22
> tree build
build
├── changelog.md
├── parquet
│   ├── changelog.md
│   ├── parquet.ado
│   ├── parquet.pkg
│   ├── parquet.sthlp
│   ├── parquet_unix.plugin
│   └── stata.toc
├── parquet.ado
├── parquet.pkg
├── parquet.sthlp
├── parquet_tests.do
├── parquet_unix.plugin
└── stata.toc

1 directory, 13 files
kyle at desktop in ~/github/stata/stata-parquet on master ✗                      [0aee3f3]  20:22
> cat stata.log
───────┬──────────────────────────────────────────────────────────────────────────────────────────
       │ File: stata.log
───────┼──────────────────────────────────────────────────────────────────────────────────────────
   1   │
   2   │   ___  ____  ____  ____  ____ (R)
   3   │  /__    /   ____/   /   ____/
   4   │ ___/   /   /___/   /   /___/   15.0   Copyright 1985-2017 StataCorp LLC
   5   │   Statistics/Data Analysis            StataCorp
   6   │                                       4905 Lakeway Drive
   7   │                                       College Station, Texas 77845 USA
   8   │                                       800-STATA-PC        http://www.stata.com
   9   │                                       979-696-4600        stata@stata.com
  10   │                                       979-696-4601 (fax)
  11   │
  12   │ 60-user Stata network license expires 14 Aug 2019:
  13   │        Serial number:  401509200886
  14   │          Licensed to:  National Bureau of Economic Research
  15   │                        Cambridge, MA
  16   │
  17   │ Notes:
  18   │       1.  Stata is running in batch mode.
  19   │       2.  Unicode is supported; see help unicode_advice.
  20   │
  21   │ . net install parquet, from(/home/kyle/github/stata/stata-parquet/build) replac
  22   │ > e
  23   │ checking parquet consistency and verifying not already installed...
  24   │ installing into /home/kyle/ado/plus/...
  25   │ installation complete.
───────┴──────────────────────────────────────────────────────────────────────────────────────────
mcaceresb commented 6 years ago

I got

stata13: symbol lookup error: ./parquet_unix.plugin: undefined symbol: _ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE
mcaceresb commented 6 years ago

So you get a symbol lookup error when you try and start Stata in the first place?

kylebarron commented 6 years ago

Yeah. I'm assuming it's just a library conflict. I'm guessing that the libatk in my ~/local/anaconda3/envs/stata-parquet/lib is too new...

mcaceresb commented 6 years ago

Maybe batch mode?

kylebarron commented 6 years ago

Oh non-gui stata works...

kylebarron commented 6 years ago

But then I get the same undefined symbol error...

. sysuse auto
(1978 Automobile Data)

. parquet save auto.parquet, replace
stata: symbol lookup error: /home/kyle/ado/plus/p/parquet_unix.plugin: undefined symbol: _ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE
kylebarron commented 6 years ago

What version of Arrow and Parquet did you develop with?

> conda list
# packages in environment at /home/kyle/local/anaconda3/envs/stata-parquet:
#
# Name                    Version                   Build  Channel
arrow-cpp                 0.11.1           py36h3bd774a_0    conda-forge
blas                      1.0                         mkl
boost-cpp                 1.68.0               h3a22d5f_0    conda-forge
bzip2                     1.0.6                h470a237_2    conda-forge
ca-certificates           2018.10.15           ha4d7672_0    conda-forge
certifi                   2018.10.15            py36_1000    conda-forge
icu                       58.2                 hfc679d8_0    conda-forge
intel-openmp              2019.0                      118
libffi                    3.2.1                hfc679d8_5    conda-forge
libgcc-ng                 7.2.0                hdf63c60_3    conda-forge
libgfortran-ng            7.2.0                hdf63c60_3    conda-forge
libstdcxx-ng              7.2.0                hdf63c60_3    conda-forge
mkl                       2019.0                      118
mkl_fft                   1.0.6                    py36_0    conda-forge
mkl_random                1.0.1                    py36_0    conda-forge
ncurses                   6.1                  hfc679d8_1    conda-forge
numpy                     1.15.0           py36h1b885b7_0
numpy-base                1.15.0           py36h3dfced4_0
openssl                   1.0.2p               h470a237_1    conda-forge
parquet-cpp               1.5.1                         1    conda-forge
pip                       18.1                  py36_1000    conda-forge
python                    3.6.6                h5001a0f_3    conda-forge
readline                  7.0                  haf1bffa_1    conda-forge
setuptools                40.5.0                   py36_0    conda-forge
sqlite                    3.25.2               hb1c47c0_0    conda-forge
tk                        8.6.8                ha92aebf_0    conda-forge
wheel                     0.32.2                   py36_0    conda-forge
xz                        5.2.4                h470a237_1    conda-forge
zlib                      1.2.11               h470a237_3    conda-forge
mcaceresb commented 6 years ago

I'm using 0.12.0

kylebarron commented 6 years ago

That hasn't been released yet... https://arrow.apache.org/

No idea if that's the problem

mcaceresb commented 6 years ago

Aow, sadness. I pulled from github. Let me see if I can swing this with conda on my local machine.

kylebarron commented 6 years ago

The nm command is super cool:

> nm libarrow.so.11.1.0 | grep _ZN5arrow5field
00000000001342b0 T _ZN5arrow5fieldERKSsRKSt10shared_ptrINS_8DataTypeEEbRKS2_IKNS_16KeyValueMetadataEE

It searches shared objects.

mcaceresb commented 6 years ago

Even a very simple command where all I do is open and close a file gives me an error along these lines...

symbol lookup error: ./parquet_unix.plugin: undefined symbol: _ZN5arrow2io16FileOutputStream4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPSt10shared_ptrIS1_E

Mmm....

kylebarron commented 6 years ago

If you search undefined symbol, you get lots of hits on the Arrow issues: https://github.com/apache/arrow/issues?utf8=%E2%9C%93&q=undefined+symbol

I don't really want this issue to show up in all of them by linking explicitly.

Apparently there's a develop and a release build, so things might be a little different in what you git clone and compile

mcaceresb commented 6 years ago

I def did the release build... But we're both trying to use the stuff from conda now.... Not sure why we get undefined lookup error...

kylebarron commented 6 years ago

Might be best to ask on Arrow issues or on Stack Overflow

mcaceresb commented 6 years ago

Not the most replicable issue w/o Stata.

kylebarron commented 6 years ago

https://issues.apache.org/jira/browse/ARROW-2783

kylebarron commented 6 years ago

Except that that issue revolves around

undefined symbol: _ZN5boost...
mcaceresb commented 6 years ago

If I run with LD_DEBUG=allit dies after

      6701:     binding file ./parquet_unix.plugin [0] to /usr/lib/libc.so.6 [0]: normal symbol `memcpy' [GLIBC_2.14]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/libarrow.so.11.1.0 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/libstdc++.so.6 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/usr/lib/libncurses.so.5 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/usr/lib/libm.so.6 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/usr/lib/libdl.so.2 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/libgcc_s.so.1 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/usr/lib/libc.so.6 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/./libz.so.1 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/./libboost_system.so.1.68.0 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/./libboost_filesystem.so.1.68.0 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/./libboost_regex.so.1.68.0 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/usr/lib/libpthread.so.0 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/usr/lib/librt.so.1 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/././libicudata.so.58 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/././libicui18n.so.58 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/././libicuuc.so.58 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=./parquet_unix.plugin [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/libarrow.so.11.1.0 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/libparquet.so.11 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/libstdc++.so.6 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/usr/lib/libm.so.6 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/libgcc_s.so.1 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/usr/lib/libc.so.6 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/usr/lib/libdl.so.2 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/./libz.so.1 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/./libboost_system.so.1.68.0 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/./libboost_filesystem.so.1.68.0 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/./libboost_regex.so.1.68.0 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/usr/lib/libpthread.so.0 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/usr/lib/librt.so.1 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/././libicudata.so.58 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/././libicui18n.so.58 [0]
      6701:     symbol=_ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE;  lookup in file=/home/mauricio/bulk/programs/miniconda3/envs/stata-parquet/lib/././libicuuc.so.58 [0]
      6701:     ./parquet_unix.plugin: error: symbol lookup error: undefined symbol: _ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE (fatal)
/home/mauricio/.local/stata13/stata: symbol lookup error: ./parquet_unix.plugin: undefined symbol: _ZN5arrow5fieldERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_8DataTypeEEbRKS8_IKNS_16KeyValueMetadataEE
kylebarron commented 6 years ago

It might not be too hard to find a reproducible non-stata example that you could post on the Arrow issues or on the conda-forge Arrow feedstock issues.

mcaceresb commented 6 years ago

It worked! I'm 99% sure that it worked just now if I use gcc from conda as well.

kylebarron commented 6 years ago

I did that and got a different error...

Maybe I need to add something else as well to my conda create statement?

. parquet save test.parquet
stata: symbol lookup error: /home/kyle/ado/plus/p/parquet_unix.plugin: undefined symbol: _ZSt7getlineIcSt11char_traitsIcESaIcEERSt13basic_istreamIT_T0_ES7_RNSt7__cxx1112basic_stringIS4_S5_T1_EES4_
> conda list                                                               
# packages in environment at /home/kyle/local/anaconda3/envs/stata-parquet:
#
# Name                    Version                   Build  Channel
arrow-cpp                 0.11.1           py36h3bd774a_0    conda-forge
blas                      1.0                         mkl  
boost-cpp                 1.68.0               h3a22d5f_0    conda-forge
bzip2                     1.0.6                h470a237_2    conda-forge
ca-certificates           2018.10.15           ha4d7672_0    conda-forge
certifi                   2018.10.15            py36_1000    conda-forge
cloog                     0.18.0                        0  
gcc                       4.8.5                         7  
gmp                       6.1.2                hfc679d8_0    conda-forge
icu                       58.2                 hfc679d8_0    conda-forge
intel-openmp              2019.0                      118  
isl                       0.12.2                        0  
libffi                    3.2.1                hfc679d8_5    conda-forge
libgcc-ng                 7.2.0                hdf63c60_3    conda-forge
libgfortran-ng            7.2.0                hdf63c60_3    conda-forge
libstdcxx-ng              7.2.0                hdf63c60_3    conda-forge
mkl                       2019.0                      118  
mkl_fft                   1.0.6                    py36_0    conda-forge
mkl_random                1.0.1                    py36_0    conda-forge
mpc                       1.1.0                         4    conda-forge
mpfr                      3.1.5                         0    conda-forge
ncurses                   6.1                  hfc679d8_1    conda-forge
numpy                     1.15.0           py36h1b885b7_0  
numpy-base                1.15.0           py36h3dfced4_0  
openssl                   1.0.2p               h470a237_1    conda-forge
parquet-cpp               1.5.1                         1    conda-forge
pip                       18.1                  py36_1000    conda-forge
python                    3.6.6                h5001a0f_3    conda-forge
readline                  7.0                  haf1bffa_1    conda-forge
setuptools                40.5.0                   py36_0    conda-forge
sqlite                    3.25.2               hb1c47c0_0    conda-forge
tk                        8.6.8                ha92aebf_0    conda-forge
wheel                     0.32.2                   py36_0    conda-forge
xz                        5.2.4                h470a237_1    conda-forge
zlib                      1.2.11               h470a237_3    conda-forge
mcaceresb commented 6 years ago

One sec; I have it on the server as well. Let me update the readme and I'll paste the cmds here as well.

mcaceresb commented 6 years ago 
conda install -n stata-parquet -c conda-forge icu arrow-cpp parquet-cpp boost gcc -y
ENV=/home/kyle/local/anaconda3/envs/stata-parquet
make GCC=${ENV}/bin/g++ UFLAGS=-std=c++11 INCLUDE=-I${ENV}/include LIBS=-L${ENV}/lib all copy
mcaceresb commented 6 years ago

git pull first.

kylebarron commented 6 years ago

OMG... it works!!!

kylebarron commented 6 years ago

You may want to update the "Using" section to say the LD_LIBRARY_PATH should be the Conda location.

If this package ends up becoming more stable, I may explore a conda installer for stata-parquet itself, so that you could do something like conda install stata-parquet and have the dependencies quietly installed in the background.

mcaceresb commented 6 years ago

Good idea. I'll open a new issue. Close this one?

kylebarron commented 6 years ago

Sounds good to me

kylebarron commented 6 years ago

Good idea to which, the full Conda installer? Should I post that issue?

mcaceresb commented 6 years ago

Just posted a new issue abt the full conda installer. Never really used conda much; this is rather ingratiating it to me.

kylebarron commented 6 years ago

Yeah there have been many times for me, especially on the servers, where I was able to just conda install some library and have it work, despite the age of GCC and other things on the server. I'm quite happy with it. It does a lot more than pip does.