pyarrow

[bschilder]

Hi Omer,

I seem to be having an issue with pyarrow (on our cluster, but not on my laptop) in extract_snpvar.py. On our cluster, I get the the error:

from pyarrow import ArrowIOError
ImportError: cannot import name 'ArrowIOError' from 'pyarrow' (unknown location)

Is there a specific version of pyarrow that works best? Here's my Conda env:

(echoR) [schilb03@li03c04 Fine_Mapping]$ conda list
# packages in environment at /sc/orga/work/schilb03/conda/envs/echoR:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                      1_llvm    conda-forge
bitarray                  1.2.1            py38h516909a_0    conda-forge
ca-certificates           2019.11.28           hecc5488_0    conda-forge
certifi                   2019.11.28               py38_0    conda-forge
decorator                 4.4.1                      py_0    conda-forge
fastparquet               0.3.3            py38hc1659b7_0    conda-forge
joblib                    0.14.1                     py_0    conda-forge
ld_impl_linux-64          2.33.1               h53a641e_8    conda-forge
libblas                   3.8.0               14_openblas    conda-forge
libcblas                  3.8.0               14_openblas    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc-ng                 9.2.0                h24d8f2e_2    conda-forge
libgfortran-ng            7.3.0                hdf63c60_5    conda-forge
liblapack                 3.8.0               14_openblas    conda-forge
libllvm8                  8.0.1                hc9558a2_0    conda-forge
libopenblas               0.3.7                h5ec1e0e_7    conda-forge
libstdcxx-ng              9.2.0                hdf63c60_2    conda-forge
llvm-openmp               9.0.1                hc9558a2_2    conda-forge
llvmlite                  0.31.0           py38h8b12597_0    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
networkx                  2.4                        py_0    conda-forge
numba                     0.48.0           py38hb3f55d8_0    conda-forge
numpy                     1.18.1           py38h95a1406_0    conda-forge
openssl                   1.1.1d               h516909a_0    conda-forge
pandas                    1.0.1            py38hb3f55d8_0    conda-forge
pip                       20.0.2                     py_2    conda-forge
pyarrow                   0.16.0                   pypi_0    pypi
python                    3.8.1                h357f687_2    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
pytz                      2019.3                     py_0    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
scikit-learn              0.22.1           py38hcdab131_1    conda-forge
scipy                     1.4.1            py38h921218d_0    conda-forge
setuptools                45.2.0                   py38_0    conda-forge
six                       1.14.0                   py38_0    conda-forge
sqlite                    3.30.1               hcee41ef_0    conda-forge
tabix                     0.2.6                ha92aebf_0    bioconda
thrift                    0.11.0          py38he1b5a44_1001    conda-forge
tk                        8.6.10               hed695b0_0    conda-forge
tqdm                      4.42.1                     py_0    conda-forge
wheel                     0.34.2                     py_1    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge

Thanks, Brian

[bschilder]

Ah, actually never mind. I figured out the issues:

1. pyarrow needs to be installed with conda (the pip installation apparently causes problems):

   conda install -c conda-forge pyarrow 

2. On the cluster, I had to load several pieces of software that are required to use pyarrow:

   module load arrow spark

Thanks anyways, Brian

[omerwe]

Sure :) Hopefully this will come in handy for other users!

On Wed, Feb 19, 2020 at 4:38 PM Brian M. Schilder notifications@github.com wrote:

Ah, actually never mind. I figured out the issues:

1. pyarrow needs to be installed with conda (the pip installation apparently causes problems):

conda install -c conda-forge pyarrow

1. On the cluster, I had to load several pieces of software that are required to use pyarrow:

module load arrow spark

Thanks anyways, Brian

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/omerwe/polyfun/issues/24?email_source=notifications&email_token=ACNCB42WTP4TQNNM65QRGD3RDWRE3A5CNFSM4KYA2MS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMJY6VI#issuecomment-588484437, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACNCB42WLDOJYIP3DSI7YQTRDWRE3ANCNFSM4KYA2MSQ .