Fix various issues with pyarrow package

ARF1 commented 3 years ago

The currently distributed pyarrow package has various problems. This can be seen from the fact that the documentation specifically recommends installing from conda-forge.

Furthermore, the solution to various bug reports on their issue tracker is "reinstall pyarrow from conda-forge".

For me the most annoying issue is, that the anaconda version is built without snappy compression support for parquet. - This is a questionable choice, since snappy is the default compression option for parquet files saved with pyarrow.

Anaconda or Miniconda version:

pyarrow-3.0.0-py38h953b917_1

Operating System:

`conda info`

``` active environment : base active env location : C:\ProgramData\Miniconda3 shell level : 1 user config file : C:\Users\My User\.condarc populated config files : conda version : 4.9.2 conda-build version : not installed python version : 3.7.6.final.0 virtual packages : __win=0=0 __archspec=1=x86_64 base environment : C:\ProgramData\Miniconda3 (writable) channel URLs : https://repo.anaconda.com/pkgs/main/win-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/win-64 https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/msys2/win-64 https://repo.anaconda.com/pkgs/msys2/noarch package cache : C:\ProgramData\Miniconda3\pkgs C:\Users\My User\.conda\pkgs C:\Users\My User\AppData\Local\conda\conda\pkgs envs directories : C:\ProgramData\Miniconda3\envs C:\Users\My User\.conda\envs C:\Users\My User\AppData\Local\conda\conda\envs platform : win-64 user-agent : conda/4.9.2 requests/2.24.0 CPython/3.7.6 Windows/10 Windows/10.0.18362 administrator : False netrc file : None offline mode : False ```

`conda list --show-channel-urls`

Sorry, I had already deleted the environment with the default `pyarrow` package when I opened this report.

katietz commented 3 years ago

not sure why you think snappy isn't supported? We support it for osx/linux. For obvious reasons we don't for windows platforms. In general we follow in our recipes pretty much what was done on conda-forge. Could you give some more pointers to the open issues you see? Spreading rumors is not necessary proofing an evidence.

ARF1 commented 3 years ago

@katietz

not sure why you think snappy isn't supported?

Because python complains with OSError: NotImplemented: Support for codec 'snappy' not built. That seems pretty clear-cut to me.

We support it for osx/linux. For obvious reasons we don't for windows platforms.

The reasons are not obvious to me.

In general we follow in our recipes pretty much what was done on conda-forge. Could you give some more pointers to the open issues you see? Spreading rumors is not necessary proofing an evidence.

It is unclear to me what you interpret as rumors. My apologies if I inadvertently offended. I assure you it was not my intention.

Regarding reproducing, execute the following commands on a Windows machine:

conda create -n conda-pyarrow python=3.8 pyarrow
activate conda-pyarrow
python

Then read any snappy compressed parquet file:

Python 3.8.8 (default, Feb 24 2021, 15:54:32) [MSC v.1928 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow.parquet
>>> table = pyarrow.parquet.read_table('test.parquet')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ProgramData\Miniconda3\envs\conda-pyarrow\lib\site-packages\pyarrow\parquet.py", line 1703, in read_table
    return dataset.read(columns=columns, use_threads=use_threads,
  File "C:\ProgramData\Miniconda3\envs\conda-pyarrow\lib\site-packages\pyarrow\parquet.py", line 1580, in read
    table = self._dataset.to_table(
  File "pyarrow\_dataset.pyx", line 372, in pyarrow._dataset.Dataset.to_table
  File "pyarrow\_dataset.pyx", line 2266, in pyarrow._dataset.Scanner.to_table
  File "pyarrow\error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow\error.pxi", line 99, in pyarrow.lib.check_status
OSError: NotImplemented: Support for codec 'snappy' not built

For what it's worth, it is no problem to me if you want to keep your pyarrow package as you currently build it. I will just switch to the conda-forge channel where everything works.

I just thought I would report that it is "not working properly" since you might not be aware of it. My understanding was that the philosophy of the anaconda distribution was that "it just works". Since you are aware of the issue and made a conscious choice to deviate from this philosophy for pyarrow please feel free to close this issue.

katietz commented 3 years ago

I updated arrow-cpp & pyarrow for Windows 64-bit. Your test works for me now on Windows too.

katietz commented 3 years ago

I want to excuse myself here about the word "rumor" ... it seems to having in English a different meaning as I expected. So, of course you are not lying ... if I got wrong here, I am sorry too.

jorisvandenbossche commented 3 years ago

@katietz this does not yet seem to be working (at least based on the Windows CI in pandas (https://github.com/pandas-dev/pandas/issues/42370), didn't check myself locally since I don't have Windows). See also my comment at https://github.com/AnacondaRecipes/arrow-cpp-feedstock/commit/abd5c84f14d1972deac0691b6cca6c6f3df4f764

silviumatei commented 2 years ago

This is a bit old but I faced the same error reported by ARF1, after installing pyarrow: OSError: NotImplemented: Support for codec 'snappy' not built Miniconda version 4.11.0 on Windows 11, python 3.9 and env 3.7. The below shows the process for the main python installation.

I uninstalled and then reinstalled from conda-forge. It did not solve the issue although it seems to have downloaded snappy. The issue got solved after installing snappy with pip. Maybe a conda issue since it installs snappy 1.1.8 while pip installs v 3.0.3

Below all the process:

Initial install of pyarrow: conda install pyarrow

## Package Plan ##

  environment location: C:\Miniconda3

  added / updated specs:
    - pyarrow

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    arrow-cpp-4.0.1            |   py39h0d1d0e5_3         4.4 MB
    aws-c-common-0.4.57        |       ha925a31_1         147 KB
    aws-c-event-stream-0.1.6   |       hd77b12b_5          26 KB
    aws-checksums-0.1.9        |       ha925a31_0          50 KB
    aws-sdk-cpp-1.8.185        |       hd77b12b_0         2.5 MB
    boost-cpp-1.73.0           |      h2bbff1b_11          25 KB
    bottleneck-1.3.2           |   py39h7cc1a96_1         107 KB
    brotli-1.0.9               |       ha925a31_2         332 KB
    bzip2-1.0.8                |       he774522_0         113 KB
    c-ares-1.18.1              |       h2bbff1b_0         110 KB
    double-conversion-3.1.5    |       ha925a31_1         254 KB
    gflags-2.2.2               |       ha925a31_0         233 KB
    glog-0.5.0                 |       hd77b12b_0          85 KB
    grpc-cpp-1.26.0            |       h351948d_0         4.8 MB
    intel-openmp-2021.4.0      |    haa95532_3556         2.2 MB
    libboost-1.73.0            |      h6c2663c_11        20.3 MB
    libcurl-7.80.0             |       h86230a5_0         295 KB
    libprotobuf-3.11.2         |       h7bd577a_0         1.8 MB
    libssh2-1.9.0              |       h7a1dbc1_1         215 KB
    lz4-c-1.9.3                |       h2bbff1b_1         132 KB
    mkl-2021.4.0               |     haa95532_640       114.9 MB
    mkl-service-2.4.0          |   py39h2bbff1b_0          51 KB
    mkl_fft-1.3.1              |   py39h277e83a_0         139 KB
    mkl_random-1.2.2           |   py39hf11a4ad_0         225 KB
    numexpr-2.8.1              |   py39hb80d3ca_0         117 KB
    numpy-1.21.2               |   py39hfca59bb_0          24 KB
    numpy-base-1.21.2          |   py39h0829f74_0         4.4 MB
    pandas-1.3.5               |   py39h6214cd6_0         8.6 MB
    pyarrow-4.0.1              |   py39h953b917_3         1.7 MB
    re2-2020.11.01             |       hd77b12b_1         892 KB
    uriparser-0.9.3            |       h33f27b4_1          45 KB
    utf8proc-2.6.1             |       h2bbff1b_0         312 KB
    xz-5.2.5                   |       h62dcd97_0         244 KB
    zlib-1.2.11                |       h8cc25b3_4         112 KB
    zstd-1.4.9                 |       h19a0ad4_0         478 KB
    ------------------------------------------------------------
                                           Total:       170.2 MB

The following NEW packages will be INSTALLED:

  arrow-cpp          pkgs/main/win-64::arrow-cpp-4.0.1-py39h0d1d0e5_3
  aws-c-common       pkgs/main/win-64::aws-c-common-0.4.57-ha925a31_1
  aws-c-event-stream pkgs/main/win-64::aws-c-event-stream-0.1.6-hd77b12b_5
  aws-checksums      pkgs/main/win-64::aws-checksums-0.1.9-ha925a31_0
  aws-sdk-cpp        pkgs/main/win-64::aws-sdk-cpp-1.8.185-hd77b12b_0
  blas               pkgs/main/win-64::blas-1.0-mkl
  boost-cpp          pkgs/main/win-64::boost-cpp-1.73.0-h2bbff1b_11
  bottleneck         pkgs/main/win-64::bottleneck-1.3.2-py39h7cc1a96_1
  brotli             pkgs/main/win-64::brotli-1.0.9-ha925a31_2
  bzip2              pkgs/main/win-64::bzip2-1.0.8-he774522_0
  c-ares             pkgs/main/win-64::c-ares-1.18.1-h2bbff1b_0
  double-conversion  pkgs/main/win-64::double-conversion-3.1.5-ha925a31_1
  gflags             pkgs/main/win-64::gflags-2.2.2-ha925a31_0
  glog               pkgs/main/win-64::glog-0.5.0-hd77b12b_0
  grpc-cpp           pkgs/main/win-64::grpc-cpp-1.26.0-h351948d_0
  intel-openmp       pkgs/main/win-64::intel-openmp-2021.4.0-haa95532_3556
  libboost           pkgs/main/win-64::libboost-1.73.0-h6c2663c_11
  libcurl            pkgs/main/win-64::libcurl-7.80.0-h86230a5_0
  libprotobuf        pkgs/main/win-64::libprotobuf-3.11.2-h7bd577a_0
  libssh2            pkgs/main/win-64::libssh2-1.9.0-h7a1dbc1_1
  lz4-c              pkgs/main/win-64::lz4-c-1.9.3-h2bbff1b_1
  mkl                pkgs/main/win-64::mkl-2021.4.0-haa95532_640
  mkl-service        pkgs/main/win-64::mkl-service-2.4.0-py39h2bbff1b_0
  mkl_fft            pkgs/main/win-64::mkl_fft-1.3.1-py39h277e83a_0
  mkl_random         pkgs/main/win-64::mkl_random-1.2.2-py39hf11a4ad_0
  numexpr            pkgs/main/win-64::numexpr-2.8.1-py39hb80d3ca_0
  numpy              pkgs/main/win-64::numpy-1.21.2-py39hfca59bb_0
  numpy-base         pkgs/main/win-64::numpy-base-1.21.2-py39h0829f74_0
  pandas             pkgs/main/win-64::pandas-1.3.5-py39h6214cd6_0
  pyarrow            pkgs/main/win-64::pyarrow-4.0.1-py39h953b917_3
  re2                pkgs/main/win-64::re2-2020.11.01-hd77b12b_1
  snappy             pkgs/main/win-64::snappy-1.1.8-h33f27b4_0
  uriparser          pkgs/main/win-64::uriparser-0.9.3-h33f27b4_1
  utf8proc           pkgs/main/win-64::utf8proc-2.6.1-h2bbff1b_0
  xz                 pkgs/main/win-64::xz-5.2.5-h62dcd97_0
  zlib               pkgs/main/win-64::zlib-1.2.11-h8cc25b3_4
  zstd               pkgs/main/win-64::zstd-1.4.9-h19a0ad4_0

The following packages will be UPDATED:

  ca-certificates    conda-forge::ca-certificates-2021.10.~ --> pkgs/main::ca-certificates-2021.10.26-haa95532_4
  certifi            conda-forge::certifi-2021.10.8-py39hc~ --> pkgs/main::certifi-2021.10.8-py39haa95532_2
  openssl            conda-forge::openssl-1.1.1l-h8ffe710_0 --> pkgs/main::openssl-1.1.1m-h2bbff1b_0

The following packages will be SUPERSEDED by a higher-priority channel:

  conda              conda-forge::conda-4.11.0-py39hcbf530~ --> pkgs/main::conda-4.11.0-py39haa95532_0

Uninstalling it: conda uninstall pyarrow

## Package Plan ##

  environment location: C:\Miniconda3

  removed specs:
    - pyarrow

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    anyio-2.2.0                |   py39haa95532_2         125 KB
    argon2-cffi-20.1.0         |   py39h2bbff1b_1          49 KB
    async_generator-1.10       |     pyhd3eb1b0_0          23 KB
    debugpy-1.5.1              |   py39hd77b12b_0         2.6 MB
    entrypoints-0.3            |   py39haa95532_0          12 KB
    importlib-metadata-4.8.2   |   py39haa95532_0          40 KB
    ipykernel-6.4.1            |   py39haa95532_1         195 KB
    ipython-7.29.0             |   py39hd4e2768_0         1.0 MB
    jedi-0.18.0                |   py39haa95532_1         912 KB
    jupyter_core-4.9.1         |   py39haa95532_0          91 KB
    jupyter_server-1.4.1       |   py39haa95532_0         328 KB
    markupsafe-2.0.1           |   py39h2bbff1b_0          24 KB
    mistune-0.8.4              |py39h2bbff1b_1000          57 KB
    nbconvert-6.1.0            |   py39haa95532_0         501 KB
    notebook-6.4.6             |   py39haa95532_0         4.5 MB
    pyrsistent-0.18.0          |   py39h196d8e1_0          90 KB
    pyzmq-22.3.0               |   py39hd77b12b_2         632 KB
    sniffio-1.2.0              |   py39haa95532_1          15 KB
    tornado-6.1                |   py39h2bbff1b_0         598 KB
    webencodings-0.5.1         |   py39haa95532_1          20 KB
    ------------------------------------------------------------
                                           Total:        11.8 MB

The following NEW packages will be INSTALLED:

  async_generator    pkgs/main/noarch::async_generator-1.10-pyhd3eb1b0_0
  importlib_metadata pkgs/main/noarch::importlib_metadata-4.8.2-hd3eb1b0_0

The following packages will be REMOVED:

  argon2-cffi-bindings-21.2.0-py39hb82d6ee_1
  arrow-cpp-4.0.1-py39h0d1d0e5_3
  asttokens-2.0.5-pyhd8ed1ab_0
  aws-c-common-0.4.57-ha925a31_1
  aws-c-event-stream-0.1.6-hd77b12b_5
  aws-checksums-0.1.9-ha925a31_0
  aws-sdk-cpp-1.8.185-hd77b12b_0
  backports-1.0-py_2
  backports.functools_lru_cache-1.6.4-pyhd8ed1ab_0
  black-22.1.0-pyhd8ed1ab_0
  blas-1.0-mkl
  boost-cpp-1.73.0-h2bbff1b_11
  bottleneck-1.3.2-py39h7cc1a96_1
  brotli-1.0.9-ha925a31_2
  bzip2-1.0.8-he774522_0
  c-ares-1.18.1-h2bbff1b_0
  click-8.0.3-py39hcbf5309_1
  dataclasses-0.8-pyhc8e2a94_3
  double-conversion-3.1.5-ha925a31_1
  executing-0.8.2-pyhd8ed1ab_0
  flit-core-3.6.0-pyhd8ed1ab_0
  gflags-2.2.2-ha925a31_0
  glog-0.5.0-hd77b12b_0
  grpc-cpp-1.26.0-h351948d_0
  importlib_resources-5.4.0-pyhd8ed1ab_0
  intel-openmp-2021.4.0-haa95532_3556
  libboost-1.73.0-h6c2663c_11
  libcurl-7.80.0-h86230a5_0
  libprotobuf-3.11.2-h7bd577a_0
  libsodium-1.0.18-h8d14728_1
  libssh2-1.9.0-h7a1dbc1_1
  lz4-c-1.9.3-h2bbff1b_1
  mkl-2021.4.0-haa95532_640
  mkl-service-2.4.0-py39h2bbff1b_0
  mkl_fft-1.3.1-py39h277e83a_0
  mkl_random-1.2.2-py39hf11a4ad_0
  mypy_extensions-0.4.3-py39hcbf5309_4
  numexpr-2.8.1-py39hb80d3ca_0
  numpy-1.21.2-py39hfca59bb_0
  numpy-base-1.21.2-py39h0829f74_0
  pandas-1.3.5-py39h6214cd6_0
  pandoc-2.17.1.1-h57928b3_0
  pathspec-0.9.0-pyhd8ed1ab_0
  platformdirs-2.3.0-pyhd8ed1ab_0
  pure_eval-0.2.2-pyhd8ed1ab_0
  pyarrow-4.0.1-py39h953b917_3
  python_abi-3.9-2_cp39
  re2-2020.11.01-hd77b12b_1
  snappy-1.1.8-h33f27b4_0
  stack_data-0.1.4-pyhd8ed1ab_0
  tomli-2.0.0-pyhd8ed1ab_1
  typed-ast-1.5.2-py39hb82d6ee_0
  typing_extensions-4.0.1-pyha770c72_0
  uriparser-0.9.3-h33f27b4_1
  utf8proc-2.6.1-h2bbff1b_0
  websocket-client-1.2.3-pyhd8ed1ab_0
  xz-5.2.5-h62dcd97_0
  zeromq-4.3.4-h0e60522_1
  zlib-1.2.11-h8cc25b3_4
  zstd-1.4.9-h19a0ad4_0

The following packages will be UPDATED:

  json5               conda-forge::json5-0.9.5-pyh9f0ad1d_0 --> pkgs/main::json5-0.9.6-pyhd3eb1b0_0
  pyzmq              conda-forge::pyzmq-22.3.0-py39he46f08~ --> pkgs/main::pyzmq-22.3.0-py39hd77b12b_2
  send2trash         conda-forge::send2trash-1.8.0-pyhd8ed~ --> pkgs/main::send2trash-1.8.0-pyhd3eb1b0_1

The following packages will be SUPERSEDED by a higher-priority channel:

  anyio              conda-forge::anyio-3.5.0-py39hcbf5309~ --> pkgs/main::anyio-2.2.0-py39haa95532_2
  argon2-cffi        conda-forge/noarch::argon2-cffi-21.3.~ --> pkgs/main/win-64::argon2-cffi-20.1.0-py39h2bbff1b_1
  attrs              conda-forge::attrs-21.4.0-pyhd8ed1ab_0 --> pkgs/main::attrs-21.4.0-pyhd3eb1b0_0
  babel               conda-forge::babel-2.9.1-pyh44b312d_0 --> pkgs/main::babel-2.9.1-pyhd3eb1b0_0
  backcall           conda-forge::backcall-0.2.0-pyh9f0ad1~ --> pkgs/main::backcall-0.2.0-pyhd3eb1b0_0
  bleach             conda-forge::bleach-4.1.0-pyhd8ed1ab_0 --> pkgs/main::bleach-4.1.0-pyhd3eb1b0_0
  colorama           conda-forge::colorama-0.4.4-pyh9f0ad1~ --> pkgs/main::colorama-0.4.4-pyhd3eb1b0_0
  debugpy            conda-forge::debugpy-1.5.1-py39h415ef~ --> pkgs/main::debugpy-1.5.1-py39hd77b12b_0
  decorator          conda-forge::decorator-5.1.1-pyhd8ed1~ --> pkgs/main::decorator-5.1.1-pyhd3eb1b0_0
  defusedxml         conda-forge::defusedxml-0.7.1-pyhd8ed~ --> pkgs/main::defusedxml-0.7.1-pyhd3eb1b0_0
  entrypoints        conda-forge/noarch::entrypoints-0.3-p~ --> pkgs/main/win-64::entrypoints-0.3-py39haa95532_0
  importlib-metadata conda-forge::importlib-metadata-4.10.~ --> pkgs/main::importlib-metadata-4.8.2-py39haa95532_0
  ipykernel          conda-forge::ipykernel-6.7.0-py39h832~ --> pkgs/main::ipykernel-6.4.1-py39haa95532_1
  ipython            conda-forge::ipython-8.0.1-py39hcbf53~ --> pkgs/main::ipython-7.29.0-py39hd4e2768_0
  ipython_genutils   conda-forge::ipython_genutils-0.2.0-p~ --> pkgs/main::ipython_genutils-0.2.0-pyhd3eb1b0_1
  jedi               conda-forge::jedi-0.18.1-py39hcbf5309~ --> pkgs/main::jedi-0.18.0-py39haa95532_1
  jinja2             conda-forge::jinja2-3.0.3-pyhd8ed1ab_0 --> pkgs/main::jinja2-3.0.2-pyhd3eb1b0_0
  jsonschema         conda-forge::jsonschema-4.4.0-pyhd8ed~ --> pkgs/main::jsonschema-3.2.0-pyhd3eb1b0_2
  jupyter_client     conda-forge::jupyter_client-7.1.2-pyh~ --> pkgs/main::jupyter_client-7.1.2-pyhd3eb1b0_0
  jupyter_core       conda-forge::jupyter_core-4.9.1-py39h~ --> pkgs/main::jupyter_core-4.9.1-py39haa95532_0
  jupyter_server     conda-forge/noarch::jupyter_server-1.~ --> pkgs/main/win-64::jupyter_server-1.4.1-py39haa95532_0
  jupyterlab_pygmen~ conda-forge::jupyterlab_pygments-0.1.~ --> pkgs/main::jupyterlab_pygments-0.1.2-py_0
  jupyterlab_server  conda-forge::jupyterlab_server-2.10.3~ --> pkgs/main::jupyterlab_server-2.10.2-pyhd3eb1b0_1
  markupsafe         conda-forge::markupsafe-2.0.1-py39hb8~ --> pkgs/main::markupsafe-2.0.1-py39h2bbff1b_0
  matplotlib-inline  conda-forge::matplotlib-inline-0.1.3-~ --> pkgs/main::matplotlib-inline-0.1.2-pyhd3eb1b0_2
  mistune            conda-forge::mistune-0.8.4-py39hb82d6~ --> pkgs/main::mistune-0.8.4-py39h2bbff1b_1000
  nbclassic          conda-forge::nbclassic-0.3.5-pyhd8ed1~ --> pkgs/main::nbclassic-0.2.6-pyhd3eb1b0_0
  nbclient           conda-forge::nbclient-0.5.10-pyhd8ed1~ --> pkgs/main::nbclient-0.5.3-pyhd3eb1b0_0
  nbconvert          conda-forge::nbconvert-6.4.1-py39hcbf~ --> pkgs/main::nbconvert-6.1.0-py39haa95532_0
  nbformat           conda-forge::nbformat-5.1.3-pyhd8ed1a~ --> pkgs/main::nbformat-5.1.3-pyhd3eb1b0_0
  nest-asyncio       conda-forge::nest-asyncio-1.5.4-pyhd8~ --> pkgs/main::nest-asyncio-1.5.1-pyhd3eb1b0_0
  notebook           conda-forge/noarch::notebook-6.4.8-py~ --> pkgs/main/win-64::notebook-6.4.6-py39haa95532_0
  packaging          conda-forge::packaging-21.3-pyhd8ed1a~ --> pkgs/main::packaging-21.3-pyhd3eb1b0_0
  pandocfilters      conda-forge::pandocfilters-1.5.0-pyhd~ --> pkgs/main::pandocfilters-1.5.0-pyhd3eb1b0_0
  parso               conda-forge::parso-0.8.3-pyhd8ed1ab_0 --> pkgs/main::parso-0.8.3-pyhd3eb1b0_0
  pickleshare        conda-forge::pickleshare-0.7.5-py_1003 --> pkgs/main::pickleshare-0.7.5-pyhd3eb1b0_1003
  prometheus_client  conda-forge::prometheus_client-0.13.1~ --> pkgs/main::prometheus_client-0.12.0-pyhd3eb1b0_0
  prompt-toolkit     conda-forge::prompt-toolkit-3.0.26-py~ --> pkgs/main::prompt-toolkit-3.0.20-pyhd3eb1b0_0
  pygments           conda-forge::pygments-2.11.2-pyhd8ed1~ --> pkgs/main::pygments-2.10.0-pyhd3eb1b0_0
  pyparsing          conda-forge::pyparsing-3.0.7-pyhd8ed1~ --> pkgs/main::pyparsing-3.0.4-pyhd3eb1b0_0
  pyrsistent         conda-forge::pyrsistent-0.18.1-py39hb~ --> pkgs/main::pyrsistent-0.18.0-py39h196d8e1_0
  python-dateutil    conda-forge::python-dateutil-2.8.2-py~ --> pkgs/main::python-dateutil-2.8.2-pyhd3eb1b0_0
  pytz                conda-forge::pytz-2021.3-pyhd8ed1ab_0 --> pkgs/main::pytz-2021.3-pyhd3eb1b0_0
  sniffio            conda-forge::sniffio-1.2.0-py39hcbf53~ --> pkgs/main::sniffio-1.2.0-py39haa95532_1
  testpath           conda-forge::testpath-0.5.0-pyhd8ed1a~ --> pkgs/main::testpath-0.5.0-pyhd3eb1b0_0
  tornado            conda-forge::tornado-6.1-py39hb82d6ee~ --> pkgs/main::tornado-6.1-py39h2bbff1b_0
  traitlets          conda-forge::traitlets-5.1.1-pyhd8ed1~ --> pkgs/main::traitlets-5.1.1-pyhd3eb1b0_0
  wcwidth            conda-forge::wcwidth-0.2.5-pyh9f0ad1d~ --> pkgs/main::wcwidth-0.2.5-pyhd3eb1b0_0
  webencodings       conda-forge/noarch::webencodings-0.5.~ --> pkgs/main/win-64::webencodings-0.5.1-py39haa95532_1
  winpty                                        conda-forge --> pkgs/main
  zipp                 conda-forge::zipp-3.7.0-pyhd8ed1ab_0 --> pkgs/main::zipp-3.7.0-pyhd3eb1b0_0

Installing from conda-forge conda install -c conda-forge pyarrow

## Package Plan ##

  environment location: C:\Miniconda3

  added / updated specs:
    - pyarrow

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    abseil-cpp-20210324.2      |       h0e60522_0         2.1 MB  conda-forge
    arrow-cpp-6.0.1            |py39h591eb41_8_cpu        15.8 MB  conda-forge
    aws-c-cal-0.5.11           |       he19cf47_0          36 KB  conda-forge
    aws-c-common-0.6.2         |       h8ffe710_0         159 KB  conda-forge
    aws-c-event-stream-0.2.7   |      h70e1b0c_13          47 KB  conda-forge
    aws-c-io-0.10.5            |       h2fe331c_0         127 KB  conda-forge
    aws-checksums-0.1.11       |       h1e232aa_7          51 KB  conda-forge
    aws-sdk-cpp-1.8.186        |       hb0612c5_3         5.5 MB  conda-forge
    bzip2-1.0.8                |       h8ffe710_4         149 KB  conda-forge
    c-ares-1.18.1              |       h8ffe710_0         114 KB  conda-forge
    gflags-2.2.2               |    ha925a31_1004          80 KB  conda-forge
    glog-0.5.0                 |       h4797de2_0          90 KB  conda-forge
    grpc-cpp-1.42.0            |       ha2e5525_1        19.1 MB  conda-forge
    intel-openmp-2022.0.0      |    h57928b3_3663         3.6 MB  conda-forge
    krb5-1.19.2                |       h20d022d_3         844 KB  conda-forge
    libblas-3.9.0              |     13_win64_mkl         5.3 MB  conda-forge
    libbrotlicommon-1.0.9      |       h8ffe710_6          67 KB  conda-forge
    libbrotlidec-1.0.9         |       h8ffe710_6          33 KB  conda-forge
    libbrotlienc-1.0.9         |       h8ffe710_6         721 KB  conda-forge
    libcblas-3.9.0             |     13_win64_mkl         5.3 MB  conda-forge
    libcurl-7.81.0             |       h789b8ee_0         302 KB  conda-forge
    liblapack-3.9.0            |     13_win64_mkl         5.3 MB  conda-forge
    libprotobuf-3.19.4         |       h7755175_0         2.4 MB  conda-forge
    libssh2-1.10.0             |       h680486a_2         227 KB  conda-forge
    libthrift-0.15.0           |       h636ae23_1         865 KB  conda-forge
    libutf8proc-2.7.0          |       hcb41399_0         101 KB  conda-forge
    libzlib-1.2.11             |    h8ffe710_1013          63 KB  conda-forge
    lz4-c-1.9.3                |       h8ffe710_1         135 KB  conda-forge
    mkl-2022.0.0               |     h0e2418a_796       181.9 MB  conda-forge
    numpy-1.22.1               |   py39h6331f09_0         6.1 MB  conda-forge
    parquet-cpp-1.5.1          |                2           3 KB  conda-forge
    pyarrow-6.0.1              |py39ha05331a_8_cpu         2.5 MB  conda-forge
    re2-2021.11.01             |       h0e60522_0         476 KB  conda-forge
    snappy-1.1.8               |       ha925a31_3          50 KB  conda-forge
    tbb-2021.5.0               |       h2d74725_0         148 KB  conda-forge
    tk-8.6.11                  |       h8ffe710_1         3.8 MB  conda-forge
    xz-5.2.5                   |       h62dcd97_1         211 KB  conda-forge
    zlib-1.2.11                |    h8ffe710_1013         106 KB  conda-forge
    zstd-1.5.2                 |       h6255e5f_0        1010 KB  conda-forge
    ------------------------------------------------------------
                                           Total:       264.5 MB

The following NEW packages will be INSTALLED:

  abseil-cpp         conda-forge/win-64::abseil-cpp-20210324.2-h0e60522_0
  arrow-cpp          conda-forge/win-64::arrow-cpp-6.0.1-py39h591eb41_8_cpu
  aws-c-cal          conda-forge/win-64::aws-c-cal-0.5.11-he19cf47_0
  aws-c-common       conda-forge/win-64::aws-c-common-0.6.2-h8ffe710_0
  aws-c-event-stream conda-forge/win-64::aws-c-event-stream-0.2.7-h70e1b0c_13
  aws-c-io           conda-forge/win-64::aws-c-io-0.10.5-h2fe331c_0
  aws-checksums      conda-forge/win-64::aws-checksums-0.1.11-h1e232aa_7
  aws-sdk-cpp        conda-forge/win-64::aws-sdk-cpp-1.8.186-hb0612c5_3
  bzip2              conda-forge/win-64::bzip2-1.0.8-h8ffe710_4
  c-ares             conda-forge/win-64::c-ares-1.18.1-h8ffe710_0
  gflags             conda-forge/win-64::gflags-2.2.2-ha925a31_1004
  glog               conda-forge/win-64::glog-0.5.0-h4797de2_0
  grpc-cpp           conda-forge/win-64::grpc-cpp-1.42.0-ha2e5525_1
  intel-openmp       conda-forge/win-64::intel-openmp-2022.0.0-h57928b3_3663
  krb5               conda-forge/win-64::krb5-1.19.2-h20d022d_3
  libblas            conda-forge/win-64::libblas-3.9.0-13_win64_mkl
  libbrotlicommon    conda-forge/win-64::libbrotlicommon-1.0.9-h8ffe710_6
  libbrotlidec       conda-forge/win-64::libbrotlidec-1.0.9-h8ffe710_6
  libbrotlienc       conda-forge/win-64::libbrotlienc-1.0.9-h8ffe710_6
  libcblas           conda-forge/win-64::libcblas-3.9.0-13_win64_mkl
  libcurl            conda-forge/win-64::libcurl-7.81.0-h789b8ee_0
  liblapack          conda-forge/win-64::liblapack-3.9.0-13_win64_mkl
  libprotobuf        conda-forge/win-64::libprotobuf-3.19.4-h7755175_0
  libssh2            conda-forge/win-64::libssh2-1.10.0-h680486a_2
  libthrift          conda-forge/win-64::libthrift-0.15.0-h636ae23_1
  libutf8proc        conda-forge/win-64::libutf8proc-2.7.0-hcb41399_0
  libzlib            conda-forge/win-64::libzlib-1.2.11-h8ffe710_1013
  lz4-c              conda-forge/win-64::lz4-c-1.9.3-h8ffe710_1
  mkl                conda-forge/win-64::mkl-2022.0.0-h0e2418a_796
  numpy              conda-forge/win-64::numpy-1.22.1-py39h6331f09_0
  parquet-cpp        conda-forge/noarch::parquet-cpp-1.5.1-2
  pyarrow            conda-forge/win-64::pyarrow-6.0.1-py39ha05331a_8_cpu
  python_abi         conda-forge/win-64::python_abi-3.9-2_cp39
  re2                conda-forge/win-64::re2-2021.11.01-h0e60522_0
  snappy             conda-forge/win-64::snappy-1.1.8-ha925a31_3
  tbb                conda-forge/win-64::tbb-2021.5.0-h2d74725_0
  tk                 conda-forge/win-64::tk-8.6.11-h8ffe710_1
  xz                 conda-forge/win-64::xz-5.2.5-h62dcd97_1
  zlib               conda-forge/win-64::zlib-1.2.11-h8ffe710_1013
  zstd               conda-forge/win-64::zstd-1.5.2-h6255e5f_0

The following packages will be SUPERSEDED by a higher-priority channel:

  ca-certificates    pkgs/main::ca-certificates-2021.10.26~ --> conda-forge::ca-certificates-2021.10.8-h5b45459_0
  certifi            pkgs/main::certifi-2021.10.8-py39haa9~ --> conda-forge::certifi-2021.10.8-py39hcbf5309_1
  conda              pkgs/main::conda-4.11.0-py39haa95532_0 --> conda-forge::conda-4.11.0-py39hcbf5309_0
  openssl              pkgs/main::openssl-1.1.1m-h2bbff1b_0 --> conda-forge::openssl-1.1.1l-h8ffe710_0

This resulted in the same error message. Then I installed snappy with pip pip install snappy --user

Collecting snappy
  Downloading snappy-3.0.3-cp39-cp39-win_amd64.whl (11.6 MB)
     |████████████████████████████████| 11.6 MB 226 kB/s
Collecting FXrays>=1.3
  Downloading FXrays-1.3.5-cp39-cp39-win_amd64.whl (25 kB)
Requirement already satisfied: ipython>=5.0 in c:\miniconda3\lib\site-packages (from snappy) (7.29.0)
Requirement already satisfied: decorator in c:\miniconda3\lib\site-packages (from snappy) (5.1.1)
Collecting spherogram>=2.1
  Downloading spherogram-2.1-cp39-cp39-win_amd64.whl (319 kB)
     |████████████████████████████████| 319 kB 930 kB/s
Collecting plink>=2.4.1
  Using cached plink-2.4.1-py3-none-any.whl (339 kB)
Collecting cypari>=2.3
  Downloading cypari-2.4.1-cp39-cp39-win_amd64.whl (5.2 MB)
     |████████████████████████████████| 5.2 MB 6.8 MB/s
Collecting pypng
  Using cached pypng-0.0.21-py3-none-any.whl (48 kB)
Collecting snappy-manifolds>=1.1.2
  Using cached snappy_manifolds-1.1.2-py3-none-any.whl (45.0 MB)
Collecting future
  Using cached future-0.18.2.tar.gz (829 kB)
Requirement already satisfied: six in c:\miniconda3\lib\site-packages (from cypari>=2.3->snappy) (1.16.0)
Requirement already satisfied: pickleshare in c:\miniconda3\lib\site-packages (from ipython>=5.0->snappy) (0.7.5)
Requirement already satisfied: jedi>=0.16 in c:\miniconda3\lib\site-packages (from ipython>=5.0->snappy) (0.18.0)
Requirement already satisfied: traitlets>=4.2 in c:\miniconda3\lib\site-packages (from ipython>=5.0->snappy) (5.1.1)
Requirement already satisfied: matplotlib-inline in c:\miniconda3\lib\site-packages (from ipython>=5.0->snappy) (0.1.2)
Requirement already satisfied: backcall in c:\miniconda3\lib\site-packages (from ipython>=5.0->snappy) (0.2.0)
Requirement already satisfied: colorama in c:\miniconda3\lib\site-packages (from ipython>=5.0->snappy) (0.4.4)
Requirement already satisfied: setuptools>=18.5 in c:\miniconda3\lib\site-packages (from ipython>=5.0->snappy) (58.0.4)
Requirement already satisfied: pygments in c:\miniconda3\lib\site-packages (from ipython>=5.0->snappy) (2.10.0)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in c:\miniconda3\lib\site-packages (from ipython>=5.0->snappy) (3.0.20)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in c:\miniconda3\lib\site-packages (from jedi>=0.16->ipython>=5.0->snappy) (0.8.3)
Requirement already satisfied: wcwidth in c:\miniconda3\lib\site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=5.0->snappy) (0.2.5)
Collecting networkx
  Using cached networkx-2.6.3-py3-none-any.whl (1.9 MB)
Collecting knot-floer-homology>=1.1
  Downloading knot_floer_homology-1.1-cp39-cp39-win_amd64.whl (80 kB)
     |████████████████████████████████| 80 kB 1.2 MB/s
Building wheels for collected packages: future
  Building wheel for future (setup.py) ... done
  Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491070 sha256=993dc6610319fad5ba0ddcbe4d7b5ecd2bc790cdbb92d8626b050bc758c0789e
  Stored in directory: c:\users\XXXX\appdata\local\pip\cache\wheels\2f\a0\d3\4030d9f80e6b3be787f19fc911b8e7aa462986a40ab1e4bb94
Successfully built future
Installing collected packages: snappy-manifolds, networkx, knot-floer-homology, future, spherogram, pypng, plink, FXrays, cypari, snappy
Successfully installed FXrays-1.3.5 cypari-2.4.1 future-0.18.2 knot-floer-homology-1.1 networkx-2.6.3 plink-2.4.1 pypng-0.0.21 snappy-3.0.3 snappy-manifolds-1.1.2 spherogram-2.1

onacrame commented 2 years ago

I'm having the same exact problem on Windows, cannot load a file saved with snappy compression. Only works for me if installing pyarrow from conda-forge

ContinuumIO / anaconda-issues