flatironinstitute / CaImAn

Computational toolbox for large scale Calcium Imaging Analysis, including movie handling, motion correction, source extraction, spike deconvolution and result visualization.
https://caiman.readthedocs.io
GNU General Public License v2.0
640 stars 370 forks source link

Caiman version on windows #1199

Closed noah-chongsiriwatana closed 1 year ago

noah-chongsiriwatana commented 1 year ago

Hello,

I am trying to run caiman on windows. In a clean conda environment I ran mamba install -c conda-forge caiman which shows the packages and versions to be installed. I noticed that the version for caiman is 1.9.8, and found that this is because the conda-forge channel for windows supports tensorflow version at 1 rather than 2. Do you have any recommendations on how to use the most up to date version of caiman?

Thank you!

Your setup:

  1. Operating System (Linux, MacOS, Windows):
  2. Hardware type (x86, ARM..) and RAM:
  3. Python Version (e.g. 3.9):
  4. Caiman version (e.g. 1.9.12):
  5. Which demo exhibits the problem (if applicable):
  6. How you installed Caiman (pure conda, conda + compile, colab, ..):
  7. Details:
EricThomson commented 1 year ago

Did you get this sorted? You should have 1.9.16 if you install using conda/mamba and follow the quickstart guide on the main caiman page: https://github.com/flatironinstitute/CaImAn

noah-chongsiriwatana commented 1 year ago

When I try to run mamba create -n caiman -c conda-forge caiman I get

Transaction

  Prefix: C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\envs\caiman

  Updating specs:

   - caiman

  Package                                     Version  Build                   Channel                  Size
--------------------------------------------------------------------------------------------------------------
  Install:
--------------------------------------------------------------------------------------------------------------

  + absl-py                                     2.0.0  pyhd8ed1ab_0            conda-forge/noarch     Cached
  + anyio                                       3.7.1  pyhd8ed1ab_0            conda-forge/noarch     Cached
  + aom                                         3.6.1  h63175ca_0              conda-forge/win-64     Cached
  + argon2-cffi                                23.1.0  pyhd8ed1ab_0            conda-forge/noarch     Cached
  + argon2-cffi-bindings                       21.2.0  py37hcc03f2d_2          conda-forge/win-64     Cached
  + astor                                       0.8.1  pyh9f0ad1d_0            conda-forge/noarch     Cached
  + attrs                                      23.1.0  pyh71513ae_1            conda-forge/noarch     Cached
  + backcall                                    0.2.0  pyh9f0ad1d_0            conda-forge/noarch     Cached
  + backports                                     1.0  pyhd8ed1ab_3            conda-forge/noarch     Cached
  + backports.functools_lru_cache               1.6.5  pyhd8ed1ab_0            conda-forge/noarch     Cached
  + beautifulsoup4                             4.12.2  pyha770c72_0            conda-forge/noarch     Cached
  + bleach                                      6.1.0  pyhd8ed1ab_0            conda-forge/noarch     Cached
  + blosc                                      1.21.5  hdccc3a2_0              conda-forge/win-64     Cached
  + bokeh                                       2.4.3  pyhd8ed1ab_3            conda-forge/noarch     Cached
  + brotli                                      1.1.0  hcfcfb64_1              conda-forge/win-64     Cached
  + brotli-bin                                  1.1.0  hcfcfb64_1              conda-forge/win-64     Cached
  + brotli-python                               1.0.9  py37hf2a7229_7          conda-forge/win-64     Cached
  + bzip2                                       1.0.8  h8ffe710_4              conda-forge/win-64     Cached
  + c-ares                                     1.20.1  hcfcfb64_0              conda-forge/win-64     Cached
  + ca-certificates                         2023.7.22  h56e8100_0              conda-forge/win-64     Cached
  + caiman                                      1.9.8  py37h9386db6_0          conda-forge/win-64     Cached
...
  + sqlite                                     3.43.2  hcfcfb64_0              conda-forge/win-64     Cached
  + svt-av1                                     1.4.1  h63175ca_0              conda-forge/win-64     Cached
  + tbb                                     2021.10.0  h91493d7_2              conda-forge/win-64      157kB
  + tensorboard                                1.14.0  py37_0                  conda-forge/win-64     Cached
  + tensorflow                                 1.14.0  h1f41ff6_0              conda-forge/win-64     Cached
  + tensorflow-base                            1.14.0  py37hc8dfbb8_0          conda-forge/win-64     Cached
  + tensorflow-estimator                       1.14.0  py37h5ca1d4c_0          conda-forge/win-64     Cached
  + termcolor                                   2.3.0  pyhd8ed1ab_0            conda-forge/noarch     Cached
  + terminado                                  0.17.0  pyh08f2357_0            conda-forge/noarch     Cached

I've tried reinstalling mamba from scratch and clearing all packages and caches. This is my system info:

  1. Operating System: Windows
  2. Hardware type: 64-bit operating system, x64-based processor, 256 GB RAM
pgunn commented 1 year ago

Hello, I see from the above that every single package is coming from conda-forge. That doesn't work on Windows with anything that needs tensorflow.

Try this instead:

mamba create -n caiman --override-channels -c base -c conda-forge caiman

That should get mamba looking in both places and hopefully will let it find the newer tensorflow.

noah-chongsiriwatana commented 1 year ago

Thank you for the suggestion, I tried that instead but am still seeing tensorflow 1.14 from conda-forge. I previously tried getting tensorflow from the anaconda channel, but the package seemed to be corrupted from that channel.

pgunn commented 1 year ago

Hm. Do you get the same results with this?

mamba create -n caiman --override-channels -c conda-forge -c base caiman

noah-chongsiriwatana commented 1 year ago

Unfortunately yes. I noticed in the Gitter chat that another user encountered the same issue as me. I also had the same @lru_cache issue, but I have already been using mambaforge (specifically miniforge-pypy3).

pgunn commented 1 year ago

I was pretty sure that was the case; we need to convince mambaforge to use both the defaults channel and the conda-forge channel in order to get this to resolve correctly. Let me see if I can figure out the right way to do that.

noah-chongsiriwatana commented 1 year ago

Looking at the mambaforge github, I notice that as of September mambaforge is discouraged, and miniforge is apparently identical to it now. This may be the issue, as I was able to install and use caiman earlier in the summer on the same machine, and only recently broke it trying to update caiman.

pgunn commented 1 year ago

What happens if we specifically tell it what tensorflow to use?

e.g.

mamba create -n caiman -c base -c conda-forge caiman tensorflow=2.12.0

pgunn commented 1 year ago

Sorry, that'll have to be

mamba create -n caiman -c base -c conda-forge caiman tensorflow=2.10.0

Looks like they're a little behind on packaging on Windows right now in base

noah-chongsiriwatana commented 1 year ago

This is the result:

$ mamba create -n caiman -c base -c conda-forge caiman tensorflow=2.10.0
conda-forge/win-64                                          Using cache
conda-forge/noarch                                          Using cache

Looking for: ['caiman', 'tensorflow=2.10.0']

Could not solve for environment specs
The following package could not be installed
└─ tensorflow 2.10.0**  does not exist (perhaps a typo or a missing channel).

I have previously tried creating an environment, installing tensorflow first via https://anaconda.org/anaconda/tensorflow, then installing caiman, but this resulted in corrupted tensorflow package.

pgunn commented 1 year ago

Oh goodness I am an idiot.

Try this:

mamba create -n caiman -c defaults -c conda-forge caiman tensorflow=2.10.0

Sorry, for some reason I was pulling up the wrong word.

noah-chongsiriwatana commented 1 year ago

This is progress as it does attempt to install the correct versions of tensorflow and caiman. However in the verifying transaction stage, there are corrupted packages and the install fails:

Looking for: ['caiman', 'tensorflow=2.10.0']

Downloading and Extracting Packages

Preparing transaction: ...working... done
Verifying transaction: ...working... failed
The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/AffineMemoryOpInterfacesIncGen/mlir/Dialect/Affine/IR/AffineMemoryOpInterfaces.cpp.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/AffineMemoryOpInterfacesIncGen/mlir/Dialect/Affine/IR/AffineMemoryOpInterfaces.h.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/AllocationOpInterfaceIncGen/mlir/Dialect/Bufferization/IR/AllocationOpInterface.cpp.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/AllocationOpInterfaceIncGen/mlir/Dialect/Bufferization/IR/AllocationOpInterface.h.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/BufferizableOpInterfaceIncGen/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.cpp.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/BufferizableOpInterfaceIncGen/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/BufferizationBaseIncGen/mlir/Dialect/Bufferization/IR/BufferizationOpsDialect.cpp.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/BufferizationBaseIncGen/mlir/Dialect/Bufferization/IR/BufferizationOpsDialect.h.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/DerivedAttributeOpInterfaceIncGen/mlir/Interfaces/DerivedAttributeOpInterface.cpp.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/DerivedAttributeOpInterfaceIncGen/mlir/Interfaces/DerivedAttributeOpInterface.h.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/LLVMIntrinsicConversionIncGen/mlir/Dialect/LLVMIR/LLVMIntrinsicToLLVMIROpPairs.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/ParallelCombiningOpInterfaceIncGen/mlir/Interfaces/ParallelCombiningOpInterface.cpp.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/ParallelCombiningOpInterfaceIncGen/mlir/Interfaces/ParallelCombiningOpInterface.h.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/SparseTensorAttrDefsIncGen/mlir/Dialect/SparseTensor/IR/SparseTensorAttrDefs.cpp.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/SparseTensorAttrDefsIncGen/mlir/Dialect/SparseTensor/IR/SparseTensorAttrDefs.h.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/SparseTensorOpsIncGen/mlir/Dialect/SparseTensor/IR/SparseTensorOpsDialect.cpp.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/tensorflow/compiler/mlir/hlo/_virtual_includes/compose_set_interface_inc_gen/mlir-hlo/Dialect/gml_st/transforms/compose_set_interface.h.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/tensorflow/compiler/mlir/hlo/_virtual_includes/fusion_interface_inc_gen/mlir-hlo/Dialect/gml_st/transforms/fusion_interface.h.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/tensorflow/compiler/mlir/hlo/_virtual_includes/lhlo_gpu_ops_attrdefs_inc_gen/mlir-hlo/Dialect/lhlo_gpu/IR/lhlo_gpu_ops_attrdefs.h.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/tensorflow/compiler/mlir/hlo/_virtual_includes/lhlo_gpu_ops_dialect_inc_gen/mlir-hlo/Dialect/lhlo_gpu/IR/lhlo_gpu_ops_dialect.h.inc'
specified in the package manifest cannot be found.

The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/tensorflow/compiler/mlir/hlo/_virtual_includes/lhlo_gpu_ops_enums_inc_gen/mlir-hlo/Dialect/lhlo_gpu/IR/lhlo_gpu_ops_enums.h.inc'
specified in the package manifest cannot be found.

This transaction has incompatible packages due to a shared path.
  packages: conda-forge/win-64::libsqlite-3.43.2-hcfcfb64_0, defaults/win-64::sqlite-3.41.2-h2bbff1b_0
  path: 'library/bin/sqlite3.dll'

This transaction has incompatible packages due to a shared path.
  packages: conda-forge/win-64::libsqlite-3.43.2-hcfcfb64_0, defaults/win-64::sqlite-3.41.2-h2bbff1b_0
  path: 'library/include/sqlite3.h'

This transaction has incompatible packages due to a shared path.
  packages: conda-forge/win-64::libsqlite-3.43.2-hcfcfb64_0, defaults/win-64::sqlite-3.41.2-h2bbff1b_0
  path: 'library/include/sqlite3ext.h'

This transaction has incompatible packages due to a shared path.
  packages: conda-forge/win-64::libsqlite-3.43.2-hcfcfb64_0, defaults/win-64::sqlite-3.41.2-h2bbff1b_0
  path: 'library/lib/sqlite3.lib'

This transaction has incompatible packages due to a shared path.
  packages: defaults/win-64::notebook-6.5.4-py310haa95532_1, defaults/win-64::notebook-6.5.4-py310haa95532_1
  path: 'menu/notebook.json'

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/AffineMemoryOpInterfacesIncGen/mlir/Dialect/Affine/IR/AffineMemoryOpInterfaces.cpp.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/AffineMemoryOpInterfacesIncGen/mlir/Dialect/Affine/IR/AffineMemoryOpInterfaces.h.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/AllocationOpInterfaceIncGen/mlir/Dialect/Bufferization/IR/AllocationOpInterface.cpp.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/AllocationOpInterfaceIncGen/mlir/Dialect/Bufferization/IR/AllocationOpInterface.h.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/BufferizableOpInterfaceIncGen/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.cpp.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/BufferizableOpInterfaceIncGen/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/BufferizationBaseIncGen/mlir/Dialect/Bufferization/IR/BufferizationOpsDialect.cpp.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/BufferizationBaseIncGen/mlir/Dialect/Bufferization/IR/BufferizationOpsDialect.h.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/DerivedAttributeOpInterfaceIncGen/mlir/Interfaces/DerivedAttributeOpInterface.cpp.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/DerivedAttributeOpInterfaceIncGen/mlir/Interfaces/DerivedAttributeOpInterface.h.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/LLVMIntrinsicConversionIncGen/mlir/Dialect/LLVMIR/LLVMIntrinsicToLLVMIROpPairs.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/ParallelCombiningOpInterfaceIncGen/mlir/Interfaces/ParallelCombiningOpInterface.cpp.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/ParallelCombiningOpInterfaceIncGen/mlir/Interfaces/ParallelCombiningOpInterface.h.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/SparseTensorAttrDefsIncGen/mlir/Dialect/SparseTensor/IR/SparseTensorAttrDefs.cpp.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/SparseTensorAttrDefsIncGen/mlir/Dialect/SparseTensor/IR/SparseTensorAttrDefs.h.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/external/llvm-project/mlir/_virtual_includes/SparseTensorOpsIncGen/mlir/Dialect/SparseTensor/IR/SparseTensorOpsDialect.cpp.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/tensorflow/compiler/mlir/hlo/_virtual_includes/compose_set_interface_inc_gen/mlir-hlo/Dialect/gml_st/transforms/compose_set_interface.h.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/tensorflow/compiler/mlir/hlo/_virtual_includes/fusion_interface_inc_gen/mlir-hlo/Dialect/gml_st/transforms/fusion_interface.h.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/tensorflow/compiler/mlir/hlo/_virtual_includes/lhlo_gpu_ops_attrdefs_inc_gen/mlir-hlo/Dialect/lhlo_gpu/IR/lhlo_gpu_ops_attrdefs.h.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/tensorflow/compiler/mlir/hlo/_virtual_includes/lhlo_gpu_ops_dialect_inc_gen/mlir-hlo/Dialect/lhlo_gpu/IR/lhlo_gpu_ops_dialect.h.inc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for tensorflow-base located at C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\pkgs\tensorflow-base-2.10.0-mkl_py310h6a7f48e_0
appears to be corrupted. The path 'Lib/site-packages/tensorflow/include/tensorflow/compiler/mlir/hlo/_virtual_includes/lhlo_gpu_ops_enums_inc_gen/mlir-hlo/Dialect/lhlo_gpu/IR/lhlo_gpu_ops_enums.h.inc'
specified in the package manifest cannot be found.

ClobberError: This transaction has incompatible packages due to a shared path.
  packages: conda-forge/win-64::libsqlite-3.43.2-hcfcfb64_0, defaults/win-64::sqlite-3.41.2-h2bbff1b_0
  path: 'library/bin/sqlite3.dll'

ClobberError: This transaction has incompatible packages due to a shared path.
  packages: conda-forge/win-64::libsqlite-3.43.2-hcfcfb64_0, defaults/win-64::sqlite-3.41.2-h2bbff1b_0
  path: 'library/include/sqlite3.h'

ClobberError: This transaction has incompatible packages due to a shared path.
  packages: conda-forge/win-64::libsqlite-3.43.2-hcfcfb64_0, defaults/win-64::sqlite-3.41.2-h2bbff1b_0
  path: 'library/include/sqlite3ext.h'

ClobberError: This transaction has incompatible packages due to a shared path.
  packages: conda-forge/win-64::libsqlite-3.43.2-hcfcfb64_0, defaults/win-64::sqlite-3.41.2-h2bbff1b_0
  path: 'library/lib/sqlite3.lib'

ClobberError: This transaction has incompatible packages due to a shared path.
  packages: defaults/win-64::notebook-6.5.4-py310haa95532_1, defaults/win-64::notebook-6.5.4-py310haa95532_1
  path: 'menu/notebook.json'

This is similar to what I had encountered when I tried installing tensorflow from anaconda channel.

noah-chongsiriwatana commented 1 year ago

A similar issue is found here: https://github.com/conda/conda/issues/10767

pgunn commented 1 year ago

In that issue they tell you how to enable long path names on Windows; you'll want to do that. After that you can repeat the operation and it should work without problems.

noah-chongsiriwatana commented 1 year ago

Thank you, I solved the long file path issue. Changing the registry value and group policy didn't help, so I had to install the packages in a higher level folder as a workaround. Now I am getting an error at the executing transaction stage:

$ mamba create -n caiman -c defaults -c conda-forge caiman tensorflow=2.10.0
Transaction

  Prefix: C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\envs\caiman

  Updating specs:

   - caiman
   - tensorflow=2.10.0

  Package                           Version  Build                Channel          Size
-----------------------------------------------------------------------------------------
  Install:
-----------------------------------------------------------------------------------------

  + _tflow_select                     2.3.0  mkl                  pkgs/main         3kB
  + ucrt                       10.0.20348.0  haa95532_0           pkgs/main       543kB
  + blas                                1.0  mkl                  pkgs/main         6kB
  + icc_rt                         2022.1.0  h6049295_2           pkgs/main         7MB
  + ca-certificates              2023.08.22  haa95532_0           pkgs/main       126kB
  + winpty                            0.4.3  4                    pkgs/main       695kB
  + python_abi                         3.10  4_cp310              conda-forge       7kB
  + vc14_runtime                14.36.32532  hdcecf7f_17          conda-forge     739kB
  + vs2015_runtime              14.36.32532  h05e6639_17          conda-forge      17kB
...
  + tensorflow                       2.10.0  mkl_py310hd99672f_0  pkgs/main         4kB
  + notebook                          6.5.4  py310haa95532_1      pkgs/main       572kB
  + jupyterlab                        3.6.3  py310haa95532_0      pkgs/main         4MB
  + jupyter                           1.0.0  py310haa95532_8      pkgs/main         8kB
  + caiman                           1.9.16  py310h00ffb61_0      conda-forge     118MB

  Summary:

  Install: 317 packages

  Total download: 964MB

-----------------------------------------------------------------------------------------

Confirm changes: [Y/n]

Looking for: ['caiman', 'tensorflow=2.10.0']

Downloading and Extracting Packages: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
ERROR conda.core.link:_execute(945): An error occurred while installing package 'defaults::scikit-learn-1.3.0-py310h4ed8f06_0'.
Rolling back transaction: ...working... done
post-link script failed for package defaults::scikit-learn-1.3.0-py310h4ed8f06_0
location of failed script: C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\envs\caiman\Scripts\.scikit-learn-post-link.bat
==> script messages <==
<None>
==> script output <==
stdout:
stderr: 'conda' is not recognized as an internal or external command,
operable program or batch file.

return code: 255

()

LinkError: post-link script failed for package defaults::scikit-learn-1.3.0-py310h4ed8f06_0
location of failed script: C:\Users\nschong.AD\AppData\Local\miniforge-pypy3\envs\caiman\Scripts\.scikit-learn-post-link.bat
==> script messages <==
<None>
==> script output <==
stdout:
stderr: 'conda' is not recognized as an internal or external command,
operable program or batch file.

return code: 255

()
pgunn commented 1 year ago

Does miniforge somehow not give you a conda command?

noah-chongsiriwatana commented 1 year ago

Not sure, I'm going to try a fresh install of miniforge without pypy.

pgunn commented 1 year ago

You might just try miniconda instead.

noah-chongsiriwatana commented 1 year ago

Thank you, it seems like there are some issues with mamba. Switching to conda seems to have solved things. Thanks very much for the help.