search

zama-ai / concrete-ml

Concrete ML: Privacy Preserving ML framework using Fully Homomorphic Encryption (FHE), built on top of Concrete, with bindings to traditional ML frameworks.
Other
986 stars 143 forks source link

Unable to run example on MacOS #56

Closed rickymagner closed 1 year ago

rickymagner commented 1 year ago

Summary

I'm unable to compile a model in my MacOS environment.

Description

Steps to reproduce:

%0 = [[ -3] [-1 ... -1] [ 4]] # ClearTensor<int8, shape=(30, 1)> %1 = _input_0 # EncryptedTensor<uint8, shape=(1, 30)> %2 = -131 # ClearScalar %3 = add(%1, %2) # EncryptedTensor<int9, shape=(1, 30)> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ only up to 8-bit integers are supported %4 = subgraph(%3) # EncryptedTensor<int8, shape=(1, 30)> %5 = matmul(%4, %0) # EncryptedTensor<int16, shape=(1, 1)> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ only up to 8-bit integers are supported %6 = subgraph(%5) # EncryptedTensor<uint8, shape=(1, 1)> return %6

Subgraphs:

%4 = subgraph(%3):

     %0 = -128                           # ClearScalar<int8>
     %1 = 127                            # ClearScalar<uint7>
     %2 = 3                              # ClearScalar<uint2>
     %3 = 0.02919340787200249            # ClearScalar<float64>
     %4 = 0.02919340787200249            # ClearScalar<float64>
     %5 = input                          # EncryptedTensor<uint2, shape=(1, 30)>
     %6 = multiply(%4, %5)               # EncryptedTensor<float64, shape=(1, 30)>
     %7 = true_divide(%6, %3)            # EncryptedTensor<float64, shape=(1, 30)>
     %8 = add(%7, %2)                    # EncryptedTensor<float64, shape=(1, 30)>
     %9 = rint(%8)                       # EncryptedTensor<float64, shape=(1, 30)>
    %10 = clip(%9, %0, %1)               # EncryptedTensor<float64, shape=(1, 30)>
    %11 = astype(%10, dtype=int_)        # EncryptedTensor<uint1, shape=(1, 30)>
    return %11

%6 = subgraph(%5):

     %0 = 0                                 # ClearScalar<uint1>
     %1 = 255                               # ClearScalar<uint8>
     %2 = 0                                 # ClearScalar<uint1>
     %3 = 0.003919792729613519              # ClearScalar<float64>
     %4 = 1.0                               # ClearScalar<float64>
     %5 = 1.0                               # ClearScalar<float64>
     %6 = [0.11045581]                      # ClearTensor<float32, shape=(1,)>
     %7 = 0.0003101464749011019             # ClearScalar<float64>
     %8 = [[30]]                            # ClearTensor<uint5, shape=(1, 1)>
     %9 = 0                                 # ClearScalar<uint1>
    %10 = input                             # EncryptedTensor<uint5, shape=(1, 1)>
    %11 = astype(%10, dtype=float32)        # EncryptedTensor<float32, shape=(1, 1)>
    %12 = add(%11, %9)                      # EncryptedTensor<float32, shape=(1, 1)>
    %13 = add(%12, %8)                      # EncryptedTensor<float64, shape=(1, 1)>
    %14 = multiply(%7, %13)                 # EncryptedTensor<float64, shape=(1, 1)>
    %15 = add(%14, %6)                      # EncryptedTensor<float64, shape=(1, 1)>
    %16 = negative(%15)                     # EncryptedTensor<float64, shape=(1, 1)>
    %17 = exp(%16)                          # EncryptedTensor<float64, shape=(1, 1)>
    %18 = add(%5, %17)                      # EncryptedTensor<float64, shape=(1, 1)>
    %19 = true_divide(%4, %18)              # EncryptedTensor<float64, shape=(1, 1)>
    %20 = true_divide(%19, %3)              # EncryptedTensor<float64, shape=(1, 1)>
    %21 = add(%20, %2)                      # EncryptedTensor<float64, shape=(1, 1)>
    %22 = rint(%21)                         # EncryptedTensor<float64, shape=(1, 1)>
    %23 = clip(%22, %0, %1)                 # EncryptedTensor<float64, shape=(1, 1)>
    %24 = astype(%23, dtype=int_)           # EncryptedTensor<uint1, shape=(1, 1)>
    return %24

Note I've also tried running in JupyterLab and get the same error when hitting the `model.compile` step. Everything else up to that works as expected.

----

The documentation is a little conflicting. This [page](https://github.com/zama-ai/concrete-ml#installation) says this method should work just fine on MacOS, but this [page](https://github.com/zama-ai/concrete-ml/blob/release/0.6.x/docs/getting-started/pip_installing.md#installation) says `glibc` is required, which isn't supported on MacOS as far as I'm aware. So is anything further required to get this running on MacOS, or is something else misconfigured?

Here are some environment stats:

$ ~ gcc --version Apple clang version 14.0.0 (clang-1400.0.29.202) Target: x86_64-apple-darwin21.6.0 Thread model: posix InstalledDir: /Library/Developer/CommandLineTools/usr/bin

fd0r commented 1 year ago

Hello @rickymagner thanks for your interest in Concrete-ML! Can you show me what python -m pip show concrete-ml outputs on your machine? From the traceback you are providing I guess that you have an outdated version of Concrete-ML installed on your system.

rickymagner commented 1 year ago

Hi, here's the output:

Name: concrete-ml
Version: 0.3.0
Summary: Concrete-ML is an open-source set of tools which aims to simplify the use of fully homomorphic encryption (FHE) for data scientists.
Home-page: https://zama.ai/concrete-ml/
Author: Zama
Author-email: hello@zama.ai
License: BSD-3-Clause-Clear
Location: /Users/rmagner/Documents/Experiments/Concrete/venv/lib/python3.8/site-packages
Requires: concrete-numpy, hummingbird-ml, numpy, onnx, protobuf, scikit-learn, scipy, setuptools, skorch, torch, xgboost
Required-by:

Looks like you're right and I ended up with an older version somehow. I tried recreating my venv and reinstalling with

pip install -U pip wheel setuptools
pip install concrete-ml==0.6.1

but get this error:

Collecting concrete-ml==0.6.1
  Using cached concrete_ml-0.6.1-py3-none-any.whl (119 kB)
Collecting concrete-ml-extensions-brevitas==0.1.0
  Using cached concrete_ml_extensions_brevitas-0.1.0-py3-none-any.whl (414 kB)
Collecting typing-extensions==4.4.0
  Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting setuptools==65.6.3
  Using cached setuptools-65.6.3-py3-none-any.whl (1.2 MB)
Collecting concrete-numpy==0.9.0
  Using cached concrete_numpy-0.9.0-py3-none-any.whl (82 kB)
Collecting protobuf==3.19.4
  Using cached protobuf-3.19.4-cp38-cp38-macosx_10_9_x86_64.whl (961 kB)
Collecting xgboost==1.6.2
  Using cached xgboost-1.6.2-py3-none-macosx_10_15_x86_64.macosx_11_0_x86_64.macosx_12_0_x86_64.whl (1.7 MB)
Collecting numpy==1.21.6
  Using cached numpy-1.21.6-cp38-cp38-macosx_10_9_x86_64.whl (16.9 MB)
Collecting hummingbird-ml[onnx]==0.4.4
  Using cached hummingbird_ml-0.4.4-py2.py3-none-any.whl (181 kB)
Collecting onnx==1.12.0
  Using cached onnx-1.12.0-cp38-cp38-macosx_10_12_x86_64.whl (12.4 MB)
Collecting onnxoptimizer==0.2.7
  Using cached onnxoptimizer-0.2.7-cp38-cp38-macosx_10_9_x86_64.whl (441 kB)
Collecting skorch==0.11.0
  Using cached skorch-0.11.0-py3-none-any.whl (155 kB)
Collecting torch==1.12.1
  Using cached torch-1.12.1-cp38-none-macosx_10_9_x86_64.whl (137.8 MB)
Collecting scikit-learn==1.0.2
  Using cached scikit_learn-1.0.2-cp38-cp38-macosx_10_13_x86_64.whl (7.9 MB)
Collecting scipy==1.7.3
  Using cached scipy-1.7.3-cp38-cp38-macosx_10_9_x86_64.whl (33.0 MB)
Collecting future-annotations
  Using cached future_annotations-1.0.0-py2.py3-none-any.whl (5.6 kB)
Collecting packaging
  Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting dependencies==2.0.1
  Using cached dependencies-2.0.1-py2.py3-none-any.whl (15 kB)
INFO: pip is looking at multiple versions of concrete-ml-extensions-brevitas to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of concrete-ml to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement concrete-compiler<0.24.0,>=0.23.1 (from concrete-numpy) (from versions: 0.1.1, 0.1.2, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 0.5.0, 0.6.0, 0.7.0, 0.8.0, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.14.0, 0.15.0, 0.16.0)
ERROR: No matching distribution found for concrete-compiler<0.24.0,>=0.23.1

It seems there's a dependency mismatch between this package's compiler version and concrete-numpy's?

fd0r commented 1 year ago

Just to be sure, you are running on intel MacOS right?

bcm-at-zama commented 1 year ago

Hey, I'm on macOS intel and I just tried, and I can't repro your issue

        [REDACTED] % mkdir temp
        [REDACTED] % cd temp
        [REDACTED]/temp % python3 -m venv .venv
        [REDACTED]/temp % source .venv/bin/activate
(.venv) [REDACTED]/temp % pip install -U pip wheel setuptools
Looking in indexes: https://pypi.org/simple, [REDACTED]/simple
Requirement already satisfied: pip in ./.venv/lib/python3.8/site-packages (22.0.4)
Collecting pip
  Using cached pip-23.0-py3-none-any.whl (2.1 MB)
Collecting wheel
  Using cached wheel-0.38.4-py3-none-any.whl (36 kB)
Requirement already satisfied: setuptools in ./.venv/lib/python3.8/site-packages (56.0.0)
Collecting setuptools
  Using cached setuptools-67.0.0-py3-none-any.whl (1.1 MB)
Installing collected packages: wheel, setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 56.0.0
    Uninstalling setuptools-56.0.0:
      Successfully uninstalled setuptools-56.0.0
  Attempting uninstall: pip
    Found existing installation: pip 22.0.4
    Uninstalling pip-22.0.4:
      Successfully uninstalled pip-22.0.4
Successfully installed pip-23.0 setuptools-67.0.0 wheel-0.38.4
(.venv) [REDACTED]/temp % pip install concrete-ml==0.6.1
Looking in indexes: https://pypi.org/simple, [REDACTED]/simple
Collecting concrete-ml==0.6.1
  Using cached concrete_ml-0.6.1-py3-none-any.whl (119 kB)
Collecting hummingbird-ml[onnx]==0.4.4
  Using cached hummingbird_ml-0.4.4-py2.py3-none-any.whl (181 kB)
Collecting setuptools==65.6.3
  Using cached setuptools-65.6.3-py3-none-any.whl (1.2 MB)
Collecting onnxoptimizer==0.2.7
  Using cached onnxoptimizer-0.2.7-cp38-cp38-macosx_10_9_x86_64.whl (441 kB)
Collecting typing-extensions==4.4.0
  Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting concrete-numpy==0.9.0
  Using cached concrete_numpy-0.9.0-py3-none-any.whl (82 kB)
Collecting concrete-ml-extensions-brevitas==0.1.0
  Using cached concrete_ml_extensions_brevitas-0.1.0-py3-none-any.whl (414 kB)
Collecting scikit-learn==1.0.2
  Using cached scikit_learn-1.0.2-cp38-cp38-macosx_10_13_x86_64.whl (7.9 MB)
Collecting torch==1.12.1
  Using cached torch-1.12.1-cp38-none-macosx_10_9_x86_64.whl (137.8 MB)
Collecting numpy==1.21.6
  Using cached numpy-1.21.6-cp38-cp38-macosx_10_9_x86_64.whl (16.9 MB)
Collecting xgboost==1.6.2
  Using cached xgboost-1.6.2-py3-none-macosx_10_15_x86_64.macosx_11_0_x86_64.macosx_12_0_x86_64.whl (1.7 MB)
Collecting skorch==0.11.0
  Using cached skorch-0.11.0-py3-none-any.whl (155 kB)
Collecting scipy==1.7.3
  Using cached scipy-1.7.3-cp38-cp38-macosx_10_9_x86_64.whl (33.0 MB)
Collecting onnx==1.12.0
  Using cached onnx-1.12.0-cp38-cp38-macosx_10_12_x86_64.whl (12.4 MB)
Collecting protobuf==3.19.4
  Using cached protobuf-3.19.4-cp38-cp38-macosx_10_9_x86_64.whl (961 kB)
Collecting dependencies==2.0.1
  Using cached dependencies-2.0.1-py2.py3-none-any.whl (15 kB)
Collecting future-annotations
  Using cached future_annotations-1.0.0-py2.py3-none-any.whl (5.6 kB)
Collecting packaging
  Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting networkx<3.0.0,>=2.6.3
  Using cached networkx-2.8.8-py3-none-any.whl (2.0 MB)
Collecting Pillow<10.0.0,>=9.0.0
  Using cached Pillow-9.4.0-2-cp38-cp38-macosx_10_10_x86_64.whl (3.3 MB)
Collecting matplotlib<4.0.0,>=3.5.1
  Using cached matplotlib-3.6.3-cp38-cp38-macosx_10_12_x86_64.whl (7.3 MB)
Collecting concrete-compiler<0.24.0,>=0.23.1
  Using cached [REDACTED]/packages/concrete-compiler/0.23.4/concrete_compiler-0.23.4-cp38-cp38-macosx_11_0_x86_64.whl (44.5 MB)
Collecting dill
  Using cached dill-0.3.6-py3-none-any.whl (110 kB)
Collecting psutil
  Using cached psutil-5.9.4-cp36-abi3-macosx_10_9_x86_64.whl (243 kB)
Collecting onnxconverter-common>=1.6.0
  Using cached onnxconverter_common-1.13.0-py2.py3-none-any.whl (83 kB)
Collecting onnxruntime>=1.0.0
  Using cached onnxruntime-1.13.1-cp38-cp38-macosx_10_15_x86_64.whl (6.2 MB)
Collecting onnxmltools>=1.6.0
  Using cached onnxmltools-1.11.1-py3-none-any.whl (308 kB)
Collecting skl2onnx>=1.7.0
  Using cached skl2onnx-1.13-py2.py3-none-any.whl (288 kB)
Collecting joblib>=0.11
  Using cached joblib-1.2.0-py3-none-any.whl (297 kB)
Collecting threadpoolctl>=2.0.0
  Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting tqdm>=4.14.0
  Using cached tqdm-4.64.1-py2.py3-none-any.whl (78 kB)
Collecting tabulate>=0.7.7
  Using cached tabulate-0.9.0-py3-none-any.whl (35 kB)
Collecting PyYAML
  Using cached PyYAML-6.0-cp38-cp38-macosx_10_9_x86_64.whl (192 kB)
Collecting cycler>=0.10
  Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting kiwisolver>=1.0.1
  Using cached kiwisolver-1.4.4-cp38-cp38-macosx_10_9_x86_64.whl (65 kB)
Collecting pyparsing>=2.2.1
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting contourpy>=1.0.1
  Using cached contourpy-1.0.7-cp38-cp38-macosx_10_9_x86_64.whl (243 kB)
Collecting python-dateutil>=2.7
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting fonttools>=4.22.0
  Using cached fonttools-4.38.0-py3-none-any.whl (965 kB)
Collecting sympy
  Using cached sympy-1.11.1-py3-none-any.whl (6.5 MB)
Collecting flatbuffers
  Using cached flatbuffers-23.1.21-py2.py3-none-any.whl (26 kB)
Collecting coloredlogs
  Using cached coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)
Collecting tokenize-rt>=3
  Using cached tokenize_rt-5.0.0-py2.py3-none-any.whl (5.8 kB)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting humanfriendly>=9.1
  Using cached humanfriendly-10.0-py2.py3-none-any.whl (86 kB)
Collecting mpmath>=0.19
  Using cached mpmath-1.2.1-py3-none-any.whl (532 kB)
Installing collected packages: mpmath, flatbuffers, typing-extensions, tqdm, tokenize-rt, threadpoolctl, tabulate, sympy, six, setuptools, PyYAML, pyparsing, psutil, protobuf, Pillow, packaging, numpy, networkx, kiwisolver, joblib, humanfriendly, fonttools, dill, dependencies, cycler, torch, scipy, python-dateutil, onnx, future-annotations, contourpy, concrete-compiler, coloredlogs, xgboost, scikit-learn, onnxruntime, onnxoptimizer, onnxconverter-common, matplotlib, concrete-ml-extensions-brevitas, skorch, skl2onnx, hummingbird-ml, concrete-numpy, onnxmltools, concrete-ml
  Attempting uninstall: setuptools
    Found existing installation: setuptools 67.0.0
    Uninstalling setuptools-67.0.0:
      Successfully uninstalled setuptools-67.0.0
Successfully installed Pillow-9.4.0 PyYAML-6.0 coloredlogs-15.0.1 concrete-compiler-0.23.4 concrete-ml-0.6.1 concrete-ml-extensions-brevitas-0.1.0 concrete-numpy-0.9.0 contourpy-1.0.7 cycler-0.11.0 dependencies-2.0.1 dill-0.3.6 flatbuffers-23.1.21 fonttools-4.38.0 future-annotations-1.0.0 humanfriendly-10.0 hummingbird-ml-0.4.4 joblib-1.2.0 kiwisolver-1.4.4 matplotlib-3.6.3 mpmath-1.2.1 networkx-2.8.8 numpy-1.21.6 onnx-1.12.0 onnxconverter-common-1.13.0 onnxmltools-1.11.1 onnxoptimizer-0.2.7 onnxruntime-1.13.1 packaging-23.0 protobuf-3.19.4 psutil-5.9.4 pyparsing-3.0.9 python-dateutil-2.8.2 scikit-learn-1.0.2 scipy-1.7.3 setuptools-65.6.3 six-1.16.0 skl2onnx-1.13 skorch-0.11.0 sympy-1.11.1 tabulate-0.9.0 threadpoolctl-3.1.0 tokenize-rt-5.0.0 torch-1.12.1 tqdm-4.64.1 typing-extensions-4.4.0 xgboost-1.6.2
(.venv) [REDACTED]/temp %
bcm-at-zama commented 1 year ago

@fd0r is right: if ever you're on M1 or M2, it will not install.

rickymagner commented 1 year ago

Yes, my "About" page says: "2.4 GHz 8-Core Intel Core i9"

Could there be any issues with pip/python versions? I'm using Python 3.8.8 and pip 23.0.

bcm-at-zama commented 1 year ago

python 3.8 is normally ok, and I just tried with pip 23: it works as well. What a mystery

bcm-at-zama commented 1 year ago

Do you have a recent macOS? 

sw_vers

I have

ProductName: macOS ProductVersion: 12.6.2 BuildVersion: 21G320

which is not the last one

rickymagner commented 1 year ago

Huh, very strange. I have the same:

ProductName:    macOS
ProductVersion: 12.6.2
BuildVersion:   21G320

Did you have any of the dependencies manually installed before already, e.g. concrete-compiler, etc? I might try adding those first and see if it helps.

bcm-at-zama commented 1 year ago

I have the impression that your original problem is that

pip install concrete-compiler==0.23.4

fails for you, do you confirm? For me it works

rickymagner commented 1 year ago

Yes, that seems to be the problem:

$ ~ pip install concrete-compiler==0.23.4
ERROR: Could not find a version that satisfies the requirement concrete-compiler==0.23.4 (from versions: 0.1.1, 0.1.2, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 0.5.0, 0.6.0, 0.7.0, 0.8.0, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.14.0, 0.15.0, 0.16.0)
ERROR: No matching distribution found for concrete-compiler==0.23.4
rickymagner commented 1 year ago

It seems that https://github.com/zama-ai/concrete-compiler is either a private or no longer existing repo. Do you know where the code for that package is coming from? I see it on PyPI still here.

bcm-at-zama commented 1 year ago

concrete-compiler is indeed not open source (for now) but still, since you install it from pip, it is supposed to work

bcm-at-zama commented 1 year ago

Let me ask @umut-sahin , who is from compiler team, if he could help

bcm-at-zama commented 1 year ago

should you try python -m pip install concrete-compiler==0.23.4, by the way? are you sure that pip is the one you think? ie, does which pip show the one in your fresh venv?

rickymagner commented 1 year ago

Hi, I get the same error when running with python -m pip, and running which pip points to an alias to pip3, and which pip3 points to the one in my venv. I tried running pip3 install concrete-compiler==0.23.4 and get the same error as above.

youben11 commented 1 year ago

could you try a pip3.10 install concrete-compiler==0.23.4? or any other version you have installed, just make sure to use the complete version

bcm-at-zama commented 1 year ago

Just for the sake of clarity, could you run the following please:

python3 --version
python3 -m venv debug_issue_venv
source debug_issue_venv/bin/activate
which python
python --version
pip install -U pip wheel setuptools
pip install concrete-ml

and show us the log of that, please? This is so strange

bcm-at-zama commented 1 year ago

(by the way, sorry for the troubles, and if you urgently want to try something, it is possible to use Concrete-ML with docker as well; but, on macOS, it is slower)

rickymagner commented 1 year ago

I tried variations of pip3.10 install… but only have pip3.8.

Here are the outputs for the last block:

$ ~ python3 --version
Python 3.8.8
$ ~ python3 -m venv debug_issue_venv
$ ~ source debug_issue_venv/bin/activate
$ ~ which python
python: aliased to python3
$ ~ python --version
Python 3.8.8
$ ~ pip install -U pip wheel setuptools
Collecting pip
  Using cached pip-23.0-py3-none-any.whl (2.1 MB)
Collecting wheel
  Using cached wheel-0.38.4-py3-none-any.whl (36 kB)
Collecting setuptools
  Using cached setuptools-67.0.0-py3-none-any.whl (1.1 MB)
Installing collected packages: pip, wheel, setuptools
  Attempting uninstall: pip
    Found existing installation: pip 20.2.3
    Uninstalling pip-20.2.3:
      Successfully uninstalled pip-20.2.3
  Attempting uninstall: setuptools
    Found existing installation: setuptools 49.2.1
    Uninstalling setuptools-49.2.1:
      Successfully uninstalled setuptools-49.2.1
Successfully installed pip-23.0 setuptools-67.0.0 wheel-0.38.4
$ ~ pip install concrete-ml
Collecting concrete-ml
  Using cached concrete_ml-0.6.1-py3-none-any.whl (119 kB)
Collecting scipy==1.7.3
  Using cached scipy-1.7.3-cp38-cp38-macosx_10_9_x86_64.whl (33.0 MB)
Collecting typing-extensions==4.4.0
  Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting xgboost==1.6.2
  Using cached xgboost-1.6.2-py3-none-macosx_10_15_x86_64.macosx_11_0_x86_64.macosx_12_0_x86_64.whl (1.7 MB)
Collecting numpy==1.21.6
  Using cached numpy-1.21.6-cp38-cp38-macosx_10_9_x86_64.whl (16.9 MB)
Collecting skorch==0.11.0
  Using cached skorch-0.11.0-py3-none-any.whl (155 kB)
Collecting hummingbird-ml[onnx]==0.4.4
  Using cached hummingbird_ml-0.4.4-py2.py3-none-any.whl (181 kB)
Collecting onnxoptimizer==0.2.7
  Using cached onnxoptimizer-0.2.7-cp38-cp38-macosx_10_9_x86_64.whl (441 kB)
Collecting torch==1.12.1
  Using cached torch-1.12.1-cp38-none-macosx_10_9_x86_64.whl (137.8 MB)
Collecting setuptools==65.6.3
  Using cached setuptools-65.6.3-py3-none-any.whl (1.2 MB)
Collecting concrete-ml-extensions-brevitas==0.1.0
  Using cached concrete_ml_extensions_brevitas-0.1.0-py3-none-any.whl (414 kB)
Collecting onnx==1.12.0
  Using cached onnx-1.12.0-cp38-cp38-macosx_10_12_x86_64.whl (12.4 MB)
Collecting concrete-numpy==0.9.0
  Using cached concrete_numpy-0.9.0-py3-none-any.whl (82 kB)
Collecting scikit-learn==1.0.2
  Using cached scikit_learn-1.0.2-cp38-cp38-macosx_10_13_x86_64.whl (7.9 MB)
Collecting protobuf==3.19.4
  Using cached protobuf-3.19.4-cp38-cp38-macosx_10_9_x86_64.whl (961 kB)
Collecting future-annotations
  Using cached future_annotations-1.0.0-py2.py3-none-any.whl (5.6 kB)
Collecting dependencies==2.0.1
  Using cached dependencies-2.0.1-py2.py3-none-any.whl (15 kB)
Collecting packaging
  Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting matplotlib<4.0.0,>=3.5.1
  Using cached matplotlib-3.6.3-cp38-cp38-macosx_10_12_x86_64.whl (7.3 MB)
Collecting Pillow<10.0.0,>=9.0.0
  Using cached Pillow-9.4.0-2-cp38-cp38-macosx_10_10_x86_64.whl (3.3 MB)
INFO: pip is looking at multiple versions of concrete-ml-extensions-brevitas to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of concrete-ml to determine which version is compatible with other requirements. This could take a while.
Collecting concrete-ml
  Using cached concrete_ml-0.6.0-py3-none-any.whl (119 kB)
Collecting scikit-learn>=1.0.2
  Using cached scikit_learn-1.2.1-cp38-cp38-macosx_10_9_x86_64.whl (9.0 MB)
Collecting hummingbird-ml[onnx]<0.5.0,>=0.4.4
  Using cached hummingbird_ml-0.4.7-py2.py3-none-any.whl (162 kB)
Collecting concrete-numpy>=0.8.0
  Using cached concrete_numpy-0.10.0-py3-none-any.whl (82 kB)
Collecting xgboost<2.0.0,>=1.5.2
  Using cached xgboost-1.7.3-py3-none-macosx_10_15_x86_64.macosx_11_0_x86_64.macosx_12_0_x86_64.whl (1.8 MB)
Collecting scipy>=1.7.3
  Using cached scipy-1.10.0-cp38-cp38-macosx_10_15_x86_64.whl (35.0 MB)
Collecting numpy>=1.21.6
  Using cached numpy-1.24.1-cp38-cp38-macosx_10_9_x86_64.whl (19.8 MB)
Collecting onnx<2.0.0,>=1.11.0
  Using cached onnx-1.13.0-cp38-cp38-macosx_10_12_x86_64.whl (12.7 MB)
Requirement already satisfied: setuptools in ./debug_issue_venv/lib/python3.8/site-packages (from concrete-ml) (67.0.0)
Collecting concrete-numpy>=0.8.0
  Using cached concrete_numpy-0.9.1-py3-none-any.whl (82 kB)
  Using cached concrete_numpy-0.8.0-py3-none-any.whl (68 kB)
INFO: pip is looking at multiple versions of dependencies to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of protobuf to determine which version is compatible with other requirements. This could take a while.
Collecting concrete-ml
  Using cached concrete_ml-0.5.1-py3-none-any.whl (103 kB)
Collecting concrete-ml-extensions-brevitas==0.0.0
  Using cached concrete_ml_extensions_brevitas-0.0.0-py3-none-any.whl (403 kB)
Collecting concrete-ml
  Using cached concrete_ml-0.5.0-py3-none-any.whl (102 kB)
Collecting torch<2.0.0,>=1.11.0
  Using cached torch-1.13.1-cp38-none-macosx_10_9_x86_64.whl (135.4 MB)
Collecting concrete-ml
  Using cached concrete_ml-0.4.0-py3-none-any.whl (97 kB)
  Using cached concrete_ml-0.3.0-py3-none-any.whl (84 kB)
Collecting concrete-numpy==0.7.0
  Using cached concrete_numpy-0.7.0-py3-none-any.whl (65 kB)
Collecting concrete-compiler<0.11.0,>=0.10.0
  Using cached concrete_compiler-0.10.0-cp38-cp38-macosx_10_15_x86_64.whl (41.0 MB)
Collecting networkx<3.0.0,>=2.6.3
  Using cached networkx-2.8.8-py3-none-any.whl (2.0 MB)
Collecting PyYAML
  Using cached PyYAML-6.0-cp38-cp38-macosx_10_9_x86_64.whl (192 kB)
Collecting dill
  Using cached dill-0.3.6-py3-none-any.whl (110 kB)
Collecting onnxconverter-common>=1.6.0
  Using cached onnxconverter_common-1.13.0-py2.py3-none-any.whl (83 kB)
Collecting hummingbird-ml[onnx]<0.5.0,>=0.4.4
  Using cached hummingbird_ml-0.4.6-py2.py3-none-any.whl (162 kB)
  Using cached hummingbird_ml-0.4.5-py2.py3-none-any.whl (162 kB)
Collecting psutil
  Using cached psutil-5.9.4-cp36-abi3-macosx_10_9_x86_64.whl (243 kB)
Collecting onnxruntime>=1.0.0
  Using cached onnxruntime-1.13.1-cp38-cp38-macosx_10_15_x86_64.whl (6.2 MB)
Collecting onnxmltools>=1.6.0
  Using cached onnxmltools-1.11.1-py3-none-any.whl (308 kB)
Collecting skl2onnx>=1.7.0
  Using cached skl2onnx-1.13-py2.py3-none-any.whl (288 kB)
Collecting threadpoolctl>=2.0.0
  Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting joblib>=1.1.1
  Using cached joblib-1.2.0-py3-none-any.whl (297 kB)
Collecting tqdm>=4.14.0
  Using cached tqdm-4.64.1-py2.py3-none-any.whl (78 kB)
Collecting tabulate>=0.7.7
  Using cached tabulate-0.9.0-py3-none-any.whl (35 kB)
Collecting python-dateutil>=2.7
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting cycler>=0.10
  Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting contourpy>=1.0.1
  Using cached contourpy-1.0.7-cp38-cp38-macosx_10_9_x86_64.whl (243 kB)
Collecting fonttools>=4.22.0
  Using cached fonttools-4.38.0-py3-none-any.whl (965 kB)
Collecting pyparsing>=2.2.1
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting kiwisolver>=1.0.1
  Using cached kiwisolver-1.4.4-cp38-cp38-macosx_10_9_x86_64.whl (65 kB)
Collecting coloredlogs
  Using cached coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)
Collecting sympy
  Using cached sympy-1.11.1-py3-none-any.whl (6.5 MB)
Collecting flatbuffers
  Using cached flatbuffers-23.1.21-py2.py3-none-any.whl (26 kB)
Collecting scikit-learn<2.0.0,>=1.1.0
  Using cached scikit_learn-1.1.1-cp38-cp38-macosx_10_13_x86_64.whl (8.5 MB)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting humanfriendly>=9.1
  Using cached humanfriendly-10.0-py2.py3-none-any.whl (86 kB)
Collecting mpmath>=0.19
  Using cached mpmath-1.2.1-py3-none-any.whl (532 kB)
Installing collected packages: mpmath, flatbuffers, typing-extensions, tqdm, threadpoolctl, tabulate, sympy, six, PyYAML, pyparsing, psutil, protobuf, Pillow, packaging, numpy, networkx, kiwisolver, joblib, humanfriendly, fonttools, dill, cycler, torch, scipy, python-dateutil, onnx, contourpy, concrete-compiler, coloredlogs, xgboost, scikit-learn, onnxruntime, onnxconverter-common, matplotlib, skorch, skl2onnx, hummingbird-ml, concrete-numpy, onnxmltools, concrete-ml
Successfully installed Pillow-9.4.0 PyYAML-6.0 coloredlogs-15.0.1 concrete-compiler-0.10.0 concrete-ml-0.3.0 concrete-numpy-0.7.0 contourpy-1.0.7 cycler-0.11.0 dill-0.3.6 flatbuffers-23.1.21 fonttools-4.38.0 humanfriendly-10.0 hummingbird-ml-0.4.4 joblib-1.2.0 kiwisolver-1.4.4 matplotlib-3.6.3 mpmath-1.2.1 networkx-2.8.8 numpy-1.24.1 onnx-1.12.0 onnxconverter-common-1.13.0 onnxmltools-1.11.1 onnxruntime-1.13.1 packaging-23.0 protobuf-3.19.4 psutil-5.9.4 pyparsing-3.0.9 python-dateutil-2.8.2 scikit-learn-1.1.1 scipy-1.10.0 six-1.16.0 skl2onnx-1.13 skorch-0.11.0 sympy-1.11.1 tabulate-0.9.0 threadpoolctl-3.1.0 torch-1.13.1 tqdm-4.64.1 typing-extensions-4.4.0 xgboost-1.7.3

Install is successful, but then running python example.py with the provided example produces the original error reported in this issue. It still installs the old version:

$ ~ python -m pip show concrete-ml
Name: concrete-ml
Version: 0.3.0
Summary: Concrete-ML is an open-source set of tools which aims to simplify the use of fully homomorphic encryption (FHE) for data scientists.
Home-page: https://zama.ai/concrete-ml/
Author: Zama
Author-email: hello@zama.ai
License: BSD-3-Clause-Clear
Location: /Users/rmagner/Documents/Experiments/Concrete/debug_issue_venv/lib/python3.8/site-packages
Requires: concrete-numpy, hummingbird-ml, numpy, onnx, protobuf, scikit-learn, scipy, setuptools, skorch, torch, xgboost
Required-by:

It's definitely a strange issue, but there's no urgency. I just wanted to test out the package a little bit. Thanks!

fd0r commented 1 year ago

Could you maybe try by explicitly specifying the version? pip install concrete-ml==0.6.1

bcm-at-zama commented 1 year ago

So yes, a strange issue. What you should do for now is to use docker, to test out the package. It is easy to install (https://docs.zama.ai/concrete-ml/getting-started/pip_installing#using-docker), but at the same time, it is normally also easy to install with pip and here we have a problem. There is something in your config that is unclear and prevents to use the last compiler/Concrete-Numpy/Concrete-ML, we're investigating

bcm-at-zama commented 1 year ago

One thing you may try @rickymagner is:

I can imagine it will not work because this wheel was filtered out for your config, but we may hope that the error you get may tell us why at the beginning, 0.6.1 was filtered out. And then, we might be able to solve it

bcm-at-zama commented 1 year ago

You can also try

pip install -U --use-deprecated=legacy-resolver concrete-ml==0.6.1

We have seen issues like https://stackoverflow.com/questions/65122957/resolving-new-pip-backtracking-runtime-issue/65793484#65793484 which may be related to your problem

rickymagner commented 1 year ago

Hi, thanks for the suggestions. Unfortunately they haven't worked. Here's the outputs:

$ ~ pip install concrete_ml-0.6.1-py3-none-any.whl
Processing ./concrete_ml-0.6.1-py3-none-any.whl
Collecting scipy==1.7.3
  Using cached scipy-1.7.3-cp38-cp38-macosx_10_9_x86_64.whl (33.0 MB)
Collecting skorch==0.11.0
  Using cached skorch-0.11.0-py3-none-any.whl (155 kB)
Collecting concrete-numpy==0.9.0
  Using cached concrete_numpy-0.9.0-py3-none-any.whl (82 kB)
Collecting numpy==1.21.6
  Using cached numpy-1.21.6-cp38-cp38-macosx_10_9_x86_64.whl (16.9 MB)
Collecting hummingbird-ml[onnx]==0.4.4
  Using cached hummingbird_ml-0.4.4-py2.py3-none-any.whl (181 kB)
Collecting torch==1.12.1
  Using cached torch-1.12.1-cp38-none-macosx_10_9_x86_64.whl (137.8 MB)
Collecting concrete-ml-extensions-brevitas==0.1.0
  Using cached concrete_ml_extensions_brevitas-0.1.0-py3-none-any.whl (414 kB)
Collecting onnxoptimizer==0.2.7
  Using cached onnxoptimizer-0.2.7-cp38-cp38-macosx_10_9_x86_64.whl (441 kB)
Collecting onnx==1.12.0
  Using cached onnx-1.12.0-cp38-cp38-macosx_10_12_x86_64.whl (12.4 MB)
Collecting setuptools==65.6.3
  Using cached setuptools-65.6.3-py3-none-any.whl (1.2 MB)
Collecting scikit-learn==1.0.2
  Using cached scikit_learn-1.0.2-cp38-cp38-macosx_10_13_x86_64.whl (7.9 MB)
Collecting typing-extensions==4.4.0
  Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting xgboost==1.6.2
  Using cached xgboost-1.6.2-py3-none-macosx_10_15_x86_64.macosx_11_0_x86_64.macosx_12_0_x86_64.whl (1.7 MB)
Collecting protobuf==3.19.4
  Using cached protobuf-3.19.4-cp38-cp38-macosx_10_9_x86_64.whl (961 kB)
Collecting future-annotations
  Using cached future_annotations-1.0.0-py2.py3-none-any.whl (5.6 kB)
Collecting dependencies==2.0.1
  Using cached dependencies-2.0.1-py2.py3-none-any.whl (15 kB)
Collecting packaging
  Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting matplotlib<4.0.0,>=3.5.1
  Using cached matplotlib-3.6.3-cp38-cp38-macosx_10_12_x86_64.whl (7.3 MB)
Collecting Pillow<10.0.0,>=9.0.0
  Using cached Pillow-9.4.0-2-cp38-cp38-macosx_10_10_x86_64.whl (3.3 MB)
Collecting networkx<3.0.0,>=2.6.3
  Using cached networkx-2.8.8-py3-none-any.whl (2.0 MB)
INFO: pip is looking at multiple versions of concrete-ml-extensions-brevitas to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of concrete-ml to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement concrete-compiler<0.24.0,>=0.23.1 (from concrete-numpy) (from versions: 0.1.1, 0.1.2, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 0.5.0, 0.6.0, 0.7.0, 0.8.0, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.14.0, 0.15.0, 0.16.0)
ERROR: No matching distribution found for concrete-compiler<0.24.0,>=0.23.1

and

 pip install -U --use-deprecated=legacy-resolver concrete-ml==0.6.1
Collecting concrete-ml==0.6.1
  Using cached concrete_ml-0.6.1-py3-none-any.whl (119 kB)
Collecting scikit-learn==1.0.2
  Using cached scikit_learn-1.0.2-cp38-cp38-macosx_10_13_x86_64.whl (7.9 MB)
Collecting typing-extensions==4.4.0
  Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting xgboost==1.6.2
  Using cached xgboost-1.6.2-py3-none-macosx_10_15_x86_64.macosx_11_0_x86_64.macosx_12_0_x86_64.whl (1.7 MB)
Collecting hummingbird-ml[onnx]==0.4.4
  Using cached hummingbird_ml-0.4.4-py2.py3-none-any.whl (181 kB)
Collecting torch==1.12.1
  Using cached torch-1.12.1-cp38-none-macosx_10_9_x86_64.whl (137.8 MB)
Collecting protobuf==3.19.4
  Using cached protobuf-3.19.4-cp38-cp38-macosx_10_9_x86_64.whl (961 kB)
Collecting onnxoptimizer==0.2.7
  Using cached onnxoptimizer-0.2.7-cp38-cp38-macosx_10_9_x86_64.whl (441 kB)
Collecting setuptools==65.6.3
  Using cached setuptools-65.6.3-py3-none-any.whl (1.2 MB)
Collecting concrete-ml-extensions-brevitas==0.1.0
  Using cached concrete_ml_extensions_brevitas-0.1.0-py3-none-any.whl (414 kB)
Collecting skorch==0.11.0
  Using cached skorch-0.11.0-py3-none-any.whl (155 kB)
Collecting scipy==1.7.3
  Using cached scipy-1.7.3-cp38-cp38-macosx_10_9_x86_64.whl (33.0 MB)
Collecting concrete-numpy==0.9.0
  Using cached concrete_numpy-0.9.0-py3-none-any.whl (82 kB)
Collecting numpy==1.21.6
  Using cached numpy-1.21.6-cp38-cp38-macosx_10_9_x86_64.whl (16.9 MB)
Collecting onnx==1.12.0
  Using cached onnx-1.12.0-cp38-cp38-macosx_10_12_x86_64.whl (12.4 MB)
Collecting joblib>=0.11
  Using cached joblib-1.2.0-py3-none-any.whl (297 kB)
Collecting threadpoolctl>=2.0.0
  Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting psutil
  Using cached psutil-5.9.4-cp36-abi3-macosx_10_9_x86_64.whl (243 kB)
Collecting onnxconverter-common>=1.6.0
  Using cached onnxconverter_common-1.13.0-py2.py3-none-any.whl (83 kB)
Collecting dill
  Using cached dill-0.3.6-py3-none-any.whl (110 kB)
Collecting onnxruntime>=1.0.0; extra == "onnx"
  Using cached onnxruntime-1.13.1-cp38-cp38-macosx_10_15_x86_64.whl (6.2 MB)
Collecting onnxmltools>=1.6.0; extra == "onnx"
  Using cached onnxmltools-1.11.1-py3-none-any.whl (308 kB)
Collecting skl2onnx>=1.7.0; extra == "onnx"
  Using cached skl2onnx-1.13-py2.py3-none-any.whl (288 kB)
Collecting future-annotations
  Using cached future_annotations-1.0.0-py2.py3-none-any.whl (5.6 kB)
Collecting packaging
  Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting dependencies==2.0.1
  Using cached dependencies-2.0.1-py2.py3-none-any.whl (15 kB)
Collecting tabulate>=0.7.7
  Using cached tabulate-0.9.0-py3-none-any.whl (35 kB)
Collecting tqdm>=4.14.0
  Using cached tqdm-4.64.1-py2.py3-none-any.whl (78 kB)
Collecting Pillow<10.0.0,>=9.0.0
  Using cached Pillow-9.4.0-2-cp38-cp38-macosx_10_10_x86_64.whl (3.3 MB)
ERROR: Could not find a version that satisfies the requirement concrete-compiler<0.24.0,>=0.23.1 (from concrete-numpy==0.9.0->concrete-ml==0.6.1) (from versions: 0.1.1, 0.1.2, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 0.5.0, 0.6.0, 0.7.0, 0.8.0, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.14.0, 0.15.0, 0.16.0)
ERROR: No matching distribution found for concrete-compiler<0.24.0,>=0.23.1 (from concrete-numpy==0.9.0->concrete-ml==0.6.1)

If it's working for others with a similar environment, there must be something else messed up in my environment. I can try using the docker solution or maybe a cloud VM to run it.

On a separate note, concrete-numpy installs and seems to work fine. Do you know if the Compiler class defined in that package is similar to the one found in the ML library? For example, if I wanted to try implementing some ML algorithms using just concrete-numpy, should they work essentially the same? That seemed to be where the dependency resolver was getting stuck for the ML version for me.

bcm-at-zama commented 1 year ago

I would go for the docker solution, really.

Yes you can use Concrete-Numpy directly but it is better / easier to use Concrete-ML, if you want to enjoy what we've been working for a bit more than a year.

And warning to use the last Concrete-Numpy: don't be stuck on old versions of Concrete-Numpy if this is the way you want to take, because of your issues in installation. As it looks your issue is at the Concrete-compiler level, I would say, you're also blocked to an old version of Concrete-Numpy, which is bad for you.

rickymagner commented 1 year ago

Thanks, I did just check that an older version of concrete-numpy was installed by default (0.7.0) for the same reason. I will try using the ML package in a different environment I can get it working in. Thanks for all of your help!

bcm-at-zama commented 1 year ago

Our pleasure and sorry for the troubles. Still not sure if it is our fault or not, sorry for that.

bcm-at-zama commented 1 year ago

If you're still interested in debugging, having logs of

pip install -v concrete-ml==0.6.1
pip install -vv concrete-ml==0.6.1
pip install -vvv concrete-ml==0.6.1

and the same for concrete-compiler==0.23.4 might help.

rickymagner commented 1 year ago

Here are the logs for those, in case you find it helpful:

$ ~ pip install -v concrete-ml==0.6.1
Using pip 23.0 from /Users/rmagner/Documents/Experiments/Concrete/venv/lib/python3.8/site-packages/pip (python 3.8)
Collecting concrete-ml==0.6.1
  Using cached concrete_ml-0.6.1-py3-none-any.whl (119 kB)
Collecting hummingbird-ml[onnx]==0.4.4
  Using cached hummingbird_ml-0.4.4-py2.py3-none-any.whl (181 kB)
Collecting scikit-learn==1.0.2
  Using cached scikit_learn-1.0.2-cp38-cp38-macosx_10_13_x86_64.whl (7.9 MB)
Collecting concrete-ml-extensions-brevitas==0.1.0
  Using cached concrete_ml_extensions_brevitas-0.1.0-py3-none-any.whl (414 kB)
Collecting xgboost==1.6.2
  Using cached xgboost-1.6.2-py3-none-macosx_10_15_x86_64.macosx_11_0_x86_64.macosx_12_0_x86_64.whl (1.7 MB)
Collecting numpy==1.21.6
  Using cached numpy-1.21.6-cp38-cp38-macosx_10_9_x86_64.whl (16.9 MB)
Collecting skorch==0.11.0
  Using cached skorch-0.11.0-py3-none-any.whl (155 kB)
Collecting onnx==1.12.0
  Using cached onnx-1.12.0-cp38-cp38-macosx_10_12_x86_64.whl (12.4 MB)
Collecting typing-extensions==4.4.0
  Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting setuptools==65.6.3
  Using cached setuptools-65.6.3-py3-none-any.whl (1.2 MB)
Collecting concrete-numpy==0.9.0
  Using cached concrete_numpy-0.9.0-py3-none-any.whl (82 kB)
Collecting protobuf==3.19.4
  Using cached protobuf-3.19.4-cp38-cp38-macosx_10_9_x86_64.whl (961 kB)
Collecting onnxoptimizer==0.2.7
  Using cached onnxoptimizer-0.2.7-cp38-cp38-macosx_10_9_x86_64.whl (441 kB)
Collecting torch==1.12.1
  Using cached torch-1.12.1-cp38-none-macosx_10_9_x86_64.whl (137.8 MB)
Collecting scipy==1.7.3
  Using cached scipy-1.7.3-cp38-cp38-macosx_10_9_x86_64.whl (33.0 MB)
Collecting dependencies==2.0.1
  Using cached dependencies-2.0.1-py2.py3-none-any.whl (15 kB)
Collecting packaging
  Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting future-annotations
  Using cached future_annotations-1.0.0-py2.py3-none-any.whl (5.6 kB)
Collecting networkx<3.0.0,>=2.6.3
  Using cached networkx-2.8.8-py3-none-any.whl (2.0 MB)
Collecting Pillow<10.0.0,>=9.0.0
  Using cached Pillow-9.4.0-2-cp38-cp38-macosx_10_10_x86_64.whl (3.3 MB)
INFO: pip is looking at multiple versions of concrete-ml-extensions-brevitas to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of concrete-ml to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement concrete-compiler<0.24.0,>=0.23.1 (from concrete-numpy) (from versions: 0.1.1, 0.1.2, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 0.5.0, 0.6.0, 0.7.0, 0.8.0, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.14.0, 0.15.0, 0.16.0)
ERROR: No matching distribution found for concrete-compiler<0.24.0,>=0.23.1

and the others are larger so I attached them here as txt files. vv.log vvv.log

rickymagner commented 1 year ago

Oops, I forgot to attach the same for concrete-compiler==0.23.1. v-cc.log vv-cc.log vvv-cc.log

bcm-at-zama commented 1 year ago

First time I have a look at that kind of file, but let me try!

I extract 0.23.4 things:

Skipping link: none of the wheel's tags (cp310-cp310-macosx_11_0_x86_64) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/f8/0b/58ee88e2abdf48bfec75f05f5d81af793f02ea90e4a1dececf26a4c3bad4/concrete_compiler-0.23.4-cp310-cp310-macosx_11_0_x86_64.whl (from https://pypi.org/simple/concrete-compiler/)

Skipping link: none of the wheel's tags (cp310-cp310-manylinux_2_28_x86_64) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/de/68/702b8c831a48ccc592c936fe3bfea2e483916496f7e0f445307db4a60f9b/concrete_compiler-0.23.4-cp310-cp310-manylinux_2_28_x86_64.whl (from https://pypi.org/simple/concrete-compiler/)

Skipping link: none of the wheel's tags (cp37-cp37m-manylinux_2_28_x86_64) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/16/74/4333e563e7e551998e95378d7182da50a1758caee011098126e6e45d2a83/concrete_compiler-0.23.4-cp37-cp37m-manylinux_2_28_x86_64.whl (from https://pypi.org/simple/concrete-compiler/)

Skipping link: none of the wheel's tags (cp38-cp38-macosx_11_0_x86_64) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/1d/72/aaac30031931611bd9ce6e4208ef9f4a4d7d2d0b2b33131f415e19a7514f/concrete_compiler-0.23.4-cp38-cp38-macosx_11_0_x86_64.whl (from https://pypi.org/simple/concrete-compiler/)

Skipping link: none of the wheel's tags (cp38-cp38-manylinux_2_28_x86_64) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/f2/56/89806c1b2b0b7bda702af2584232e4fa5b84c7a35f8905db481234142491/concrete_compiler-0.23.4-cp38-cp38-manylinux_2_28_x86_64.whl (from https://pypi.org/simple/concrete-compiler/)

Skipping link: none of the wheel's tags (cp39-cp39-macosx_11_0_x86_64) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/e3/6f/153b47fd13545b6be5cbe3850953572965c190f93d96ead16879cc590076/concrete_compiler-0.23.4-cp39-cp39-macosx_11_0_x86_64.whl (from https://pypi.org/simple/concrete-compiler/)

Skipping link: none of the wheel's tags (cp39-cp39-manylinux_2_28_x86_64) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/5a/89/1f2bde24d7fb2fd384e6e478e70a64f00b3ece8a6355b39ef34dea195b0a/concrete_compiler-0.23.4-cp39-cp39-manylinux_2_28_x86_64.whl (from https://pypi.org/simple/concrete-compiler/)

Three of them are for macOS: cp38-cp38-macosx_11_0_x86_64, cp39-cp39-macosx_11_0_x86_64, cp310-cp310-macosx_11_0_x86_64

Then I do pip debug --verbose: I can check that there are compatible things

% pip debug --verbose | grep "cp38-cp38-macosx_11_0_x86_64\|cp39-cp39-macosx_11_0_x86_64\|cp310-cp310-macosx_11_0_x86_64"
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
  cp38-cp38-macosx_11_0_x86_64

So, now, @rickymagner, could you try:

rickymagner commented 1 year ago

Hi, yes here are the outputs of the commands:

$ ~ pip debug --verbose | grep "cp38-cp38-macosx_11_0_x86_64\|cp39-cp39-macosx_11_0_x86_64\|cp310-cp310-macosx_11_0_x86_64"
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.

This has exit code 1.

The next command pip debug --verbose is attached. And the final two are the same as above: exit code 1 with the same warning about debugging.

Thanks!

pip_debug_verbose.txt

bcm-at-zama commented 1 year ago

Hum, I don't see any macosx_11_0 in your pip_debug_verbose.txt, which is surprising. You have a 12.6.2 (from your sw_vers) but your python config seems very strange.

And why do I see /Users/rmagner/opt/anaconda3/lib/python3.8/site-packages/pip? Aren't you in your venv?

For me: which pip says [REDACTED]/concrete-ml-internal/.venv/bin/pip, for example

rickymagner commented 1 year ago

Hi, sorry for the late reply. Yes, I am in the venv, so it's strange that which pip3 is pointing to the conda version. Perhaps there's something with conda polluting the venv? I haven't used conda very much, so perhaps I can try removing it and retrying? Though I'm not sure if that would break anything else…

bcm-at-zama commented 1 year ago

Maybe you should have a look to your PATH. Maybe conda path is put in the first place? Hard to debug that remotely, it looks very related to your configuration

bcm-at-zama commented 1 year ago

Or, instead of pip, try to do python -m pip, which may prevent your system to use conda

rickymagner commented 1 year ago

I did end up removing conda from my system and checking it was removed from my $PATH successfully. Now it looks like which pip3 points to the venv location: 

$ ~ which pip3
/Users/rmagner/Documents/Experiments/Concrete/venv/bin/pip3

This seems like progress, but unfortunately following the guide to install e.g. wheel, etc. and then concrete-ml==0.6.1 results in the same concrete-compiler issues as above. Now my environment seems to be in python3.9.6. Could this be an issue?

Also, thanks for your help so far, but it's possible we could close the issue if you'd like since it seems like it's probably something weird with my environment rather than a problem with the package itself.

bcm-at-zama commented 1 year ago

python3.7 to python3.9 are supported, so now, it should be fine.

bcm-at-zama commented 1 year ago

OK yes, we'll close the issue as you proposed, since something seems quite wrong in your system. Sorry that it's too hard to debug that remotely.