search

SAME-Project / same-project

https://sameproject.ml/
Apache License 2.0
19 stars 8 forks source link

Need to figure out something in re: grpc having a dependency on cygrpc #120

Open aronchick opened 2 years ago

aronchick commented 2 years ago

Repro steps:

Output:

❯ python3 sameproject/main.py program run -f test/testdata/minimal_notebook/same.yaml -t kubeflow
Traceback (most recent call last):
  File "/Users/daaronch/code/same-project/sameproject/main.py", line 1, in <module>
    from cli import run, version, init, verify
  File "/Users/daaronch/code/same-project/sameproject/cli/__init__.py", line 4, in <module>
    from .run import run
  File "/Users/daaronch/code/same-project/sameproject/cli/run.py", line 1, in <module>
    from sameproject.ops import notebooks as nbproc
  File "/Users/daaronch/code/same-project/sameproject/ops/notebooks.py", line 8, in <module>
    import sameproject.ops.backends
  File "/Users/daaronch/code/same-project/sameproject/ops/backends.py", line 4, in <module>
    import sameproject.ops.kubeflow as kubeflow
  File "/Users/daaronch/code/same-project/sameproject/ops/kubeflow/__init__.py", line 2, in <module>
    from .deploy import deploy
  File "/Users/daaronch/code/same-project/sameproject/ops/kubeflow/deploy.py", line 18, in <module>
    from google.cloud import aiplatform
  File "/Users/daaronch/code/same-project/.venv/sameproject-PlHIzMk6-py3.10/lib/python3.10/site-packages/google/cloud/aiplatform/__init__.py", line 24, in <module>
    from google.cloud.aiplatform import initializer
  File "/Users/daaronch/code/same-project/.venv/sameproject-PlHIzMk6-py3.10/lib/python3.10/site-packages/google/cloud/aiplatform/initializer.py", line 26, in <module>
    from google.api_core import gapic_v1
  File "/Users/daaronch/code/same-project/.venv/sameproject-PlHIzMk6-py3.10/lib/python3.10/site-packages/google/api_core/gapic_v1/__init__.py", line 16, in <module>
    from google.api_core.gapic_v1 import config
  File "/Users/daaronch/code/same-project/.venv/sameproject-PlHIzMk6-py3.10/lib/python3.10/site-packages/google/api_core/gapic_v1/config.py", line 23, in <module>
    import grpc
  File "/Users/daaronch/code/same-project/.venv/sameproject-PlHIzMk6-py3.10/lib/python3.10/site-packages/grpc/__init__.py", line 22, in <module>
    from grpc import _compression
  File "/Users/daaronch/code/same-project/.venv/sameproject-PlHIzMk6-py3.10/lib/python3.10/site-packages/grpc/_compression.py", line 15, in <module>
    from grpc._cython import cygrpc
ImportError: dlopen(/Users/daaronch/code/same-project/.venv/sameproject-PlHIzMk6-py3.10/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-darwin.so, 0x0002): tried: '/Users/daaronch/code/same-project/.venv/sameproject-PlHIzMk6-py3.10/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e'))

We REALLY should do our best to avoid requiring users to compile in different environments. Thoughts?

cc @lukemarsden @Bubblyworld

Bubblyworld commented 2 years ago

Hmm, as far as I can tell that's a dependency of kubeflow right? I think the stacktrace is coming from ops/kubeflow because the kubeflow sdk does some code generation magic - we don't directly depend on google.cloud anywhere, for instance. 🤔 puts us in an awkward position

Last meeting you said something about pushing yamls to a kubeflow server rather than using the sdk, I think @aronchick, wonder if that would solve this issue? (since we maybe wouldn't need the sdk)