search

grpc / grpc

The C based gRPC (C++, Python, Ruby, Objective-C, PHP, C#)
https://grpc.io
Apache License 2.0
41.69k stars 10.51k forks source link

gRPC does not compile on Mac OS Catalina (10.15.6), Big Sur (11.0.1) #24677

Closed imarinmed closed 2 years ago

imarinmed commented 3 years ago

What version of gRPC and what language are you using?

Several versions of gRPC are used, and this error is reproduced in all of them:

What operating system (Linux, Windows,...) and version?

This problem is observed in both macOS Catalina (10.15.6) and macOS Big Sur (11.0.1).

What runtime / compiler are you using (e.g. python version or version of gcc)

For macOS Catalina:

Python 3.8.5 (default) 
[Clang 6.0 (clang-600.0.57)] on darwin

For macOS Big Sur:

Python 3.9.0 (default) 
[Clang 12.0.0 (clang-1200.0.31.1)] on darwin

What did you do?

Several approaches, trying to observe or correct the problem itself. It was first discovered when installing kubemq, a library that has grpcio as a direct dependency. Then, the problem arises when installing it, either by poetry (which uses pip) or directly by pip:

  1. poetry install kubemq Or directly,

  2. poetry install grpcio This problem evidently arises the same, if using pip, as mentioned above.

    What did you expect to see?

    A successful installation of either kubemq, or grpcio.

    What did you see instead?

    Here is the full error message, plus the stack trace:

    
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/usr/include -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -D_WIN32_WINNT=1536 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 -DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject* -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -Ithird_party/cares -Ithird_party/cares/cares -Ithird_party/cares/config_darwin -Ithird_party/re2 -Ithird_party/boringssl-with-bazel/src/include -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/zlib -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/user/Developer/dev/external/grpc/env/include -I/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c third_party/zlib/gzlib.c -o python_build/temp.macosx-11.0-x86_64-3.9/third_party/zlib/gzlib.o -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -pthread
third_party/zlib/gzlib.c:252:9: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        LSEEK(state->fd, 0, SEEK_END);  /* so gzoffset() is correct */
        ^
third_party/zlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
#  define LSEEK lseek
                ^
third_party/zlib/gzlib.c:252:9: note: did you mean 'fseek'?
third_party/zlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
#  define LSEEK lseek
                ^
/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/usr/include/stdio.h:162:6: note: 'fseek' declared here
int      fseek(FILE *, long, int);
         ^
third_party/zlib/gzlib.c:258:24: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        state->start = LSEEK(state->fd, 0, SEEK_CUR);
                       ^
third_party/zlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
#  define LSEEK lseek
                ^
third_party/zlib/gzlib.c:359:9: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    if (LSEEK(state->fd, state->start, SEEK_SET) == -1)
        ^
third_party/zlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
#  define LSEEK lseek
                ^
third_party/zlib/gzlib.c:400:15: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        ret = LSEEK(state->fd, offset - state->x.have, SEEK_CUR);
              ^
third_party/zlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
#  define LSEEK lseek
                ^
third_party/zlib/gzlib.c:496:14: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    offset = LSEEK(state->fd, 0, SEEK_CUR);
             ^
third_party/zlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
#  define LSEEK lseek
                ^
5 errors generated.
creating None/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/tmpfh161rfz
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/usr/include -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/user/Developer/dev/external/grpc/env/include -I/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c /var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/tmpfh161rfz/a.c -o None/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/tmpfh161rfz/a.o
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/unixccompiler.py", line 117, in _compile
    self.spawn(compiler_so + cc_args + [src, '-o', obj] +
  File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/_spawn_patch.py", line 54, in _commandfile_spawn
    _classic_spawn(self, command)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/ccompiler.py", line 910, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/spawn.py", line 87, in spawn
    raise DistutilsExecError(
distutils.errors.DistutilsExecError: command '/usr/bin/clang' failed with exit code 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/commands.py", line 268, in build_extensions
    build_ext.build_ext.build_extensions(self)
  File "/Users/user/Developer/dev/external/grpc/env/lib/python3.9/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
    _build_ext.build_ext.build_extensions(self)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/Users/user/Developer/dev/external/grpc/env/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 208, in build_extension
    _build_ext.build_extension(self, ext)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 529, in build_extension
    objects = self.compiler.compile(sources,
  File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/_parallel_compile_patch.py", line 58, in _parallel_compile
    multiprocessing.pool.ThreadPool(BUILD_EXT_COMPILER_JOBS).map(
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/_parallel_compile_patch.py", line 54, in _compile_single_file
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/commands.py", line 251, in new_compile
    return old_compile(obj, src, ext, cc_args, extra_postargs,
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/unixccompiler.py", line 120, in _compile
    raise CompileError(msg)
distutils.errors.CompileError: command '/usr/bin/clang' failed with exit code 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/setup.py", line 471, in <module>
    setuptools.setup(
  File "/Users/user/Developer/dev/external/grpc/env/lib/python3.9/site-packages/setuptools/__init__.py", line 165, in setup
    return distutils.core.setup(**attrs)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/user/Developer/dev/external/grpc/env/lib/python3.9/site-packages/setuptools/command/install.py", line 61, in run
    return orig.install.run(self)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/install.py", line 546, in run
    self.run_command('build')
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/user/Developer/dev/external/grpc/env/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 87, in run
    _build_ext.run(self)
  File "/Users/user/Developer/dev/external/grpc/env/lib/python3.9/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/commands.py", line 272, in build_extensions
    raise CommandError(
commands.CommandError: Failed `build_ext` step:
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/unixccompiler.py", line 117, in _compile
    self.spawn(compiler_so + cc_args + [src, '-o', obj] +
  File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/_spawn_patch.py", line 54, in _commandfile_spawn
    _classic_spawn(self, command)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/ccompiler.py", line 910, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/spawn.py", line 87, in spawn
    raise DistutilsExecError(
distutils.errors.DistutilsExecError: command '/usr/bin/clang' failed with exit code 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/commands.py", line 268, in build_extensions
    build_ext.build_ext.build_extensions(self)
  File "/Users/user/Developer/dev/external/grpc/env/lib/python3.9/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
    _build_ext.build_ext.build_extensions(self)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/Users/user/Developer/dev/external/grpc/env/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 208, in build_extension
    _build_ext.build_extension(self, ext)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 529, in build_extension
    objects = self.compiler.compile(sources,
  File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/_parallel_compile_patch.py", line 58, in _parallel_compile
    multiprocessing.pool.ThreadPool(BUILD_EXT_COMPILER_JOBS).map(
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/_parallel_compile_patch.py", line 54, in _compile_single_file
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/commands.py", line 251, in new_compile
    return old_compile(obj, src, ext, cc_args, extra_postargs,
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/unixccompiler.py", line 120, in _compile
    raise CompileError(msg)
distutils.errors.CompileError: command '/usr/bin/clang' failed with exit code 1

----------------------------------------
ERROR: Command errored out with exit status 1: /Users/user/Developer/dev/external/grpc/env/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/setup.py'"'"'; __file__='"'"'/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-record-6ckz6lgw/install-record.txt --single-version-externally-managed --compile --install-headers /Users/user/Developer/dev/external/grpc/env/include/site/python3.9/grpcio Check the logs for full command output.


### Anything else we should know about your project / environment?
Insofar as I have observed, the key here is the LSEEK complaint: I presume Clang tries to execute zlib code that is not compliant with proper C99 expression.

As a proposal, to solve this problem temporarily would be to pass a `Wnoimplicit-function-declaration` as a C flag during compilation to ignore this issue, and then further address the offending section on `zlib` in the future.
tfcace commented 3 years ago

I'm seeing similar issues with zlib on MacOS Big Sur

smellman commented 3 years ago

Workaround for MacOS Big Sur(Intel and Arm) with Python 3.9.0.

GRPC_PYTHON_BUILD_SYSTEM_ZLIB=true pip install grpcio
tfcace commented 3 years ago

With GRPC_PYTHON_BUILD_SYSTEM_ZLIB=true I'm now getting different errors related to openssl My Python is installed with the brew openSSL.

     In file included from third_party/boringssl-with-bazel/src/ssl/ssl_transcript.cc:138:
      In file included from third_party/boringssl-with-bazel/src/include/openssl/buf.h:60:
      third_party/boringssl-with-bazel/src/include/openssl/base.h:176:9: warning: 'OPENSSL_VERSION_NUMBER' macro redefined [-Wmacro-redefined]
      #define OPENSSL_VERSION_NUMBER 0x1010107f
              ^
      /usr/local/opt/openssl@1.1/include/openssl/opensslv.h:42:10: note: previous definition is here
      # define OPENSSL_VERSION_NUMBER  0x1010108fL
               ^
      In file included from third_party/boringssl-with-bazel/src/ssl/ssl_transcript.cc:138:
      In file included from third_party/boringssl-with-bazel/src/include/openssl/buf.h:60:
      third_party/boringssl-with-bazel/src/include/openssl/base.h:211:9: warning: 'OPENSSL_EXPORT' macro redefined [-Wmacro-redefined]
      #define OPENSSL_EXPORT
              ^
      /usr/local/opt/openssl@1.1/include/openssl/e_os2.h:171:11: note: previous definition is here
      #  define OPENSSL_EXPORT extern
                ^
      In file included from third_party/boringssl-with-bazel/src/ssl/ssl_transcript.cc:138:
      In file included from third_party/boringssl-with-bazel/src/include/openssl/buf.h:60:
      third_party/boringssl-with-bazel/src/include/openssl/base.h:318:13: error: typedef redefinition with different types ('int' vs 'struct crypto_threadid_st')
      typedef int CRYPTO_THREADID;
                  ^
      /usr/local/opt/openssl@1.1/include/openssl/crypto.h:231:3: note: previous definition is here
      } CRYPTO_THREADID;
        ^
      In file included from third_party/boringssl-with-bazel/src/ssl/ssl_transcript.cc:138:
      In file included from third_party/boringssl-with-bazel/src/include/openssl/buf.h:60:
      third_party/boringssl-with-bazel/src/include/openssl/base.h:388:29: error: typedef redefinition with different types ('struct ecdsa_sig_st' vs 'struct ECDSA_SIG_st')
      typedef struct ecdsa_sig_st ECDSA_SIG;
                                  ^
      /usr/local/opt/openssl@1.1/include/openssl/ec.h:1127:29: note: previous definition is here
      typedef struct ECDSA_SIG_st ECDSA_SIG;
                                  ^
      In file included from third_party/boringssl-with-bazel/src/ssl/ssl_transcript.cc:138:
      In file included from third_party/boringssl-with-bazel/src/include/openssl/buf.h:60:
      third_party/boringssl-with-bazel/src/include/openssl/base.h:390:30: error: typedef redefinition with different types ('struct env_md_ctx_st' vs 'struct evp_md_ctx_st')
      typedef struct env_md_ctx_st EVP_MD_CTX;
                                   ^
      /usr/local/opt/openssl@1.1/include/openssl/ossl_typ.h:92:30: note: previous definition is here
      typedef struct evp_md_ctx_st EVP_MD_CTX;
                                   ^
      In file included from third_party/boringssl-with-bazel/src/ssl/ssl_transcript.cc:138:
      In file included from third_party/boringssl-with-bazel/src/include/openssl/buf.h:60:
      third_party/boringssl-with-bazel/src/include/openssl/base.h:391:26: error: typedef redefinition with different types ('struct env_md_st' vs 'struct evp_md_st')
      typedef struct env_md_st EVP_MD;
                               ^
      /usr/local/opt/openssl@1.1/include/openssl/ossl_typ.h:91:26: note: previous definition is here
      typedef struct evp_md_st EVP_MD;
                               ^
      In file included from third_party/boringssl-with-bazel/src/ssl/ssl_transcript.cc:138:
      In file included from third_party/boringssl-with-bazel/src/include/openssl/buf.h:60:
      third_party/boringssl-with-bazel/src/include/openssl/base.h:395:34: error: typedef redefinition with different types ('struct evp_encode_ctx_st' vs 'struct evp_Encode_Ctx_st')
      typedef struct evp_encode_ctx_st EVP_ENCODE_CTX;
                                       ^
      /usr/local/opt/openssl@1.1/include/openssl/ossl_typ.h:100:34: note: previous definition is here
      typedef struct evp_Encode_Ctx_st EVP_ENCODE_CTX;
                                       ^
      In file included from third_party/boringssl-with-bazel/src/ssl/ssl_transcript.cc:138:
      In file included from third_party/boringssl-with-bazel/src/include/openssl/buf.h:60:
      third_party/boringssl-with-bazel/src/include/openssl/base.h:411:32: error: typedef redefinition with different types ('struct sha256_state_st' vs 'struct SHA256state_st')
      typedef struct sha256_state_st SHA256_CTX;
                                     ^
      /usr/local/opt/openssl@1.1/include/openssl/sha.h:56:3: note: previous definition is here
      } SHA256_CTX;
        ^
      In file included from third_party/boringssl-with-bazel/src/ssl/ssl_transcript.cc:138:
      In file included from third_party/boringssl-with-bazel/src/include/openssl/buf.h:60:
      third_party/boringssl-with-bazel/src/include/openssl/base.h:412:32: error: typedef redefinition with different types ('struct sha512_state_st' vs 'struct SHA512state_st')
      typedef struct sha512_state_st SHA512_CTX;
                                     ^
      /usr/local/opt/openssl@1.1/include/openssl/sha.h:103:3: note: previous definition is here
      } SHA512_CTX;
        ^
      In file included from third_party/boringssl-with-bazel/src/ssl/ssl_transcript.cc:138:
      In file included from third_party/boringssl-with-bazel/src/include/openssl/buf.h:60:
      third_party/boringssl-with-bazel/src/include/openssl/base.h:413:29: error: typedef redefinition with different types ('struct sha_state_st' vs 'struct SHAstate_st')
      typedef struct sha_state_st SHA_CTX;
                                  ^
      /usr/local/opt/openssl@1.1/include/openssl/sha.h:39:3: note: previous definition is here
      } SHA_CTX;
        ^
      In file included from third_party/boringssl-with-bazel/src/ssl/ssl_transcript.cc:138:
      third_party/boringssl-with-bazel/src/include/openssl/buf.h:71:8: error: redefinition of 'buf_mem_st'
      struct buf_mem_st {
             ^
      /usr/local/opt/openssl@1.1/include/openssl/buffer.h:38:8: note: previous definition is here
      struct buf_mem_st {
             ^
      In file included from third_party/boringssl-with-bazel/src/ssl/ssl_transcript.cc:138:
      third_party/boringssl-with-bazel/src/include/openssl/buf.h:104:22: error: expected parameter declarator
      OPENSSL_EXPORT char *BUF_strdup(const char *str);
                           ^
      /usr/local/opt/openssl@1.1/include/openssl/buffer.h:31:24: note: expanded from macro 'BUF_strdup'
      # define BUF_strdup(s) OPENSSL_strdup(s)
                             ^
      /usr/local/opt/openssl@1.1/include/openssl/crypto.h:132:28: note: expanded from macro 'OPENSSL_strdup'
              CRYPTO_strdup(str, OPENSSL_FILE, OPENSSL_LINE)
                                 ^
      /usr/local/opt/openssl@1.1/include/openssl/opensslconf.h:133:24: note: expanded from macro 'OPENSSL_FILE'
      #  define OPENSSL_FILE __FILE__
                             ^
      <scratch space>:67:1: note: expanded from here
      "third_party/boringssl-with-bazel/src/include/openssl/buf.h"
      ^
      In file included from third_party/boringssl-with-bazel/src/ssl/ssl_transcript.cc:138:
      third_party/boringssl-with-bazel/src/include/openssl/buf.h:104:22: error: expected ')'
      /usr/local/opt/openssl@1.1/include/openssl/buffer.h:31:24: note: expanded from macro 'BUF_strdup'
      # define BUF_strdup(s) OPENSSL_strdup(s)
                             ^
      /usr/local/opt/openssl@1.1/include/openssl/crypto.h:132:28: note: expanded from macro 'OPENSSL_strdup'
              CRYPTO_strdup(str, OPENSSL_FILE, OPENSSL_LINE)
                                 ^
      /usr/local/opt/openssl@1.1/include/openssl/opensslconf.h:133:24: note: expanded from macro 'OPENSSL_FILE'
      #  define OPENSSL_FILE __FILE__
                             ^
      <scratch space>:67:1: note: expanded from here
      "third_party/boringssl-with-bazel/src/include/openssl/buf.h"
      ^
      third_party/boringssl-with-bazel/src/include/openssl/buf.h:104:22: note: to match this '('
      /usr/local/opt/openssl@1.1/include/openssl/buffer.h:31:24: note: expanded from macro 'BUF_strdup'
      # define BUF_strdup(s) OPENSSL_strdup(s)
                             ^
      /usr/local/opt/openssl@1.1/include/openssl/crypto.h:132:22: note: expanded from macro 'OPENSSL_strdup'
              CRYPTO_strdup(str, OPENSSL_FILE, OPENSSL_LINE)
                           ^
      In file included from third_party/boringssl-with-bazel/src/ssl/ssl_transcript.cc:138:
      third_party/boringssl-with-bazel/src/include/openssl/buf.h:104:22: error: conflicting types for 'CRYPTO_strdup'
      OPENSSL_EXPORT char *BUF_strdup(const char *str);
                           ^
      /usr/local/opt/openssl@1.1/include/openssl/buffer.h:31:24: note: expanded from macro 'BUF_strdup'
      # define BUF_strdup(s) OPENSSL_strdup(s)
                             ^
      /usr/local/opt/openssl@1.1/include/openssl/crypto.h:132:9: note: expanded from macro 'OPENSSL_strdup'
              CRYPTO_strdup(str, OPENSSL_FILE, OPENSSL_LINE)
              ^
      /usr/local/opt/openssl@1.1/include/openssl/crypto.h:269:7: note: previous declaration is here
      char *CRYPTO_strdup(const char *str, const char *file, int line);
            ^
      In file included from third_party/boringssl-with-bazel/src/ssl/ssl_transcript.cc:138:
      third_party/boringssl-with-bazel/src/include/openssl/buf.h:110:22: error: expected parameter declarator
      OPENSSL_EXPORT char *BUF_strndup(const char *str, size_t size);
                           ^
      /usr/local/opt/openssl@1.1/include/openssl/buffer.h:32:31: note: expanded from macro 'BUF_strndup'
      # define BUF_strndup(s, size) OPENSSL_strndup(s, size)
                                    ^
      /usr/local/opt/openssl@1.1/include/openssl/crypto.h:134:32: note: expanded from macro 'OPENSSL_strndup'
              CRYPTO_strndup(str, n, OPENSSL_FILE, OPENSSL_LINE)
                                     ^
      /usr/local/opt/openssl@1.1/include/openssl/opensslconf.h:133:24: note: expanded from macro 'OPENSSL_FILE'
      #  define OPENSSL_FILE __FILE__
                             ^
      <scratch space>:69:1: note: expanded from here
      "third_party/boringssl-with-bazel/src/include/openssl/buf.h"
      ^
      In file included from third_party/boringssl-with-bazel/src/ssl/ssl_transcript.cc:138:
      third_party/boringssl-with-bazel/src/include/openssl/buf.h:110:22: error: expected ')'
      /usr/local/opt/openssl@1.1/include/openssl/buffer.h:32:31: note: expanded from macro 'BUF_strndup'
      # define BUF_strndup(s, size) OPENSSL_strndup(s, size)
                                    ^
      /usr/local/opt/openssl@1.1/include/openssl/crypto.h:134:32: note: expanded from macro 'OPENSSL_strndup'
              CRYPTO_strndup(str, n, OPENSSL_FILE, OPENSSL_LINE)
                                     ^
      /usr/local/opt/openssl@1.1/include/openssl/opensslconf.h:133:24: note: expanded from macro 'OPENSSL_FILE'
      #  define OPENSSL_FILE __FILE__
                             ^
smellman commented 3 years ago

I use openssl with homebrew but I didn't get this error. @tfcace Do you try to build with GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=true?

imarinmed commented 3 years ago

Unfortunately, the GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=true approach does not work with us either: Clang continues to complain in the same way. However, you have given me an idea: is there any way we can provide to the pip install grpcio with the ignores (-Wunused-variable, -Wnoexceptions-raised, etc) we would need, at least, to 'build' it?

I am aware that we may end up building a broken grpcio, but if any, this could be interesting in the future as a way to resolve these compilation problems in short-term, while better minds than me solve the issue for long-term...

tfcace commented 3 years ago

I haven't tried GRPC_PYTHON_BUILD_SYSTEM_OPENSSL, I wonder what version of OpenSSL that will build against? The brew one? And if yes, how? I believe I tried brew link for openSSL and MacOS wasn't happy about it.

smellman commented 3 years ago

@tfcace I found the code so I asked you. My case is only "workaround", and it is not "solution".

e-simpson commented 3 years ago

I am also encountering these errors pertaining to zlib on Mac OS Big Sur. For the time being I was able to get around these build errors by making the following changes in my local zlib files:

This is a workaround and may have un-intended consequences.

VagifMammadaliyev commented 3 years ago

@e-simpson Did you just changed those files and pip install grpcio?

VagifMammadaliyev commented 3 years ago

Finally made it work by running:

pip install wheel && GRPC_BUILD_WITH_BORING_SSL_ASM="" GRPC_PYTHON_BUILD_SYSTEM_RE2=true GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=true GRPC_PYTHON_BUILD_SYSTEM_ZLIB=true pip install grpcio

I'm very tired now, so to be honest the only thing I did was just copy-pasting every env variable from every issue I found ☺️. Main references: #24248 #24449

tmarkovski commented 3 years ago

Thanks @VagifMammadaliyev - this gave me a huge headache. Your solution finally worked. Hoping the team addresses this.

imarinmed commented 3 years ago

Wonderful solution, @VagifMammadaliyev! You have really made my day 👍🏼 Beers are on me whenever you visit Berlin! 🍻

neeraj15022001 commented 3 years ago

Thanks @VagifMammadaliyev for this workaround. This issue was freaking me out from past 2 days and i just came across your workaround and it worked. Once again thanks a lot.

mikkelam commented 3 years ago

The above didn't work for me for whatever reason but, changing the macos version of the wheel worked for me:

Here i'm using 1.33.2 on cpython 3.8. FOr other versions, check out https://pypi.org/project/grpcio/1.33.2/#files

❯ curl -LOJ "https://files.pythonhosted.org/packages/fb/4d/4b6146ed211a3211231b1253cfeb58818875b5b57a4622602d1b87e16f73/grpcio-1.33.2-cp38-cp38-macosx_10_9_x86_64.whl"
❯ mv grpcio-1.33.2-cp38-cp38-macosx_10_9_x86_64.whl grpcio-1.33.2-cp38-cp38-macosx_11_0_x86_64.whl
❯ pip install grpcio-1.33.2-cp38-cp38-macosx_11_0_x86_64.whl
owlas commented 3 years ago

@VagifMammadaliyev - I introduced another error with GRPC_PYTHON_BUILD_SYSTEM_RE2=true

The following works for me:

GRPC_BUILD_WITH_BORING_SSL_ASM="" GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=true GRPC_PYTHON_BUILD_SYSTEM_ZLIB=true

SarvagyaVaish commented 3 years ago

@VagifMammadaliyev you are a hero!

One additional comment - i had to make sure my pip was up to date before the commend in https://github.com/grpc/grpc/issues/24677#issuecomment-729983060 worked for me. 

python -m pip install --upgrade pip
flunderpero commented 3 years ago

For me, all it took was a python -m pip install --upgrade pip to make it work.

bajohn commented 3 years ago

I had a similar issue, my specific case being google-cloud-bigquery being unable to install grpcio using pipenv on Big Sur.

A quick fix that worked for me was running 

export SYSTEM_VERSION_COMPAT=1

Afterwards, clang errors were no longer thrown and I was able to install grpcio and google-cloud-bigquery via pipenv. Not sure if this fix applies to the original issue but hope it helps!

jrlindell commented 3 years ago 
pip install wheel && GRPC_BUILD_WITH_BORING_SSL_ASM="" GRPC_PYTHON_BUILD_SYSTEM_RE2=true GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=true GRPC_PYTHON_BUILD_SYSTEM_ZLIB=true pip install grpcio

did not work for the new Mac M1 chip, but was able to download grpc with https://github.com/grpc/grpc/issues/24677#issuecomment-741814641

AdamCraven commented 3 years ago

In the end on the Mac I went with tf-nightly (https://www.tensorflow.org/install). grpcio 1.32.0 bundled with tensorflow 2.4.1 is hard to get working on Big Sur. Whereas 1.34.1 works ok.

JoePacman commented 3 years ago 
pip install wheel && GRPC_BUILD_WITH_BORING_SSL_ASM="" GRPC_PYTHON_BUILD_SYSTEM_RE2=true GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=true GRPC_PYTHON_BUILD_SYSTEM_ZLIB=true pip install grpcio

did not work for the new Mac M1 chip, but was able to download grpc with #24677 (comment)

I had similar clang related problems installing grpcio on mac. In the end I completely reinstalled brew and pyenv using Rosetta terminal (following https://www.reddit.com/r/MacOS/comments/kghwef/best_method_to_install_homebrew_on_m1/), and then installing grpcio worked fine using an up to date version of pip (without needing to set any additional environment variables in the command). Posting in case it helps anyone 👍

lidalei commented 3 years ago

Upgrading pip to 21.1.1 solved my issue https://issues.apache.org/jira/browse/ARROW-10744.

berndverst commented 3 years ago

On M1 ARM Mac it is possible to run into all kinds of issues. You must not use GNU binary utils or a version of Python built with the GNU binary utils. I had installed Python via pyenv which used the binary utils installed by homebrew. In the end here is what worked for me:

brew uninstall binutils
pyenv uninstall 3.9.4
pyenv install 3.9.4
GRPC_PYTHON_BUILD_SYSTEM_ZLIB=true pip3 install grpcio grpcio-tools

The easiest way to know whether you are using the wrong binary utils is to check the output of $(which ranlib) -V. If you are using the Apple provided binary utils as you should, you will see the following error (which is a good sign):

Apple Inc. version cctools-980
error: /Library/Developer/CommandLineTools/usr/bin/ranlib: no archives specified
Usage: /Library/Developer/CommandLineTools/usr/bin/ranlib [-sactfqLT] [-] archive [...]

Hope this helps someone!

thebaron commented 3 years ago

The grpc...true pip 21. upgrade combo worked for me. Big sur. 11.3. M1.

tongyifan commented 3 years ago

For me:

brew install openssl re2
LDFLAGS="-L$(/opt/homebrew/bin/brew --prefix openssl)/lib -L$(/opt/homebrew/bin/brew --prefix re2)/lib" CPPFLAGS="-I$(/opt/homebrew/bin/brew --prefix openssl)/include -I$(/opt/homebrew/bin/brew --prefix re2)/include" GRPC_BUILD_WITH_BORING_SSL_ASM="" GRPC_PYTHON_BUILD_SYSTEM_RE2=true GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=true GRPC_PYTHON_BUILD_SYSTEM_ZLIB=true pip install grpcio
llimllib commented 3 years ago

@tongyifan thank you very much, none of the other options worked for me and I figured I was going to have to go down a deep dark LDFLAGS hole... but you did that for the rest of us

gnossen commented 3 years ago

This has been fixed on master and will be included in 1.39.0, which will be released a week from now.

dgaedcke commented 3 years ago

I'm running this: pip3 install grpcio==1.39.0rc1 --no-cache-dir --force

and still getting failure on M1 (terminal in arm64 mode)

Successfully built grpcio WARNING: Ignoring invalid distribution -rpcio (/opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.9/site-packages) Installing collected packages: six, grpcio Attempting uninstall: six WARNING: Ignoring invalid distribution -rpcio (/opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.9/site-packages) Found existing installation: six 1.16.0 Uninstalling six-1.16.0: Successfully uninstalled six-1.16.0 WARNING: Ignoring invalid distribution -rpcio (/opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.9/site-packages) WARNING: Ignoring invalid distribution -rpcio (/opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.9/site-packages) Successfully installed grpcio-1.39.0rc1 six-1.16.0 WARNING: Ignoring invalid distribution -rpcio (/opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.9/site-packages) WARNING: Ignoring invalid distribution -rpcio (/opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.9/site-packages)

now start Py3 and test import

(env) ~/dev/flask_gcp_py3_datastore$ python Python 3.9.5 | packaged by conda-forge | (default, Jun 19 2021, 00:24:55) [Clang 11.1.0 ] on darwin Type "help", "copyright", "credits" or "license" for more information.

import google.cloud.ndb as ndb Traceback (most recent call last): File "", line 1, in File "/opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.9/site-packages/google/cloud/ndb/init.py", line 28, in from google.cloud.ndb.client import Client File "/opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.9/site-packages/google/cloud/ndb/client.py", line 18, in import grpc File "/opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.9/site-packages/grpc/init.py", line 23, in from grpc._cython import cygrpc as _cygrpc ImportError: dlopen(/opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.9/site-packages/grpc/_cython/cygrpc.cpython-39-darwin.so, 2): Symbol not found: _CFRelease Referenced from: /opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.9/site-packages/grpc/_cython/cygrpc.cpython-39-darwin.so Expected in: flat namespace in /opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.9/site-packages/grpc/_cython/cygrpc.cpython-39-darwin.so

dgaedcke commented 3 years ago

version 1.39.0rc1 is not working for me

but miniforge has a version that seems to work https://github.com/conda-forge/grpcio-feedstock

gnossen commented 3 years ago

@dgaedcke We're looking into the issue you're seeing on M1. Given the time constraints though, the fix will not make it into the 1.39 release.

kavyapadmanabha commented 3 years ago

@tongyifan , Thanks very much. Your solution worked for me.

lclpedro commented 2 years ago

if doesn't resolved above, try install python 3.9.8 or superior..

wojiushixiaobai commented 2 years ago

@VagifMammadaliyev - I introduced another error with GRPC_PYTHON_BUILD_SYSTEM_RE2=true

The following works for me:

GRPC_BUILD_WITH_BORING_SSL_ASM="" GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=true GRPC_PYTHON_BUILD_SYSTEM_ZLIB=true

thk. @owlas

gnossen commented 2 years ago

Closing since this was resolved by the 1.39.0 release, long ago.

PiotrWegrzyn commented 1 year ago

Still an issue for me on a M1 mac with Ventura OS. This helped:

env LDFLAGS="-L$(brew --prefix openssl@1.1)/lib" CFLAGS="-I$(brew --prefix openssl@1.1)/include" pip install git+https://github.com/shivaRamdeen/passbook.git

nimirium commented 1 year ago

Here's my collection of env variables that finally worked for me on my M2 chip:

export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1
export GRPC_PYTHON_BUILD_SYSTEM_RE2=1
export SYSTEM_VERSION_COMPAT=1
export LDFLAGS="-L$(/opt/homebrew/bin/brew --prefix openssl)/lib -L$(/opt/homebrew/bin/brew --prefix re2)/lib"
export CPPFLAGS="-I$(/opt/homebrew/bin/brew --prefix openssl)/include -I$(/opt/homebrew/bin/brew --prefix re2)/include" 
export CFLAGS="-I$(/opt/homebrew/bin/brew --prefix openssl)/include -I$(/opt/homebrew/bin/brew --prefix re2)/include"
export GRPC_BUILD_WITH_BORING_SSL_ASM=""

it's similar to @tongyifan 's answer but I also added re2 to CFLAGS