mozilla / DeepSpeech

DeepSpeech is an open source embedded (offline, on-device) speech-to-text engine which can run in real time on devices ranging from a Raspberry Pi 4 to high power GPU servers.
Mozilla Public License 2.0
25.36k stars 3.97k forks source link

no such package '@com_google_protobuf//' #2986

Closed onnoeberhard closed 4 years ago

onnoeberhard commented 4 years ago

Hello, I am trying to build DeepSpeech according to the instructions on https://github.com/mozilla/DeepSpeech/tree/master/native_client. When running the bazel build command (written above, unchanged from the instructions) I get the following output:

Starting local Bazel server and connecting to it...
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=137
INFO: Reading rc options for 'build' from /Users/onno/Projects/ltl/deepspeech/tensorflow/.bazelrc:
  'build' options: --apple_platform_type=macos --define framework_shared_object=true --define open_source_build=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone --strategy=Genrule=standalone -c opt --announce_rc --define=grpc_no_ares=true --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include
INFO: Reading rc options for 'build' from /Users/onno/Projects/ltl/deepspeech/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/Users/onno/anaconda3/envs/deepspeech/bin/python --action_env PYTHON_LIB_PATH=/Users/onno/anaconda3/envs/deepspeech/lib/python3.7/site-packages --python_path=/Users/onno/anaconda3/envs/deepspeech/bin/python --action_env TF_CONFIGURE_IOS=0
INFO: Found applicable config definition build:monolithic in file /Users/onno/Projects/ltl/deepspeech/tensorflow/.bazelrc: --define framework_shared_object=false
INFO: Repository 'com_google_protobuf' used the following cache hits instead of downloading the corresponding file.
 * Hash 'b9e92f9af8819bbbc514e2902aec860415b70209f31dfc8c4fa72515a5df9d59' for https://storage.googleapis.com/mirror.tensorflow.org/githu
b.com/protocolbuffers/protobuf/archive/310ba5ee72661c081129eb878c1bbcec936b20f0.tar.gz
If the definition of 'com_google_protobuf' was updated, verify that the hashes were also updated.
ERROR: /Users/onno/Projects/ltl/deepspeech/tensorflow/native_client/BUILD:91:1: error loading package '@org_tensorflow//tensorflow': in /
private/var/tmp/_bazel_onno/840defbc8d3bafbd85a3cb336e64325d/external/org_tensorflow/tensorflow/core/platform/default/build_config.bzl: E
ncountered error while reading extension file 'protobuf.bzl': no such package '@com_google_protobuf//': Traceback (most recent call last):
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 108
                _apply_patch(ctx, ctx.attr.patch_file)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 75, in _apply_patch
                _execute_and_check_ret_code(ctx, cmd)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 56, in _execute_and_check_ret_code
                fail("Non-zero return code({1}) when ...))
Non-zero return code(1) when executing 'git apply -v /Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/protobuf/protobuf.patch':
Stdout: 
Stderr: Checking patch BUILD...
error: while searching for:
# ZLIB configuration?
################################################################################?
?
ZLIB_DEPS = ["@zlib//:zlib"]?
?
################################################################################?
# Protobuf Runtime Library?

error: patch failed: BUILD:19
error: BUILD: patch does not apply
 and referenced by '//native_client:libdeepspeech.so'
ERROR: /Users/onno/Projects/ltl/deepspeech/tensorflow/native_client/BUILD:91:1: error loading package '@org_tensorflow//tensorflow': in /
private/var/tmp/_bazel_onno/840defbc8d3bafbd85a3cb336e64325d/external/org_tensorflow/tensorflow/core/platform/default/build_config.bzl: E
ncountered error while reading extension file 'protobuf.bzl': no such package '@com_google_protobuf//': Traceback (most recent call last):
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 108
                _apply_patch(ctx, ctx.attr.patch_file)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 75, in _apply_patch
                _execute_and_check_ret_code(ctx, cmd)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 56, in _execute_and_check_ret_code
                fail("Non-zero return code({1}) when ...))
Non-zero return code(1) when executing 'git apply -v /Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/protobuf/protobuf.patch':
Stdout: 
Stderr: Checking patch BUILD...
error: while searching for:
# ZLIB configuration?
################################################################################?
?
ZLIB_DEPS = ["@zlib//:zlib"]?
?
################################################################################?
# Protobuf Runtime Library?

error: patch failed: BUILD:19
error: BUILD: patch does not apply
 and referenced by '//native_client:libdeepspeech.so'
ERROR: /Users/onno/Projects/ltl/deepspeech/tensorflow/native_client/BUILD:91:1: error loading package '@org_tensorflow//tensorflow': in /
private/var/tmp/_bazel_onno/840defbc8d3bafbd85a3cb336e64325d/external/org_tensorflow/tensorflow/core/platform/default/build_config.bzl: E
ncountered error while reading extension file 'protobuf.bzl': no such package '@com_google_protobuf//': Traceback (most recent call last):
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 108
                _apply_patch(ctx, ctx.attr.patch_file)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 75, in _apply_patch
                _execute_and_check_ret_code(ctx, cmd)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 56, in _execute_and_check_ret_code
                fail("Non-zero return code({1}) when ...))
Non-zero return code(1) when executing 'git apply -v /Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/protobuf/protobuf.patch':
Stdout: 
Stderr: Checking patch BUILD...
error: while searching for:
# ZLIB configuration?
################################################################################?
?
ZLIB_DEPS = ["@zlib//:zlib"]?
?
################################################################################?
# Protobuf Runtime Library?

error: patch failed: BUILD:19
error: BUILD: patch does not apply
 and referenced by '//native_client:libdeepspeech.so'
ERROR: /Users/onno/Projects/ltl/deepspeech/tensorflow/native_client/BUILD:91:1: error loading package '@org_tensorflow//tensorflow': in /
private/var/tmp/_bazel_onno/840defbc8d3bafbd85a3cb336e64325d/external/org_tensorflow/tensorflow/core/platform/default/build_config.bzl: E
ncountered error while reading extension file 'protobuf.bzl': no such package '@com_google_protobuf//': Traceback (most recent call last):
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 108
                _apply_patch(ctx, ctx.attr.patch_file)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 75, in _apply_patch
                _execute_and_check_ret_code(ctx, cmd)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 56, in _execute_and_check_ret_code
                fail("Non-zero return code({1}) when ...))
Non-zero return code(1) when executing 'git apply -v /Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/protobuf/protobuf.patch':
Stdout: 
Stderr: Checking patch BUILD...
error: while searching for:
# ZLIB configuration?
################################################################################?
?
ZLIB_DEPS = ["@zlib//:zlib"]?
?
################################################################################?
# Protobuf Runtime Library?

error: patch failed: BUILD:19
error: BUILD: patch does not apply
 and referenced by '//native_client:libdeepspeech.so'
ERROR: /Users/onno/Projects/ltl/deepspeech/tensorflow/native_client/BUILD:91:1: error loading package '@org_tensorflow//tensorflow': in /
private/var/tmp/_bazel_onno/840defbc8d3bafbd85a3cb336e64325d/external/org_tensorflow/tensorflow/core/platform/default/build_config.bzl: E
ncountered error while reading extension file 'protobuf.bzl': no such package '@com_google_protobuf//': Traceback (most recent call last):
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 108
                _apply_patch(ctx, ctx.attr.patch_file)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 75, in _apply_patch
                _execute_and_check_ret_code(ctx, cmd)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 56, in _execute_and_check_ret_code
                fail("Non-zero return code({1}) when ...))
Non-zero return code(1) when executing 'git apply -v /Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/protobuf/protobuf.patch':
Stdout: 
Stderr: Checking patch BUILD...
error: while searching for:
# ZLIB configuration?
################################################################################?
?
ZLIB_DEPS = ["@zlib//:zlib"]?
?
################################################################################?
# Protobuf Runtime Library?

error: patch failed: BUILD:19
error: BUILD: patch does not apply
 and referenced by '//native_client:libdeepspeech.so'
ERROR: /Users/onno/Projects/ltl/deepspeech/tensorflow/native_client/BUILD:91:1: error loading package '@org_tensorflow//tensorflow': in /
private/var/tmp/_bazel_onno/840defbc8d3bafbd85a3cb336e64325d/external/org_tensorflow/tensorflow/core/platform/default/build_config.bzl: E
ncountered error while reading extension file 'protobuf.bzl': no such package '@com_google_protobuf//': Traceback (most recent call last):
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 108
                _apply_patch(ctx, ctx.attr.patch_file)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 75, in _apply_patch
                _execute_and_check_ret_code(ctx, cmd)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 56, in _execute_and_check_ret_code
                fail("Non-zero return code({1}) when ...))
Non-zero return code(1) when executing 'git apply -v /Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/protobuf/protobuf.patch':
Stdout: 
Stderr: Checking patch BUILD...
error: while searching for:
# ZLIB configuration?
################################################################################?
?
ZLIB_DEPS = ["@zlib//:zlib"]?
?
################################################################################?
# Protobuf Runtime Library?

error: patch failed: BUILD:19
error: BUILD: patch does not apply
 and referenced by '//native_client:libdeepspeech.so'
ERROR: /Users/onno/Projects/ltl/deepspeech/tensorflow/native_client/BUILD:91:1: error loading package '@org_tensorflow//tensorflow': in /
private/var/tmp/_bazel_onno/840defbc8d3bafbd85a3cb336e64325d/external/org_tensorflow/tensorflow/core/platform/default/build_config.bzl: E
ncountered error while reading extension file 'protobuf.bzl': no such package '@com_google_protobuf//': Traceback (most recent call last):
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 108
                _apply_patch(ctx, ctx.attr.patch_file)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 75, in _apply_patch
                _execute_and_check_ret_code(ctx, cmd)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 56, in _execute_and_check_ret_code
                fail("Non-zero return code({1}) when ...))
Non-zero return code(1) when executing 'git apply -v /Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/protobuf/protobuf.patch':
Stdout: 
Stderr: Checking patch BUILD...
error: while searching for:
# ZLIB configuration?
################################################################################?
?
ZLIB_DEPS = ["@zlib//:zlib"]?
?
################################################################################?
# Protobuf Runtime Library?

error: patch failed: BUILD:19
error: BUILD: patch does not apply
 and referenced by '//native_client:libdeepspeech.so'
ERROR: /Users/onno/Projects/ltl/deepspeech/tensorflow/native_client/BUILD:91:1: error loading package '@org_tensorflow//tensorflow': in /
private/var/tmp/_bazel_onno/840defbc8d3bafbd85a3cb336e64325d/external/org_tensorflow/tensorflow/core/platform/default/build_config.bzl: E
ncountered error while reading extension file 'protobuf.bzl': no such package '@com_google_protobuf//': Traceback (most recent call last):
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 108
                _apply_patch(ctx, ctx.attr.patch_file)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 75, in _apply_patch
                _execute_and_check_ret_code(ctx, cmd)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 56, in _execute_and_check_ret_code
                fail("Non-zero return code({1}) when ...))
Non-zero return code(1) when executing 'git apply -v /Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/protobuf/protobuf.patch':
Stdout: 
Stderr: Checking patch BUILD...
error: while searching for:
# ZLIB configuration?
################################################################################?
?
ZLIB_DEPS = ["@zlib//:zlib"]?
?
################################################################################?
# Protobuf Runtime Library?

error: patch failed: BUILD:19
error: BUILD: patch does not apply
 and referenced by '//native_client:libdeepspeech.so'
ERROR: /Users/onno/Projects/ltl/deepspeech/tensorflow/native_client/BUILD:91:1: error loading package '@org_tensorflow//tensorflow': in /
private/var/tmp/_bazel_onno/840defbc8d3bafbd85a3cb336e64325d/external/org_tensorflow/tensorflow/core/platform/default/build_config.bzl: E
ncountered error while reading extension file 'protobuf.bzl': no such package '@com_google_protobuf//': Traceback (most recent call last):
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 108
                _apply_patch(ctx, ctx.attr.patch_file)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 75, in _apply_patch
                _execute_and_check_ret_code(ctx, cmd)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 56, in _execute_and_check_ret_code
                fail("Non-zero return code({1}) when ...))
Non-zero return code(1) when executing 'git apply -v /Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/protobuf/protobuf.patch':
Stdout: 
Stderr: Checking patch BUILD...
error: while searching for:
# ZLIB configuration?
################################################################################?
?
ZLIB_DEPS = ["@zlib//:zlib"]?
?
################################################################################?
# Protobuf Runtime Library?

error: patch failed: BUILD:19
error: BUILD: patch does not apply
 and referenced by '//native_client:libdeepspeech.so'
ERROR: /Users/onno/Projects/ltl/deepspeech/tensorflow/native_client/BUILD:91:1: error loading package '@org_tensorflow//tensorflow': in /
private/var/tmp/_bazel_onno/840defbc8d3bafbd85a3cb336e64325d/external/org_tensorflow/tensorflow/core/platform/default/build_config.bzl: E
ncountered error while reading extension file 'protobuf.bzl': no such package '@com_google_protobuf//': Traceback (most recent call last):
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 108
                _apply_patch(ctx, ctx.attr.patch_file)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 75, in _apply_patch
                _execute_and_check_ret_code(ctx, cmd)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 56, in _execute_and_check_ret_code
                fail("Non-zero return code({1}) when ...))
Non-zero return code(1) when executing 'git apply -v /Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/protobuf/protobuf.patch':
Stdout: 
Stderr: Checking patch BUILD...
error: while searching for:
# ZLIB configuration?
################################################################################?
?
ZLIB_DEPS = ["@zlib//:zlib"]?
?
################################################################################?
# Protobuf Runtime Library?

error: patch failed: BUILD:19
error: BUILD: patch does not apply
 and referenced by '//native_client:libdeepspeech.so'
ERROR: Analysis of target '//native_client:libdeepspeech.so' failed; build aborted: error loading package '@org_tensorflow//tensorflow': 
in /private/var/tmp/_bazel_onno/840defbc8d3bafbd85a3cb336e64325d/external/org_tensorflow/tensorflow/core/platform/default/build_config.bz
l: Encountered error while reading extension file 'protobuf.bzl': no such package '@com_google_protobuf//': Traceback (most recent call last):
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 108
                _apply_patch(ctx, ctx.attr.patch_file)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 75, in _apply_patch
                _execute_and_check_ret_code(ctx, cmd)
        File "/Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/repo.bzl", line 56, in _execute_and_check_ret_code
                fail("Non-zero return code({1}) when ...))
Non-zero return code(1) when executing 'git apply -v /Users/onno/Projects/ltl/deepspeech/tensorflow/third_party/protobuf/protobuf.patch':
Stdout: 
Stderr: Checking patch BUILD...
error: while searching for:
# ZLIB configuration?
################################################################################?
?
ZLIB_DEPS = ["@zlib//:zlib"]?
?
################################################################################?
# Protobuf Runtime Library?

error: patch failed: BUILD:19
error: BUILD: patch does not apply
INFO: Elapsed time: 4.720s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (2 packages loaded, 0 targets configured)
    currently loading: tensorflow ... (2 packages)

It seems that the actual error message is no such package '@com_google_protobuf//'. I do have protobuf version 3.11.3 installed. I found similar issues mentioned elsewhere but the solution always seemed to be to change the bazel version, but bazel 0.24.1 ist explicitly mentioned in the instructions (and I get different errors when trying different bazel versions).

lissyx commented 4 years ago

It seems it fails to apply a patch. I have no idea why and it looks more tensorflow level than DeepSpeech. Make sure you are on our r1.15 branch and not upstream one, but 0.24.1 should work. Try purging bazel cache as well?

onnoeberhard commented 4 years ago

Yes I am on the r1.15 branch. I tried removing bazel using rm -fr ~/.bazel ~/.bazelrc ~/.cache/bazel and reinstalling it but the error remains the same..

reuben commented 4 years ago

It can be tricky to understand all the ways bazel does cashing, so instead of deleting folders by hand try running bazel clean --expunge.

Am 08.05.2020 um 19:36 schrieb Onno Eberhard notifications@github.com:

 Yes I am on the r1.15 branch. I tried removing bazel using rm -fr ~/.bazel ~/.bazelrc ~/.cache/bazel and reinstalling it but the error remains the same..

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

onnoeberhard commented 4 years ago

I don't really know what else to try.. I ran the following script:

rm -fr ~/.bazel ~/.bazelrc ~/.cache/bazel    # Uninstall bazel

# Install bazel
curl -LO https://github.com/bazelbuild/bazel/releases/download/0.24.1/bazel-0.24.1-installer-darwin-x86_64.sh
chmod +x bazel-0.24.1-installer-darwin-x86_64.sh
./bazel-0.24.1-installer-darwin-x86_64.sh --user
rm bazel-0.24.1-installer-darwin-x86_64.sh

# Clone tensorflow
git clone https://github.com/mozilla/tensorflow.git
cd tensorflow
git checkout origin/r1.15

bazel clean --expunge    # Whether or not I run this command, nothing changes

# Build libdeepspeech
ln -s ../DeepSpeech/native_client ./
bazel build --workspace_status_command="bash native_client/bazel_workspace_status_cmd.sh" --config=monolithic -c opt --copt=-O3 --copt="-D_GLIBCXX_USE_CXX11_ABI=0" --copt=-fvisibility=hidden //native_client:libdeepspeech.so

And I get the error message above..

reuben commented 4 years ago
  1. Are you sure you're running the bazel you installed and not a different copy of it that's on the system?
  2. I run that same command, also on macOS 10.15, and it works fine, so whatever it is, it's on your side.

Note the message about there being a cache hit for the protobuf package, indicating you're not properly cleaning the (correct) cache.

reuben commented 4 years ago

Also you forgot the configure step. For libdeepspeech alone I don't think it makes any difference but to be safe, do that.

onnoeberhard commented 4 years ago
  1. Yes I am sure, after the "uninstall" step (first line) bazel is not installed anymore, so runnning a bazel command returns zsh: command not found: bazel. After the installation, bazel version returns:
    Build label: 0.24.1
    Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
    Build time: Tue Apr 2 16:32:47 2019 (1554222767)
    Build timestamp: 1554222767
    Build timestamp as int: 1554222767

As for the configure script, I get another weird bug. When running ./configure, I get env: bash\r: No such file or directory, even though the file is in the directory (shows up in ls). As a workaround, I just ran python configure.py, which works just fine and the configure script doesn't seem to be doing anything else either. But the error when running the bazel command still does not change.

I don't know how to clean the cache bazel uses.. Apparently removing the ~/.cache/bazel directory did not do the trick, neither does specifying --repository_cache="" when running the command, though I don't know what exactly that option does. It just says in the documentary "An empty string as argument requests the cache to be disabled." However, the info message about the protobuf cache hit still appears.

reuben commented 4 years ago

I get env: bash\r: No such file or directory

Hm. The \r in this message, plus the ? symbols in the patch apply error message, plus the fact that the error is a patch failing to apply, seems to indicate there's something wrong with your Git config where it's writing bad newlines in the files it checks out. You should check if your git config doesn't include any bad newline settings.

onnoeberhard commented 4 years ago

You were right! I deleted my .gitconfig file and re-cloned both mozilla/DeepSpeech and mozilla/tensorflow; after that the ./configure command worked and the bazel build command succeeded without error. Thank you so much!

lock[bot] commented 4 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.