search

googleapis / google-cloud-python

Google Cloud Client Library for Python
https://googleapis.github.io/google-cloud-python/
Apache License 2.0
4.82k stars 1.52k forks source link

Synthesis failed for language #9181

Closed yoshi-automation closed 5 years ago

yoshi-automation commented 5 years ago

Hello! Autosynth couldn't regenerate language. :broken_heart:

Here's the output from running synth.py:

Cloning into 'working_repo'...
Switched to branch 'autosynth-language'
Running synthtool
['/tmpfs/src/git/autosynth/env/bin/python3', '-m', 'synthtool', 'synth.py', '--']
synthtool > Executing /tmpfs/src/git/autosynth/working_repo/language/synth.py.
synthtool > Ensuring dependencies.
synthtool > Pulling artman image.
latest: Pulling from googleapis/artman
Digest: sha256:0e6f3a668cd68afc768ecbe08817cf6e56a0e64fcbdb1c58c3b97492d12418a1
Status: Image is up to date for googleapis/artman:latest
synthtool > Cloning googleapis.
synthtool > Running generator for google/cloud/language/artman_language_v1beta2.yaml.
synthtool > Failed executing docker run --name artman-docker --rm -i -e HOST_USER_ID=1000 -e HOST_GROUP_ID=1000 -e RUNNING_IN_ARTMAN_DOCKER=True -v /home/kbuilder/.cache/synthtool/googleapis:/home/kbuilder/.cache/synthtool/googleapis -v /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles:/home/kbuilder/.cache/synthtool/googleapis/artman-genfiles -w /home/kbuilder/.cache/synthtool/googleapis googleapis/artman:latest /bin/bash -c artman --local --config google/cloud/language/artman_language_v1beta2.yaml --generator-args='--dev_samples' generate python_gapic:

artman> Final args:
artman>   api_name: language
artman>   api_version: v1beta2
artman>   artifact_type: GAPIC
artman>   aspect: ALL
artman>   gapic_code_dir: /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1beta2
artman>   gapic_yaml: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1beta2/language_gapic.yaml
artman>   generator_args: --dev_samples
artman>   import_proto_path:
artman>     - /home/kbuilder/.cache/synthtool/googleapis
artman>   language: python
artman>   organization_name: google-cloud
artman>   output_dir: /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles
artman>   proto_deps:
artman>     - name: google-common-protos
artman>   proto_package: ''
artman>   root_dir: /home/kbuilder/.cache/synthtool/googleapis
artman>   samples: ''
artman>   service_yaml: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/language_v1beta2.yaml
artman>   src_proto_path:
artman>     - /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1beta2
artman>   toolkit_path: /toolkit
artman>   
artman> Creating GapicClientPipeline.
artman.output >
WARNING: toplevel: (lint) control-presence: Service language.googleapis.com does not have control environment configured.
ERROR: toplevel: Unexpected exception:
java.lang.IllegalArgumentException: No available flattening for smoke test to use.
    at com.google.api.codegen.transformer.TestCaseTransformer.lambda$getSmokeTestFlatteningGroup$2(TestCaseTransformer.java:498)
    at java.util.Optional.orElseThrow(Optional.java:290)
    at com.google.api.codegen.transformer.TestCaseTransformer.getSmokeTestFlatteningGroup(TestCaseTransformer.java:497)
    at com.google.api.codegen.transformer.py.PythonPackageMetadataTransformer.generateExampleMethods(PythonPackageMetadataTransformer.java:312)
    at com.google.api.codegen.transformer.py.PythonPackageMetadataTransformer.generateMetadataView(PythonPackageMetadataTransformer.java:193)
    at com.google.api.codegen.transformer.py.PythonPackageMetadataTransformer.generateTopLevelFiles(PythonPackageMetadataTransformer.java:179)
    at com.google.api.codegen.transformer.py.PythonPackageMetadataTransformer.transform(PythonPackageMetadataTransformer.java:118)
    at com.google.api.codegen.transformer.py.PythonPackageMetadataTransformer.transform(PythonPackageMetadataTransformer.java:68)
    at com.google.api.codegen.gapic.GapicGenerator.generate(GapicGenerator.java:64)
    at com.google.api.codegen.gapic.GapicGeneratorApp.process(GapicGeneratorApp.java:246)
    at com.google.api.tools.framework.tools.GenericToolDriverBase.run(GenericToolDriverBase.java:90)
    at com.google.api.tools.framework.tools.ToolDriverBase.run(ToolDriverBase.java:73)
    at com.google.api.codegen.GeneratorMain.gapicGeneratorMain(GeneratorMain.java:338)
    at com.google.api.codegen.GeneratorMain.main(GeneratorMain.java:190)

artman> Traceback (most recent call last):
  File "/artman/artman/cli/main.py", line 72, in main
    engine.run()
  File "/usr/local/lib/python3.5/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run
    for _state in self.run_iter(timeout=timeout):
  File "/usr/local/lib/python3.5/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
    failure.Failure.reraise_if_any(er_failures)
  File "/usr/local/lib/python3.5/dist-packages/taskflow/types/failure.py", line 339, in reraise_if_any
    failures[0].reraise()
  File "/usr/local/lib/python3.5/dist-packages/taskflow/types/failure.py", line 346, in reraise
    six.reraise(*self._exc_info)
  File "/usr/local/lib/python3.5/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
    result = task.execute(**arguments)
  File "/artman/artman/tasks/gapic_tasks.py", line 146, in execute
    task_utils.gapic_gen_task(toolkit_path, [gapic_artifact] + args))
  File "/artman/artman/tasks/task_base.py", line 64, in exec_command
    raise e
  File "/artman/artman/tasks/task_base.py", line 56, in exec_command
    output = subprocess.check_output(args, stderr=subprocess.STDOUT)
  File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.5/subprocess.py", line 708, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['java', '-cp', '/toolkit/build/libs/gapic-generator-latest-fatjar.jar', 'com.google.api.codegen.GeneratorMain', 'LEGACY_GAPIC_AND_PACKAGE', '--descriptor_set=/home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/google-cloud-language-v1beta2_updated_py_docs.desc', '--package_yaml2=/home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python_google-cloud-language-v1beta2_package2.yaml', '--output=/home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1beta2', '--language=python', '--service_yaml=/home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/language_v1beta2.yaml', '--gapic_yaml=/home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1beta2/language_gapic.yaml', '--dev_samples']' returned non-zero exit status 1

Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/__main__.py", line 87, in <module>
    main()
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/__main__.py", line 79, in main
    spec.loader.exec_module(synth_module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/tmpfs/src/git/autosynth/working_repo/language/synth.py", line 35, in <module>
    include_samples=True
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/gcp/gapic_generator.py", line 50, in py_library
    return self._generate_code(service, version, "python", **kwargs)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/gcp/gapic_generator.py", line 138, in _generate_code
    generator_args=generator_args,
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/gcp/artman.py", line 141, in run
    shell.run(cmd, cwd=root_dir)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/shell.py", line 39, in run
    raise exc
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/shell.py", line 33, in run
    encoding="utf-8",
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['docker', 'run', '--name', 'artman-docker', '--rm', '-i', '-e', 'HOST_USER_ID=1000', '-e', 'HOST_GROUP_ID=1000', '-e', 'RUNNING_IN_ARTMAN_DOCKER=True', '-v', '/home/kbuilder/.cache/synthtool/googleapis:/home/kbuilder/.cache/synthtool/googleapis', '-v', '/home/kbuilder/.cache/synthtool/googleapis/artman-genfiles:/home/kbuilder/.cache/synthtool/googleapis/artman-genfiles', '-w', PosixPath('/home/kbuilder/.cache/synthtool/googleapis'), 'googleapis/artman:latest', '/bin/bash', '-c', "artman --local --config google/cloud/language/artman_language_v1beta2.yaml --generator-args='--dev_samples' generate python_gapic"]' returned non-zero exit status 32.
synthtool > Cleaned up 1 temporary directories.
synthtool > Wrote metadata to synth.metadata.

Synthesis failed

Google internal developers can see the full log here.

tseaver commented 5 years ago

@beccasaurus Like #9182, this one fails due to a missing gen-manifest in the artman image.

beccasaurus commented 5 years ago

Ah, we added sample-tester to the noxfile for running the sample test suite, but it's not installed by the time synth is run :'(

The gen-manifest command comes with sample-tester.

manifest contains sample metadata including command to run the sample (used by tests)

Can we add it to the Kokoro image?

... that said, I thought missing gen-manifest would print a warning but continue successfully?

gen-manifest is only relevant for projects that are tested, it should synthtool ok without.

misc we'll move this step into the GAPIC generator itself which'll remove gen-manifest.

        try:
            log.debug(f"Writing samples manifest {manifest_arguments}")
            shell.run(manifest_arguments, cwd=samples_root_dir)
        except subprocess.CalledProcessError:
            log.warning("gen-manifest failed (sample-tester may not be installed)")

https://github.com/googleapis/synthtool/blob/e4aa9bb2357527c9dd20ea462fe2f6e1ffcb8bf2/synthtool/gcp/gapic_generator.py#L327

beccasaurus commented 5 years ago 
WARNING: toplevel: (lint) control-presence: Service language.googleapis.com does not have control environment configured.
ERROR: toplevel: Unexpected exception:
java.lang.IllegalArgumentException: No available flattening for smoke test to use

Some recent changes made to the GAPIC config yesterday (as part of the migration to annotations): https://github.com/googleapis/googleapis/commit/541c015395812b3d4de53cc2141eecefd08ebed2

This seems related. This breaks on regular artman generation.

yoshi-automation commented 5 years ago

Autosynth is still having trouble generating language. :sob:

Here's the output from running synth.py:

Cloning into 'working_repo'...
Switched to branch 'autosynth-language'
Running synthtool
['/tmpfs/src/git/autosynth/env/bin/python3', '-m', 'synthtool', 'synth.py', '--']
synthtool > Executing /tmpfs/src/git/autosynth/working_repo/language/synth.py.
synthtool > Ensuring dependencies.
synthtool > Pulling artman image.
latest: Pulling from googleapis/artman
Digest: sha256:0e6f3a668cd68afc768ecbe08817cf6e56a0e64fcbdb1c58c3b97492d12418a1
Status: Image is up to date for googleapis/artman:latest
synthtool > Cloning googleapis.
synthtool > Running generator for google/cloud/language/artman_language_v1beta2.yaml.
synthtool > Generated code into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1beta2.
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1beta2/language_service.proto to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1beta2/google/cloud/language_v1beta2/proto/language_service.proto
synthtool > Placed proto files into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1beta2/google/cloud/language_v1beta2/proto.
synthtool > No files in sources /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1beta2/samples were copied. Does the source contain files?
synthtool > Running generator for google/cloud/language/artman_language_v1.yaml.
synthtool > Generated code into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1.
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/language_service.proto to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/google/cloud/language_v1/proto/language_service.proto
synthtool > Placed proto files into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/google/cloud/language_v1/proto.
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/analyzing_sentiment.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/analyzing_sentiment.test.yaml
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/analyzing_entity_sentiment.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/analyzing_entity_sentiment.test.yaml
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/analyzing_syntax.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/analyzing_syntax.test.yaml
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/analyzing_entities.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/analyzing_entities.test.yaml
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/classifying_content.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/classifying_content.test.yaml
synthtool > Writing samples manifest ['gen-manifest', '--env=python', '--bin=python3', '--output=v1/test/samples.manifest.yaml', '--chdir={@manifest_dir}/../..', 'v1/language_entity_sentiment_gcs.py', 'v1/language_classify_gcs.py', 'v1/language_syntax_gcs.py', 'v1/language_entities_text.py', 'v1/language_classify_text.py', 'v1/language_syntax_text.py', 'v1/language_entity_sentiment_text.py', 'v1/language_entities_gcs.py', 'v1/language_sentiment_gcs.py', 'v1/language_sentiment_text.py']
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/__main__.py", line 87, in <module>
    main()
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/__main__.py", line 79, in main
    spec.loader.exec_module(synth_module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/tmpfs/src/git/autosynth/working_repo/language/synth.py", line 35, in <module>
    include_samples=True
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/gcp/gapic_generator.py", line 50, in py_library
    return self._generate_code(service, version, "python", **kwargs)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/gcp/gapic_generator.py", line 174, in _generate_code
    self._include_samples(language, version, genfiles, googleapis_service_dir)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/gcp/gapic_generator.py", line 325, in _include_samples
    shell.run(manifest_arguments, cwd=samples_root_dir)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/shell.py", line 33, in run
    encoding="utf-8",
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/subprocess.py", line 403, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/subprocess.py", line 1326, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'gen-manifest'
synthtool > Cleaned up 1 temporary directories.
synthtool > Wrote metadata to synth.metadata.

Synthesis failed

Google internal developers can see the full log here.

yoshi-automation commented 5 years ago

Autosynth is still having trouble generating language. :sob:

Here's the output from running synth.py:

Cloning into 'working_repo'...
Switched to branch 'autosynth-language'
Running synthtool
['/tmpfs/src/git/autosynth/env/bin/python3', '-m', 'synthtool', 'synth.py', '--']
synthtool > Executing /tmpfs/src/git/autosynth/working_repo/language/synth.py.
synthtool > Ensuring dependencies.
synthtool > Pulling artman image.
latest: Pulling from googleapis/artman
Digest: sha256:0e6f3a668cd68afc768ecbe08817cf6e56a0e64fcbdb1c58c3b97492d12418a1
Status: Image is up to date for googleapis/artman:latest
synthtool > Cloning googleapis.
synthtool > Running generator for google/cloud/language/artman_language_v1beta2.yaml.
synthtool > Generated code into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1beta2.
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1beta2/language_service.proto to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1beta2/google/cloud/language_v1beta2/proto/language_service.proto
synthtool > Placed proto files into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1beta2/google/cloud/language_v1beta2/proto.
synthtool > No files in sources /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1beta2/samples were copied. Does the source contain files?
synthtool > Running generator for google/cloud/language/artman_language_v1.yaml.
synthtool > Generated code into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1.
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/language_service.proto to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/google/cloud/language_v1/proto/language_service.proto
synthtool > Placed proto files into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/google/cloud/language_v1/proto.
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/analyzing_sentiment.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/analyzing_sentiment.test.yaml
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/analyzing_entity_sentiment.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/analyzing_entity_sentiment.test.yaml
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/analyzing_syntax.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/analyzing_syntax.test.yaml
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/analyzing_entities.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/analyzing_entities.test.yaml
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/classifying_content.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/classifying_content.test.yaml
synthtool > Writing samples manifest ['gen-manifest', '--env=python', '--bin=python3', '--output=v1/test/samples.manifest.yaml', '--chdir={@manifest_dir}/../..', 'v1/language_entity_sentiment_gcs.py', 'v1/language_classify_gcs.py', 'v1/language_syntax_gcs.py', 'v1/language_entities_text.py', 'v1/language_classify_text.py', 'v1/language_syntax_text.py', 'v1/language_entity_sentiment_text.py', 'v1/language_entities_gcs.py', 'v1/language_sentiment_gcs.py', 'v1/language_sentiment_text.py']
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/__main__.py", line 87, in <module>
    main()
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/__main__.py", line 79, in main
    spec.loader.exec_module(synth_module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/tmpfs/src/git/autosynth/working_repo/language/synth.py", line 35, in <module>
    include_samples=True
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/gcp/gapic_generator.py", line 50, in py_library
    return self._generate_code(service, version, "python", **kwargs)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/gcp/gapic_generator.py", line 174, in _generate_code
    self._include_samples(language, version, genfiles, googleapis_service_dir)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/gcp/gapic_generator.py", line 325, in _include_samples
    shell.run(manifest_arguments, cwd=samples_root_dir)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/shell.py", line 33, in run
    encoding="utf-8",
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/subprocess.py", line 403, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/subprocess.py", line 1326, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'gen-manifest'
synthtool > Cleaned up 1 temporary directories.
synthtool > Wrote metadata to synth.metadata.

Synthesis failed

Google internal developers can see the full log here.

yoshi-automation commented 5 years ago

Autosynth is still having trouble generating language. :sob:

Here's the output from running synth.py:

Cloning into 'working_repo'...
Switched to branch 'autosynth-language'
Running synthtool
['/tmpfs/src/git/autosynth/env/bin/python3', '-m', 'synthtool', 'synth.py', '--']
synthtool > Executing /tmpfs/src/git/autosynth/working_repo/language/synth.py.
synthtool > Ensuring dependencies.
synthtool > Pulling artman image.
latest: Pulling from googleapis/artman
Digest: sha256:0e6f3a668cd68afc768ecbe08817cf6e56a0e64fcbdb1c58c3b97492d12418a1
Status: Image is up to date for googleapis/artman:latest
synthtool > Cloning googleapis.
synthtool > Running generator for google/cloud/language/artman_language_v1beta2.yaml.
synthtool > Generated code into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1beta2.
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1beta2/language_service.proto to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1beta2/google/cloud/language_v1beta2/proto/language_service.proto
synthtool > Placed proto files into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1beta2/google/cloud/language_v1beta2/proto.
synthtool > No files in sources /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1beta2/samples were copied. Does the source contain files?
synthtool > Running generator for google/cloud/language/artman_language_v1.yaml.
synthtool > Generated code into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1.
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/language_service.proto to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/google/cloud/language_v1/proto/language_service.proto
synthtool > Placed proto files into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/google/cloud/language_v1/proto.
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/analyzing_sentiment.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/analyzing_sentiment.test.yaml
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/analyzing_entity_sentiment.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/analyzing_entity_sentiment.test.yaml
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/analyzing_syntax.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/analyzing_syntax.test.yaml
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/analyzing_entities.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/analyzing_entities.test.yaml
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/classifying_content.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/classifying_content.test.yaml
synthtool > Writing samples manifest ['gen-manifest', '--env=python', '--bin=python3', '--output=v1/test/samples.manifest.yaml', '--chdir={@manifest_dir}/../..', 'v1/language_entity_sentiment_gcs.py', 'v1/language_classify_gcs.py', 'v1/language_syntax_gcs.py', 'v1/language_entities_text.py', 'v1/language_classify_text.py', 'v1/language_syntax_text.py', 'v1/language_entity_sentiment_text.py', 'v1/language_entities_gcs.py', 'v1/language_sentiment_gcs.py', 'v1/language_sentiment_text.py']
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/__main__.py", line 87, in <module>
    main()
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/__main__.py", line 79, in main
    spec.loader.exec_module(synth_module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/tmpfs/src/git/autosynth/working_repo/language/synth.py", line 35, in <module>
    include_samples=True
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/gcp/gapic_generator.py", line 50, in py_library
    return self._generate_code(service, version, "python", **kwargs)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/gcp/gapic_generator.py", line 174, in _generate_code
    self._include_samples(language, version, genfiles, googleapis_service_dir)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/gcp/gapic_generator.py", line 325, in _include_samples
    shell.run(manifest_arguments, cwd=samples_root_dir)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/shell.py", line 33, in run
    encoding="utf-8",
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/subprocess.py", line 403, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/subprocess.py", line 1326, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'gen-manifest'
synthtool > Cleaned up 1 temporary directories.
synthtool > Wrote metadata to synth.metadata.

Synthesis failed

Google internal developers can see the full log here.

yoshi-automation commented 5 years ago

Autosynth is still having trouble generating language. :sob:

Here's the output from running synth.py:

Cloning into 'working_repo'...
Switched to branch 'autosynth-language'
Running synthtool
['/tmpfs/src/git/autosynth/env/bin/python3', '-m', 'synthtool', 'synth.py', '--']
synthtool > Executing /tmpfs/src/git/autosynth/working_repo/language/synth.py.
synthtool > Ensuring dependencies.
synthtool > Pulling artman image.
latest: Pulling from googleapis/artman
Digest: sha256:0e6f3a668cd68afc768ecbe08817cf6e56a0e64fcbdb1c58c3b97492d12418a1
Status: Image is up to date for googleapis/artman:latest
synthtool > Cloning googleapis.
synthtool > Running generator for google/cloud/language/artman_language_v1beta2.yaml.
synthtool > Generated code into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1beta2.
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1beta2/language_service.proto to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1beta2/google/cloud/language_v1beta2/proto/language_service.proto
synthtool > Placed proto files into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1beta2/google/cloud/language_v1beta2/proto.
synthtool > No files in sources /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1beta2/samples were copied. Does the source contain files?
synthtool > Running generator for google/cloud/language/artman_language_v1.yaml.
synthtool > Generated code into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1.
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/language_service.proto to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/google/cloud/language_v1/proto/language_service.proto
synthtool > Placed proto files into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/google/cloud/language_v1/proto.
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/analyzing_sentiment.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/analyzing_sentiment.test.yaml
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/analyzing_entity_sentiment.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/analyzing_entity_sentiment.test.yaml
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/analyzing_syntax.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/analyzing_syntax.test.yaml
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/analyzing_entities.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/analyzing_entities.test.yaml
synthtool > Copy: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/language/v1/samples/test/classifying_content.test.yaml to /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/python/language-v1/samples/v1/test/classifying_content.test.yaml
synthtool > Writing samples manifest ['gen-manifest', '--env=python', '--bin=python3', '--output=v1/test/samples.manifest.yaml', '--chdir={@manifest_dir}/../..', 'v1/language_entity_sentiment_gcs.py', 'v1/language_classify_gcs.py', 'v1/language_syntax_gcs.py', 'v1/language_entities_text.py', 'v1/language_classify_text.py', 'v1/language_syntax_text.py', 'v1/language_entity_sentiment_text.py', 'v1/language_entities_gcs.py', 'v1/language_sentiment_gcs.py', 'v1/language_sentiment_text.py']
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/__main__.py", line 87, in <module>
    main()
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/__main__.py", line 79, in main
    spec.loader.exec_module(synth_module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/tmpfs/src/git/autosynth/working_repo/language/synth.py", line 35, in <module>
    include_samples=True
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/gcp/gapic_generator.py", line 50, in py_library
    return self._generate_code(service, version, "python", **kwargs)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/gcp/gapic_generator.py", line 174, in _generate_code
    self._include_samples(language, version, genfiles, googleapis_service_dir)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/gcp/gapic_generator.py", line 325, in _include_samples
    shell.run(manifest_arguments, cwd=samples_root_dir)
  File "/tmpfs/src/git/autosynth/env/lib/python3.6/site-packages/synthtool/shell.py", line 33, in run
    encoding="utf-8",
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/subprocess.py", line 403, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/subprocess.py", line 1326, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'gen-manifest'
synthtool > Cleaned up 1 temporary directories.
synthtool > Wrote metadata to synth.metadata.

Synthesis failed

Google internal developers can see the full log here.

busunkim96 commented 5 years ago

Ah, we added sample-tester to the noxfile for running the sample test suite, but it's not installed by the time synth is run :'(

The gen-manifest command comes with sample-tester.

manifest contains sample metadata including command to run the sample (used by tests)

Can we add it to the Kokoro image?

Definitely. See Gerrit PR.

... that said, I thought missing gen-manifest would print a warning but continue successfully?

When I run synth locally without sample-tester installed and I get the same error: FileNotFoundError: [Errno 2] No such file or directory: 'gen-manifest': 'gen-manifest'. I opened https://github.com/googleapis/synthtool/pull/286.

yoshi-automation commented 5 years ago

Autosynth passed, closing! :green_heart: