search

googleapis / synthtool

Apache License 2.0
87 stars 84 forks source link

feat: remove java_library templates #1923

Open diegomarquezp opened 7 months ago

diegomarquezp commented 7 months ago

Our transferred owlbot implementation will contain the java templates currently contained in synthtool. This branch (more specifically, its HEAD commit) will be used by our hermetic build scripts.

I'm planning to leave this as an alternative branch used by the hermetic build scripts in sdk-platform-java since merging it will likely break HW libraries or the monorepo. Once our hermetic build scripts are live, we will merge this PR

Testing locally

Building the image

[hi on] diegomarquezp@diegomarquezp:~/Desktop/synthtool$ git checkout no-java-templates
M   docker/owlbot/java/Dockerfile
Already on 'no-java-templates'
Your branch is up to date with 'origin/no-java-templates'.
[hi on] diegomarquezp@diegomarquezp:~/Desktop/synthtool$ git diff
diff --git a/docker/owlbot/java/Dockerfile b/docker/owlbot/java/Dockerfile
index 97d3e70..dc13636 100644
--- a/docker/owlbot/java/Dockerfile
+++ b/docker/owlbot/java/Dockerfile
@@ -26,7 +26,7 @@ RUN apt-get install -y --no-install-recommends jq
 COPY docker/owlbot/java/bin /owlbot/bin
 COPY docker/owlbot/java/src /owlbot/src
 COPY docker/owlbot/java/templates /owlbot/templates
-COPY post-processor-changes.txt /post-processor-changes.txt
+# COPY post-processor-changes.txt /post-processor-changes.txt
 RUN cd /owlbot/src && \
     python3 -m pip install --require-hashes -r requirements.txt
 ADD https://repo1.maven.org/maven2/com/google/googlejavaformat/google-java-format/${JAVA_FORMAT_VERSION}/google-java-format-${JAVA_FORMAT_VERSION}-all-deps.jar /owlbot/google-java-format.jar
[hi on] diegomarquezp@diegomarquezp:~/Desktop/synthtool$ docker build -f docker/owlbot/java/Dockerfile .
Sending build context to Docker daemon  35.35MB
Step 1/16 : FROM gcr.io/cloud-devrel-public-resources/java8
 ---> 762b049398f8
Step 2/16 : ARG JAVA_FORMAT_VERSION=1.7
 ---> Using cache
 ---> 9188ceb74614
Step 3/16 : RUN apt-get install -y --no-install-recommends jq
 ---> Using cache
 ---> 4ecd3a9a9de2
Step 4/16 : COPY docker/owlbot/java/bin /owlbot/bin
 ---> Using cache
 ---> 43e4d96f1ca8
Step 5/16 : COPY docker/owlbot/java/src /owlbot/src
 ---> Using cache
 ---> fb77bef7d4a3
Step 6/16 : COPY docker/owlbot/java/templates /owlbot/templates
 ---> Using cache
 ---> 1795a62bab7e
Step 7/16 : RUN cd /owlbot/src &&     python3 -m pip install --require-hashes -r requirements.txt
 ---> Using cache
 ---> aa1cff92677b
Step 8/16 : ADD https://repo1.maven.org/maven2/com/google/googlejavaformat/google-java-format/${JAVA_FORMAT_VERSION}/google-java-format-${JAVA_FORMAT_VERSION}-all-deps.jar /owlbot/google-java-format.jar
Downloading [==================================================>]  6.094MB/6.094MB

 ---> Using cache
 ---> 0a0472b3eca8
Step 9/16 : COPY . /synthtool/
 ---> e8d78cf02d2d
Step 10/16 : WORKDIR /synthtool
 ---> Running in c8f03171499d
Removing intermediate container c8f03171499d
 ---> 120fbd107faa
Step 11/16 : RUN python3 -m pip install --no-deps -e .
 ---> Running in 4226efc2e12d
Obtaining file:///synthtool
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Checking if build backend supports build_editable: started
  Checking if build backend supports build_editable: finished with status 'done'
  Getting requirements to build editable: started
  Getting requirements to build editable: finished with status 'done'
  Preparing editable metadata (pyproject.toml): started
  Preparing editable metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: gcp-synthtool
  Building editable for gcp-synthtool (pyproject.toml): started
  Building editable for gcp-synthtool (pyproject.toml): finished with status 'done'
  Created wheel for gcp-synthtool: filename=gcp_synthtool-2020.2.4-0.editable-py3-none-any.whl size=7371 sha256=72771126095ffd9d7eaac33c7b90dcd8556906d1194c807ebceb9555642f90be
  Stored in directory: /tmp/pip-ephem-wheel-cache-q3vuoxki/wheels/a9/0d/7d/1c0a3332e149460011905abc942250bc681632f29491c02844
Successfully built gcp-synthtool
Installing collected packages: gcp-synthtool
Successfully installed gcp-synthtool-2020.2.4
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.3.2 -> 24.0
[notice] To update, run: pip install --upgrade pip
Removing intermediate container 4226efc2e12d
 ---> 43e84670f727
Step 12/16 : RUN python3 -m pip install --require-hashes -r docker/owlbot/java/src/requirements.txt
 ---> Running in 2fdee65f586e
Requirement already satisfied: attrs==23.1.0 in /root/.pyenv/versions/3.9.13/lib/python3.9/site-packages (from -r docker/owlbot/java/src/requirements.txt (line 7)) (23.1.0)
Requirement already satisfied: certifi==2023.7.22 in /root/.pyenv/versions/3.9.13/lib/python3.9/site-packages (from -r docker/owlbot/java/src/requirements.txt (line 11)) (2023.7.22)
Requirement already satisfied: charset-normalizer==3.2.0 in /root/.pyenv/versions/3.9.13/lib/python3.9/site-packages (from -r docker/owlbot/java/src/requirements.txt (line 15)) (3.2.0)
Requirement already satisfied: click==8.1.4 in /root/.pyenv/versions/3.9.13/lib/python3.9/site-packages (from -r docker/owlbot/java/src/requirements.txt (line 92)) (8.1.4)
Requirement already satisfied: colorlog==6.7.0 in /root/.pyenv/versions/3.9.13/lib/python3.9/site-packages (from -r docker/owlbot/java/src/requirements.txt (line 96)) (6.7.0)
Requirement already satisfied: idna==3.4 in /root/.pyenv/versions/3.9.13/lib/python3.9/site-packages (from -r docker/owlbot/java/src/requirements.txt (line 100)) (3.4)
Requirement already satisfied: jinja2==3.1.2 in /root/.pyenv/versions/3.9.13/lib/python3.9/site-packages (from -r docker/owlbot/java/src/requirements.txt (line 104)) (3.1.2)
Requirement already satisfied: lxml==4.9.3 in /root/.pyenv/versions/3.9.13/lib/python3.9/site-packages (from -r docker/owlbot/java/src/requirements.txt (line 108)) (4.9.3)
Requirement already satisfied: markupsafe==2.1.3 in /root/.pyenv/versions/3.9.13/lib/python3.9/site-packages (from -r docker/owlbot/java/src/requirements.txt (line 202)) (2.1.3)
Requirement already satisfied: protobuf==4.23.4 in /root/.pyenv/versions/3.9.13/lib/python3.9/site-packages (from -r docker/owlbot/java/src/requirements.txt (line 256)) (4.23.4)
Requirement already satisfied: pyyaml==6.0 in /root/.pyenv/versions/3.9.13/lib/python3.9/site-packages (from -r docker/owlbot/java/src/requirements.txt (line 271)) (6.0)
Requirement already satisfied: requests==2.31.0 in /root/.pyenv/versions/3.9.13/lib/python3.9/site-packages (from -r docker/owlbot/java/src/requirements.txt (line 313)) (2.31.0)
Requirement already satisfied: typing==3.7.4.3 in /root/.pyenv/versions/3.9.13/lib/python3.9/site-packages (from -r docker/owlbot/java/src/requirements.txt (line 317)) (3.7.4.3)
Requirement already satisfied: urllib3==2.0.7 in /root/.pyenv/versions/3.9.13/lib/python3.9/site-packages (from -r docker/owlbot/java/src/requirements.txt (line 321)) (2.0.7)
Requirement already satisfied: watchdog==3.0.0 in /root/.pyenv/versions/3.9.13/lib/python3.9/site-packages (from -r docker/owlbot/java/src/requirements.txt (line 325)) (3.0.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.3.2 -> 24.0
[notice] To update, run: pip install --upgrade pip
Removing intermediate container 2fdee65f586e
 ---> 04cd77e21da2
Step 13/16 : RUN chmod +rx /root/ /root/.pyenv && chmod +r /owlbot/google-java-format.jar
 ---> Running in 6ed3482cc246
Removing intermediate container 6ed3482cc246
 ---> 31163fe6adc6
Step 14/16 : ENV SYNTHTOOL_TEMPLATES="/synthtool/synthtool/gcp/templates"     PYTHON_PATH="/owlbot/src"
 ---> Running in a46b0660f1f8
Removing intermediate container a46b0660f1f8
 ---> 0d63bc6f60ea
Step 15/16 : WORKDIR /workspace
 ---> Running in 2c5a22ce3898
Removing intermediate container 2c5a22ce3898
 ---> 024baa46e830
Step 16/16 : CMD [ "/owlbot/bin/entrypoint.sh" ]
 ---> Running in 5c8f19557668
Removing intermediate container 5c8f19557668
 ---> 101ee293f1c8
Successfully built 101ee293f1c8

Testing in java-bigtable

[hi on] diegomarquezp@diegomarquezp:~/Desktop/synthtool$ cd ../java-bigtable
[hi on] diegomarquezp@diegomarquezp:~/Desktop/java-bigtable$ git pull origin main
From https://github.com/googleapis/java-bigtable
 * branch              main       -> FETCH_HEAD
Already up to date.
[hi on] diegomarquezp@diegomarquezp:~/Desktop/java-bigtable$ docker run --rm -v $(pwd):/workspace 101ee293f1c8
ls: cannot access '*/.OwlBot.yaml': No such file or directory
Processing a split repo
Generating templates and retrieving files from owl-bot-staging directory...
2024-02-06 19:13:17,404 synthtool [CRITICAL] > You are running the synthesis script directly, this will be disabled in a future release of Synthtool. Please use python3 -m synthtool instead.
2024-02-06 19:13:17,784 synthtool [INFO] > monorepo: False, split_repo: True, library_type: GAPIC_COMBO, repo_short: java-bigtable, migrated_split_repo: False
2024-02-06 19:13:17,784 synthtool [DEBUG] > Using local templates at /synthtool/synthtool/gcp/templates
.github/workflows/ci.yaml
.github/workflows/renovate_config_check.yaml
.kokoro/requirements.txt
renovate.json
...done
Generating missing pom.xml...
/owlbot
working directory: /workspace
monorepo? False
updating modules in cloud pom.xml
updating modules in bom pom.xml
updating modules in parent pom.xml
updating modules in ./versions.txt
...done
Generating clirr-ignored-differences.xml...
...done
Fixing missing license headers...
2024-02-06 19:13:18,392 synthtool [CRITICAL] > You are running the synthesis script directly, this will be disabled in a future release of Synthtool. Please use python3 -m synthtool instead.
2024-02-06 19:13:18,574 synthtool [WARNING] > No replacements made in <generator object _filter_no_header at 0x7fdac3686510> for pattern // Generated by the protocol buffer compiler.  DO NOT EDIT!, maybe replacement is no longer needed?
2024-02-06 19:13:18,578 synthtool [WARNING] > No replacements made in [PosixPath('/workspace/proto-google-cloud-bigtable-v2/src/**/*Name.java'), PosixPath('/workspace/proto-google-cloud-bigtable-v2/src/**/*Names.java')] for pattern /\*
 \* Copyright \d{4} Google LLC
 \*
 \* Licensed under the Apache License, Version 2.0 \(the "License"\); you may not use this file except
 \* in compliance with the License. You may obtain a copy of the License at
 \*
 \* http://www.apache.org/licenses/LICENSE-2.0
 \*
 \* Unless required by applicable law or agreed to in writing, software distributed under the License
 \* is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
 \* or implied. See the License for the specific language governing permissions and limitations under
 \* the License.
 \*/
, maybe replacement is no longer needed?
2024-02-06 19:13:18,595 synthtool [WARNING] > No replacements made in <generator object _filter_no_header at 0x7fdac3686510> for pattern // Generated by the protocol buffer compiler.  DO NOT EDIT!, maybe replacement is no longer needed?
2024-02-06 19:13:18,598 synthtool [WARNING] > No replacements made in [PosixPath('/workspace/proto-google-cloud-bigtable-admin-v2/src/**/*Name.java'), PosixPath('/workspace/proto-google-cloud-bigtable-admin-v2/src/**/*Names.java')] for pattern /\*
 \* Copyright \d{4} Google LLC
 \*
 \* Licensed under the Apache License, Version 2.0 \(the "License"\); you may not use this file except
 \* in compliance with the License. You may obtain a copy of the License at
 \*
 \* http://www.apache.org/licenses/LICENSE-2.0
 \*
 \* Unless required by applicable law or agreed to in writing, software distributed under the License
 \* is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
 \* or implied. See the License for the specific language governing permissions and limitations under
 \* the License.
 \*/
, maybe replacement is no longer needed?
2024-02-06 19:13:18,599 synthtool [WARNING] > No replacements made in <generator object _filter_no_header at 0x7fdac3686510> for pattern ^package (.*);, maybe replacement is no longer needed?
2024-02-06 19:13:18,601 synthtool [WARNING] > No replacements made in <generator object _filter_no_header at 0x7fdac3686510> for pattern ^package (.*);, maybe replacement is no longer needed?
...done
Reformatting source...
...done
[hi on] diegomarquezp@diegomarquezp:~/Desktop/java-bigtable$ git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

Testing in java-storage

[hi on] diegomarquezp@diegomarquezp:~/Desktop/java-bigtable$ cd ../java-storage/
[hi on] diegomarquezp@diegomarquezp:~/Desktop/java-storage$ git checkout main
Already on 'main'
Your branch is up to date with 'origin/main'.
[hi on] diegomarquezp@diegomarquezp:~/Desktop/java-storage$ git pull
Already up to date.
[hi on] diegomarquezp@diegomarquezp:~/Desktop/java-storage$ docker run --rm -v $(pwd):/workspace 101ee293f1c8
ls: cannot access '*/.OwlBot.yaml': No such file or directory
Processing a split repo
Generating templates and retrieving files from owl-bot-staging directory...
2024-02-06 19:15:31,786 synthtool [CRITICAL] > You are running the synthesis script directly, this will be disabled in a future release of Synthtool. Please use python3 -m synthtool instead.
2024-02-06 19:15:32,189 synthtool [INFO] > monorepo: False, split_repo: True, library_type: GAPIC_COMBO, repo_short: java-storage, migrated_split_repo: False
2024-02-06 19:15:32,189 synthtool [DEBUG] > Using local templates at /synthtool/synthtool/gcp/templates
.github/workflows/ci.yaml
.github/workflows/renovate_config_check.yaml
.kokoro/requirements.txt
renovate.json
...done
Generating missing pom.xml...
/owlbot
working directory: /workspace
monorepo? False
updating modules in cloud pom.xml
updating modules in bom pom.xml
updating modules in parent pom.xml
updating modules in ./versions.txt
...done
Generating clirr-ignored-differences.xml...
...done
Fixing missing license headers...
2024-02-06 19:15:32,772 synthtool [CRITICAL] > You are running the synthesis script directly, this will be disabled in a future release of Synthtool. Please use python3 -m synthtool instead.
2024-02-06 19:15:32,934 synthtool [WARNING] > No replacements made in <generator object _filter_no_header at 0x7f052b025510> for pattern // Generated by the protocol buffer compiler.  DO NOT EDIT!, maybe replacement is no longer needed?
2024-02-06 19:15:32,938 synthtool [WARNING] > No replacements made in [PosixPath('/workspace/proto-google-cloud-storage-control-v2/src/**/*Name.java'), PosixPath('/workspace/proto-google-cloud-storage-control-v2/src/**/*Names.java')] for pattern /\*
 \* Copyright \d{4} Google LLC
 \*
 \* Licensed under the Apache License, Version 2.0 \(the "License"\); you may not use this file except
 \* in compliance with the License. You may obtain a copy of the License at
 \*
 \* http://www.apache.org/licenses/LICENSE-2.0
 \*
 \* Unless required by applicable law or agreed to in writing, software distributed under the License
 \* is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
 \* or implied. See the License for the specific language governing permissions and limitations under
 \* the License.
 \*/
, maybe replacement is no longer needed?
2024-02-06 19:15:32,946 synthtool [WARNING] > No replacements made in <generator object _filter_no_header at 0x7f052b025510> for pattern // Generated by the protocol buffer compiler.  DO NOT EDIT!, maybe replacement is no longer needed?
2024-02-06 19:15:32,948 synthtool [WARNING] > No replacements made in [PosixPath('/workspace/proto-google-cloud-storage-v2/src/**/*Name.java'), PosixPath('/workspace/proto-google-cloud-storage-v2/src/**/*Names.java')] for pattern /\*
 \* Copyright \d{4} Google LLC
 \*
 \* Licensed under the Apache License, Version 2.0 \(the "License"\); you may not use this file except
 \* in compliance with the License. You may obtain a copy of the License at
 \*
 \* http://www.apache.org/licenses/LICENSE-2.0
 \*
 \* Unless required by applicable law or agreed to in writing, software distributed under the License
 \* is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
 \* or implied. See the License for the specific language governing permissions and limitations under
 \* the License.
 \*/
, maybe replacement is no longer needed?
2024-02-06 19:15:32,949 synthtool [WARNING] > No replacements made in <generator object _filter_no_header at 0x7f052b025510> for pattern ^package (.*);, maybe replacement is no longer needed?
2024-02-06 19:15:32,950 synthtool [WARNING] > No replacements made in <generator object _filter_no_header at 0x7f052b025510> for pattern ^package (.*);, maybe replacement is no longer needed?
...done
Reformatting source...
...done
[hi on] diegomarquezp@diegomarquezp:~/Desktop/java-storage$ git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean
conventional-commit-lint-gcf[bot] commented 7 months ago

🤖 I detect that the PR title and the commit message differ and there's only one commit. To use the PR title for the commit history, you can use Github's automerge feature with squashing, or use automerge label. Good luck human!

-- conventional-commit-lint bot https://conventionalcommits.org/

suztomo commented 7 months ago

@diegomarquezp Can you build (docker build) owlbot postprocessor with this pull request and run the container against java-storage and java-bigtable to see everything is fine? Add observation in the pull request description.

diegomarquezp commented 7 months ago

@suztomo At a glance, this will affect the repos depending on synthtool (maybe java-bigtable will complain about missing files). I was planning to leave this branch as parallel to the main branch until we roll out the new postprocessor.

That said, I'll give a try to using the postprocessor with this image

diegomarquezp commented 7 months ago

@suztomo I added observations of a local run of this image against java-bigtable and java-storage. They look OK.

suztomo commented 7 months ago

Nice. What needs to happen before merging this?

suztomo commented 7 months ago

Once our hermetic build scripts are live, we will merge this PR

I just read this. Wait for the hermetic build script then.

Memo to myself: ensure no repositories, including ones outside 2-week release cycle, depend on the old container image.