Closed nicknezis closed 2 years ago
I decided to remove the Travis CI special settings because some of them were outdated settings which were needed to use a modern C++ compiler. We are now on a more modern version of Ubuntu, so it's not needed anymore. Also With the other .bazelrc
cleanup, I thought it would be a good opportunity to clean it up.
One of the benefits I see is that the build completed in 1 hr 45 min. This is down from the nearly 3 hour long builds. I'm not sure what the difference was. One theory is maybe it was due to us re-running the style check on each stage of the process (build, non-flaky unit test, flaky unit test). Another theory is that by limiting the memory used and also the jobs to 25, maybe that slowed things down. But I'm taking this opportunity to test TravisCI and it seems like it's working.
One of the benefits I see is that the build completed in 1 hr 45 min.
Yes, this is awesome! 🥳
2. Default is
fastbuild
, but you can add-c opt
to build optimized build.
We should document this in the build script as well as the online documentation?
@joshfischer1108 has this alleviated your build issues on Apple Silicone?
We should document this in the build script as well as the online documentation?
Updated the Compiling overview, compiling linux and compiling MacOS pages in the docs. Also found some other issues with old version of Bazel being referenced (0.26). I updated our docs to use Bazelisk, which automatically manages the verzion of Bazel. This will save us from having to update the docs each time we bump the version. This was especially annoying each time because of the old versioned docs in the repo that should not be updated.
Some other bazel.rc
related cleanup. I removed the tools/docker/bazel.rc
file because it was no longer needed. It was limiting the cpu/memory of Bazel when running in Docker. This was primarily due to an issue with JDK locking up due to memory bloat. This is no longer an issue with JDK 11, so we don't need it anymore. Removed the file and removed references to it in the Docker scripts.
Here is an example linked off of the Bazel issue in which they document the fact that JDK 11 resolved many Docker related issues. https://github.com/bazelbuild/bazel/issues/6592
I also removed mention of Applatix
CI scripts. Applatix was a Kubernetes related hosting platform which was acquired by Intuit in 2018. They seemed to be a copy of our other CI scripts, but were not used by Heron and were not maintained. So they are now purged.
I was just looking through the build logs and I am not sure if I am interpreting what I am seeing correctly but the only set of tests that seem to be running are the integration tests.
Are the "regular"/non-integration battery of tests still run? I may have missed the output in the logs. I remember heron test
taking at least 30mins before and logs indicate heron test flaky/non-flaky
are taking a combined ~6mins.
Are the "regular"/non-integration battery of tests still run? I may have missed the output in the logs. I remember
heron test
taking at least 30mins before and logs indicateheron test flaky/non-flaky
are taking a combined ~6mins.
Interesting. I see them mentioned, but it's hard to tell with the way to log output is hidden away. I'll do some local testing.
I think everything is okay with the tests. I did not see anything in the diff which would turn it off and this PR run log has the same sort of output without bazel test
being specifically noted.
Great work on this PR, there are some really important changes. I know some of these changes are cascading and build on/require each other but it would be better to introduce them as multiple small PRs. This way if there is an issue with some of the changes, we can revert the culprits without losing all changes.
I would get @nwangtw to once over and approve the changes before merging. It looks good to me. I have a few comments/questions - none of which would hold anything up on my end.
It would be nice if @joshfischer1108, or someone else with an Apple Silicone machine, could let us know if they are encountering any other build issues.
# BigSur-11.6.5 (20G527)
% clang -v
Apple clang version 12.0.5 (clang-1205.0.22.11)
Target: x86_64-apple-darwin20.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
% ./docker/scripts/test-unittest.sh darwin 0.20.5
================================================================================
INFO: Elapsed time: 954.681s, Critical Path: 305.35s
INFO: 3328 processes: 1144 internal, 2184 local.
INFO: Build completed successfully, 3328 total actions
Test cases: finished with 1189 passing and 0 failing out of 1189 test cases
Executed 247 out of 247 tests: 247 tests pass.
INFO: Build completed successfully, 3328 total actions
Cleaning up scratch dir
Just tested on the ubuntu22.04, and found that,
If docker version is lower than (<20.10.10), the apt update
will fail.
Should we mention in somewhere or in doc to remind people to upgrade?
@windhamwong I added a comment above the apt update
for both the compile
Dockerfile.ubuntu22.04
and the dist
Dockerfile.dist.ubuntu22.04
.
% ./docker/scripts/test-unittest.sh centos7 0.20.5
================================================================================
INFO: Elapsed time: 1583.309s, Critical Path: 234.97s
INFO: 3608 processes: 1160 internal, 2448 local.
INFO: Build completed successfully, 3608 total actions
Test cases: finished with 1189 passing and 0 failing out of 1189 test cases
Executed 247 out of 247 tests: 247 tests pass.
INFO: Build completed successfully, 3608 total actions
Cleaning up scratch dir
-ADD bazelrc /root/.bazelrc
./docker/scripts/test-unittest.sh rocky8 0.20.5
================================================================================
INFO: Elapsed time: 1673.469s, Critical Path: 259.39s
INFO: 3608 processes: 1160 internal, 2448 local.
INFO: Build completed successfully, 3608 total actions
Test cases: finished with 1189 passing and 0 failing out of 1189 test cases
Executed 247 out of 247 tests: 247 tests pass.
INFO: Build completed successfully, 3608 total actions
Cleaning up scratch dir
+1
Hmm I don't have a Reviewer, so I don't have an Approved button.
+1
Hmm I don't have a Reviewer, so I don't have an Approved button.
We need to get you added to the Apache Foundation Github Org as a contributor. Have you done the necessary steps to get this enabled after you were given committer access to the repo?
Great work on this PR, there are some really important changes. I know some of these changes are cascading and build on/require each other but it would be better to introduce them as multiple small PRs. This way if there is an issue with some of the changes, we can revert the culprits without losing all changes.
I would get @nwangtw to once over and approve the changes before merging. It looks good to me. I have a few comments/questions - none of which would hold anything up on my end.
It would be nice if @joshfischer1108, or someone else with an Apple Silicone machine, could let us know if they are encountering any other build issues.
I will attempt to build this weekend.
I will attempt to build this weekend.
Thank you, it would be great to know if all is well on Apple Silicone. Choi Se confirmed everything is building on Apple x86_64.
We need to get you added to the Apache Foundation Github Org as a contributor. Have you done the necessary steps to get this enabled after you were given committer access to the repo?
Let us get this set up so he can merge PRs.
I think this is now ready to merge. For the Apple M1 support. I think this currently won't yet support it. But primarily because we need to add some missing ARM dependencies. For example the helm
binary is currently x86
. But we can add M1 support in a future PR. This PR definitely gets us much closer to being able to support it.
I think this is now ready to merge.
Completely agree and Apple Silicon support should not block this. It has a lot of very broad and required changes.
Resolved a silly merge conflict in the WORKSPACE
related to extra whitespace.
Some build script cleanup leveraging the Bazel platforms feature.
https://github.com/aosp-riscv/platform-build-bazel
Context below:
While working through a bug we ran into with MacOS failing to compile, I got a better understanding of some of the Bazel settings. In another thread with Josh, who is compiling on the Apple M1 for the first time, he ran into issues with Linux vs OSX.
It dawned on me that we were explicitly setting the config to modify a number of settings. Some of them include the following items:
In a recent PR, I collapsed all of the Linux flavors to a standard
--config=linux
. This was born out of my frustration with the build scripts not jiving with Bazel standards. But it was not actually solving the problem. I'm hoping that this PR will help resolve some of the issues.Changes:
--config
. Bazel will detect the OS automatically.fastbuild
, but you can add-c opt
to build optimized build. a. You can even build both side by side with Bazel making a folder like darwin-fastbuild and darwin-opt3. If you want to run stylecheck, you can add--config=stylecheck
tools/platform/BUILD
tools/platform/BUILD
6. Scripts outside of Bazel were updated to remove--config
.-c opt
or--config=stylecheck
when needed.