Install overview might be missing some things.

gedalia commented 1 year ago

Environment

OS Version: Ubuntu 22.04
Source or binary build? Top of tree

Description

Expected behavior: build would go a bit better
Actual behavior: not so much

Steps to reproduce

Hi I started trying to build this repro from scratch and hit a bunch of errors, wasn't sure if things had just gotten stale or if ubuntu 22.04 was behaving unexpectedly, I tried this on 2 different machines and hit slightly different issues.

Things I might add to the readme: pip3 install -U vcstool or sudo pip install -U vcstool
It seems not super obvious where this repo goes. I wound up doing git clone https://github.com/gazebosim/gz-bazel.git inside of the gz folder, and then having to rename it to bazel.
gz/sdformat/BUILD.bazel is currently failing for me : needed load("@gz//bazel/lint:lint.bzl", "add_lint_tests") load("@gz//bazel/skylark:gz_py.bzl", "gz_py_binary") and I had to removed them from load( "@gz//bazel/skylark:build_defs.bzl"
failed to compile gz/msgs/src/Generator.cc needed to change: for (const auto name : ns) to: for (const auto & name : ns)

The package install script seemed to fail for me on dart, wound up using a shell script:

packages=(
  ./sdformat/.github/ci/packages.apt
....
)
# Iterate through the package lists and install packages
for package_list in "${packages[@]}"; do
  sudo apt-get install -y $(cat "$package_list")
done

I still needed to install many packages, not sure how these got missed: sudo apt-get install libtinyxml2-dev libgts-dev libfreeimage3 libfreeimage-dev libtinyxml2 libtinyxml2-dev libavcodec-dev ffmpeg libavformat-dev libgdal-dev libswscale-dev libfmt-dev libyaml-dev freeglut3-dev libzmq3-dev libzip-dev libjsoncpp-dev Needed for dart example sudo apt-get install libopenscenegraph-dev openscenegraph

*had to build and install nlopt, it then wound up in /usr/local/include and local lib. the symlink in the bazel externals cache was then invalid.

Thanks for providing this starting point getting gazebo to run with bazel, hopefully these bits might make it easier for the next person to build it.

mjcarroll commented 1 year ago

Hey, I'm in the process of updating the bazel build across the board for garden and harmonic. I will update the instructions to incorporate your suggestions here.

gedalia commented 1 year ago

I now have things building but I'm a bit confused. I'm a bit new to building gazebo and I'm trying to grok the overall purpose of this repo, I was expecting a bunch of top level bazel targets that would establish header file paths and library linking info, I'm gradually realizing that this repo seems independent of the core gazebo sim code in https://github.com/gazebosim/gz-sim. I've got a number of gazebo plugins that are building with bazel that I'm trying to upgrade from classic to ignition, but I'm realizing that those headers seem to be in gz-sim. My apologies if this isn't the best forum for questions wasn't sure where else to ask.

mjcarroll commented 1 year ago

This is the correct place.

When we moved from gazebo classic to new gazebo, part of the work was to break the monolithic code of gz-sim into the component pieces that could be reused in different places. To maintain that while also supporting bazel, this repo was created, as well as BUILD.bazel files were added to each of the libraries.

So how this should look

your_ws/
   WORKSPACE
   BUILD
   bazel
   math
   common
   ...
   sim

The full list of repositories and their branches are kept in the example/bazel.repos file. At this point for garden, this is supporting up to transport/fuel_tools. There is a PR here: https://github.com/gazebosim/gz-bazel/pull/61 that adds sensors, rendering, and gui.

There needs to be one more PR to add sim support. I have it working locally but still need to add a PR.

Once your workspace is checked out and build with bazel, you should be able to do something like the following.

bazel build //transport
bazel build //sim

bazel test //msgs/...

The packages.apt mechanism exists so we can just re-use the package dependencies from upstream Ubuntu. It would have been too tedious to provide BAZEL build files for each of our dependencies, as Gazebo has quite a few.

I haven't considered an example of building plugins with bazel, but I could also add that for you if it helps.

gedalia commented 1 year ago

Thanks for the info, away on vacation this week, but I'll take a look at that PR.

bd-mdelasa commented 1 year ago

@mjcarroll thanks for the context/details.

Out of curiosity, is there a reason you've opted to managed repos/branches with vcs instead of bazel's built-in mechanisms (e.g., git_repository, etc.) for doing the same? We've found the ability to manage many repos as externals to be very powerful.

In terms of landing https://github.com/gazebosim/gz-bazel/pull/61 and supporting a full bazel-based build of gz-sim, how far do you estimate things are?

I'm open to learning about where gaps remains and what you see as the areas needing some assistance/TLC.

Thanks!

mjcarroll commented 1 year ago

Out of curiosity, is there a reason you've opted to managed repos/branches with vcs instead of bazel's built-in mechanisms (e.g., git_repository, etc.) for doing the same? We've found the ability to manage many repos as externals to be very powerful.

Namely because I was actively developing on all of those repos. The git_repository rules seem to be really effective if you are pulling in a dependency at a specific revision/tag, but aren't very helpful if you want to also work on the contents of the repo.

Using VCS gets you closer to "mono-repo" approach that seems to be favored for developing with bazel.

mjcarroll commented 1 year ago

In terms of landing https://github.com/gazebosim/gz-bazel/pull/61 and supporting a full bazel-based build of gz-sim, how far do you estimate things are?

Probably not far off with a little bit of attention. I want to get the remaining updates landed on garden branches before the harmonic feature freeze next week.

bd-mdelasa commented 1 year ago

Out of curiosity, is there a reason you've opted to managed repos/branches with vcs instead of bazel's built-in mechanisms (e.g., git_repository, etc.) for doing the same? We've found the ability to manage many repos as externals to be very powerful.

Namely because I was actively developing on all of those repos. The git_repository rules seem to be really effective if you are pulling in a dependency at a specific revision/tag, but aren't very helpful if you want to also work on the contents of the repo.

Using VCS gets you closer to "mono-repo" approach that seems to be favored for developing with bazel.

Makes sense. In case it's useful, I've found --override_repository to be useful in those cases.

Longer term, do you expect the workflow to continue to use vcs? Perhaps it's too soon to tell?

bd-mdelasa commented 1 year ago

In terms of landing #61 and supporting a full bazel-based build of gz-sim, how far do you estimate things are?

Probably not far off with a little bit of attention. I want to get the remaining updates landed on garden branches before the harmonic feature freeze next week.

Thanks for the context. It sounds like the vision is to have harmonic support a native bazel-based build. Is that correct?

mjcarroll commented 1 year ago

It sounds like the vision is to have harmonic support a native bazel-based build. Is that correct?

Correct!

mjcarroll commented 1 year ago

Makes sense. In case it's useful, I've found --override_repository to be useful in those cases.

I'm unfamiliar with that, I will take a look.

Longer term, do you expect the workflow to continue to use vcs? Perhaps it's too soon to tell?

Probably too soon to tell, I'm open to alternatives. vcs is a pretty common tool in the ROS ecosystem, so it's generally available.

gedalia commented 1 year ago

it appears that the commit can be a branch name as well, git_repository( name = "", remote = "", commit = "origin/branch_name", ) commit is fed to: git reset --hard COMIT_STRING so it can support a sha or a branch name.

gedalia commented 1 year ago

Tried the git_repository approach here: WORKSPACE.txt utils works:

bazel test @utils//...

but most of the other packages are unhappy due to wanting BUILD files.

gedalia commented 1 year ago

I suspect all the GZ_ROOT + "subdir_name" paths are now incorrect.

moriarty commented 3 months ago

Oh hey @gedalia and @bd-mdelasa :smile: what a small world...

I've tried twice in the past couple of weeks to get this to build without success.

moriarty commented 3 months ago

There are a few weird things going on that are missing from the install overview @mjcarroll is the README up for suggestions?

Bouncing between the gitlab ci files and the readme and mixing in your two open PRs I've gotten some things compiling to some state but not finished... and the where it's failing is likely on my specific env setup... I'll try on another clean Ubuntu machine when I get a chance... this is just a personal side project

mjcarroll commented 3 months ago

is the README up for suggestions?

Of course! I also started working on migration to bzlmod since it's the future (I'm told), but after running into multiple dependency issues, I back-burnered it for a little while. I would like to get back to working on that at some point, but I don't forsee it happening before ROSCon in October.

Any contributions you would like to put forward, I'm happy to review and take though.

gazebosim / gz-bazel