improve build system

[luxe]

step 1: make sure the build is more hermetic. see if sam can actually build it.
find out what problems he runs into. We recently transitioned to a monorepo and went from a custom build system / dependency manger to bazel; no one has built it except me on 2 different ubuntu machines.

step 2: start to bridge the gap of madness to accommodate new developers. projectional editor will be a better barrier of entry

step 3: set up CI so people can also contribute without having to ensure things locally

[luxe]

we also need to start a small generated book. it will be a nice 20 minute read and get people onboard with the core ideas/philosophies of what we are building

[luxe]

^ started this on the main readme

[sfortson]

First attempt at bazel build --cxxopt='--std=c++17' //...:all:

Starting local Bazel server and connecting to it...

INFO: Invocation ID: de930dcc-086b-42a2-9f96-0f499158793e

ERROR: /home/samuel/unilang/source/code/utilities/types/strings/observers/path/BUILD:5:1: no such package '@com_github_boost_range//': The native git_repository rule is deprecated. load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository") for a replacement. Use --incompatible_remove_native_git_repository=false to temporarily continue using the native rule. and referenced by '//code/utilities/types/strings/observers/path:lib'

ERROR: Analysis of target '//code/utilities/types/strings/observers/path:lib' failed; build aborted: no such package '@com_github_boost_range//': The native git_repository rule is deprecated. load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository") for a replacement. Use --incompatible_remove_native_git_repository=false to temporarily continue using the native rule.

INFO: Elapsed time: 10.764s

INFO: 0 processes. FAILED: Build did NOT complete successfully (406 packages loaded, 737 targets configured) Fetching @com_github_boost_range; fetching Fetching @com_github_boost_algorithm; fetching

When I add the flag mentioned in the error, I get this:

ERROR: /home/samuel/unilang/source/code/other/test/BUILD:1:1: no such package '@gtest//': The native new_http_archive rule is deprecated. load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") for a drop-in replacement. Use --incompatible_remove_native_http_archive=false to temporarily continue using the native rule. and referenced by '//code/other/test:hello-test' ERROR: Analysis of target '//code/other/test:hello-test' failed; build aborted: no such package '@gtest//': The native new_http_archive rule is deprecated. load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") for a drop-in replacement. Use --incompatible_remove_native_http_archive=false to temporarily continue using the native rule. INFO: Elapsed time: 27.485s INFO: 0 processes. FAILED: Build did NOT complete successfully (424 packages loaded, 1173 targets configured) Fetching @gtest; fetching 17s

When I add that flag, I get this:

ERROR: /home/samuel/unilang/source/code/transcompilers/hcp-compiler/bootstrap/program_options/BUILD:5:1: C++ compilation of rule '//code/transcompilers/hcp-compiler/bootstrap/program_options:lib' failed (Exit 1) gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 23 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox In file included from ./code/transcompilers/hcp-compiler/bootstrap/program_options/program_options_checker.hpp:2, from code/transcompilers/hcp-compiler/bootstrap/program_options/program_options_checker.cpp:2: ./code/transcompilers/hcp-compiler/bootstrap/program_options/program_options.hpp:7:10: fatal error: boost/program_options.hpp: No such file or directory

include "boost/program_options.hpp"

      ^~~~~~~~~~~~~~~~~~~~~~~~~~~

compilation terminated. INFO: Elapsed time: 30.468s, Critical Path: 2.66s INFO: 9 processes: 9 processwrapper-sandbox. FAILED: Build did NOT complete successfully

[luxe]

i wonder whats harder, building code or writing code.

revised trevor steps:

• [x] fix boost bazel deps within bazel I know some of these are wrong. Make sure I think all of them are right.

resolution: ok, everything is pulling from more correct rules now.
The only thing I couldn't get built out of the box was boost graph.
added a rule, currently patched https://github.com/nelhage/rules_boost/issues/101

• [x] ensure no boost import is being used without using this package (its currently looking for global includes where it failed)

resolution: ctrl+f replaced all the old deps.
one thing that worries me, is a "header only" file might be used, but retrieved from the system instead of from bazel. I'm not actually sure that if that's possible. I think bazel would complain?
We'll know for sure when we run the build in a container, but based on just looking around, I think I got em.

• [x] remove all of the old boost build files

resolution: all deleted

we were considering travis ci, but they give you 50 minutes of build time.
it will be better to just make a bare bones docker container, and run the build in there for a little more sanity so its def not looking for system includes eh, i change my mind. we could have travis ci test as much as possible without going over.
for example, it would be good to have travis ci at least clone the repo, and run a bazel command that prints hello world. then we know the general build can bootstrap properly (since sam had to install java and bazel).
we can branch out to a few free CIs and get those going.
Its not even a hard requirement that these all pass, but it help give us some visibility

• [x] travis ci run a bazel hello world (echoing)
• [x] circle ci run a bazel hello world (echoing)
• [x] appveyor run a bazel hello world (echoing)

resolution: pretty simple. just dropped some files on the root repo to trigger these things.
And added the badges to the main readme

• [x] automatically pull and use the same bazel version internally in the repo

resolution: ok, we have a bootstrapping bazel. seems to work for me.
The bazel executable is actually installed in the user's home directory.
Is that a good idea? should we install it inside the repo itself??
We can reconsider later, but noted that in the script called user_build

• [x] make a single shell hello world

resolution: bare bones hello world in sh

• [x] have all the CIs build and run that shell hello world.
  Then we can prove that builds are able to bootstrap and build something simple

resolution: yes, they are.

so hopefully our bare minimum of requirement will just me a working shell and a "normal: machine. sh? bash? hopefully not matter. might want to check for certain things like, is git installed, is wget installed, stuff like that.

• [x] pull the same compiler (if possible; not sure how hard that is)

resolution: ok, I've sort of done this, but using a local compiler inside the repo can't build all of the code. like right off the bat we see: no member named 'ptr_fun' in namespace 'std' maybe its depricated or something; I need to look into that. I'm going to turn this compiler off for the moment. fix all of the warnings with the system compiler that works. then we can go through all of the individual issues on the local compiler. Hopefully the local compiler is actually newer?? and stuff has gotten depricated/removed from the std library? not sure. and I'm not sure what new errors we will see.
I'd rather start with a warning free version on a working compiler first. added a new step to turn this back on later

• [x] fix all current compiler warnings with system installed gcc
• [x] evaluate all other system includes/imports
• [x] adjust those so we think they are being used correctly in the build
• [x] disable any that aren't important and aren't easy to integrate at this time.
  off the top of my head, possibly xdo and imgui. curl we can pull in (if we aren't already) (we'll know when we look at all the systems)

resolution: eh, things like gmp are not hermetic. I turned off xdo and imgui.
I'd rather find these issues later with CI and docker, and different toolchain... and maybe the sysroot thing

• [x] remove patch after author accepts build rule fixes
  • [x] make a diagram of this build and bootstraping process so we can maintain it in the future.
    Start from the very beginning of having some kind of shell (we can discuss containers, VMs, bare bones arch linux, etc... bootstraping the bazel build. building with bazel. pulling in dependencies. building our own tools/transcompilers, etc. all the way to the entire repo being built.

resolution: to the best of my understanding. added to readme in install section

• [x] turn on local llvm compiler
• [x] fix all of the issues on the new local llvm compiler

resolution: surprisingly not as painful as I expected. 2 things needed fixed with the compiler switch.

• ptr_fun depreicated; replaced with lambda

• here was some function that was used before it was declared. I guess gcc allows it, but clang prevents it. whatever, i fixed it.

• [x] quick test to see if tools execute.
  Its a breadth of fresh air to get everything to finally build again after changing a lot of tooling,
  but I've had random crashes on startup of binaries in the past after doing this.

resolution: just ran --help on some tools. other tools are actually used as part of the build so they must be working. considering it good.

• [x] then fix the build command, so you don't have to pass that extra copts flag testing...

• [x] more accurate/complete CIs

• [x] fix new issues on CIs extra boost graph dependencies not being met. contributing back to other repo agian.

  • [ ] something with sysroot? I have a meeting about this

  • [ ] disable external linking I have a meeting about this

• [ ] set a default compiler toolchain? (instead of specifying directly on cli)

  • [ ] fix gmp dep You have to run gmp's configure script that does a bunch of system stuff so its hard to verify that it will actually succeed for people.
    According to gmp documentation:

    It might be possible to build without the help of ‘configure’,   
    certainly all the code is there, but unfortunately you’ll be on your own.

    "IT MIGHT BE POSSIBLE"... since this is autotools we are kind of stuck.
    This is a known issue in the bazel community. I'll punt this effort for now. https://github.com/bazelbuild/bazel/issues/2814

  When I think things really are hermetic now:

  • [x] have CIs do more of a real build using the bazel-bootstraping CIs are basically doing this now.

  • [ ] docker container build everything Some of the CIs are basically docker containers. so this might be kind of the same thing

• [ ] build on ubuntu laptop

• [ ] build on ubuntu desktop

• [ ] build on ubuntu work machine

• [x] build on ubuntu VM

after all that, we can ask Sam to build again. then go from there.

[luxe]

Things have gotten better. They are not perfect.
We have CIs passing (which is evidence that the build can work). I moved the rest of these unfinished tasks into the readme (under Installation) We can now create tickets for more fine-grained build issues.