Bazel Rules for Cuelang

cueckoo commented 3 years ago

Originally opened by @gravypod in https://github.com/cuelang/cue/issues/163

I was wondering if there was a canonical set of rules for using cuelang in a bazel build environment. For my usecase this would be helpful for reducing the tooling a new engineer will need to learn and allow for better integration with things like tilt bazel run //deployments:development would be very nice.

I think this also makes sense to mainline in cuelang as, from the README.md, code generation from CUE is a planned feature. The ability to have a cue_library and automatically link generated code into projects of different languages would be very helpful.

cueckoo commented 3 years ago

Original reply by @mpvl in https://github.com/cuelang/cue/issues/163#issuecomment-560378747

Can you give a more concrete example of how things look like for what you had in mind?

cueckoo commented 3 years ago

Original reply by @gravypod in https://github.com/cuelang/cue/issues/163#issuecomment-561222042

@mpvl most languages contain some form of rules_<language> that supply <language>_library and <language>_binary.

Jsonnet, for example, has a rules_jsonnet which supplies some common functionality that you might want to do:

jsonnet_library: collection of jsonnet files that become importable
jsonnet_to_json: Convert jsonnet libraries into json data

Something that might work for cuelang would likely be:

cue_library: Making sources visible to your build
cue_gen_proto: Generate .protos from cue definitions
cue_gen_openapi: Generate openapi from cue definitions
cue_gen_yaml: Gen yaml
cue_gen_json: Gen json
cue_gen_kube_yaml: Maybe include some magic that allows higher level abstractions that could integrate with other bazel rules (rules_docker for example). Please.build has an example of this working.

Some of these might be better to implement outside of a rules_cuelang (specifically 5)

cueckoo commented 3 years ago

Original reply by @mpvl in https://github.com/cuelang/cue/issues/163#issuecomment-561303783

Gotcha. This would have to be build int the bazelbuild Github repo.

Also, I'm quite unfamiliar with bazel and this should ideally be build by someone that is using this in practice. I am happy to help or guide the person doing this, though.

cueckoo commented 3 years ago

Original reply by @rudolph9 in https://github.com/cuelang/cue/issues/163#issuecomment-563433480

Though I'm not familiar with this bazel rules either, a step in this direction may be to support builds using tinyGo. I may be mistaken but I believe some of the same limitations around go modules are shared between a bazelbuild and a tinyGo build. A tinyGo build would also likely reduce the overhead of running cue and make the WASM target much lighter (relate wasm/npm ticket)

cueckoo commented 3 years ago

Original reply by @davidzchen in https://github.com/cuelang/cue/issues/163#issuecomment-573369998

@mpvl I'm a Bazel contributor and the original author of bazelbuild/rules_rust, bazelbuild/rules_jsonnet, and bazelbuild/rules_d. I'm interested in writing Bazel rules for Cue.

Since Bazel Cue rules will depend on the cue executable, my understanding is that we would need to add BUILD files for building Cue (using bazelbuild/rules_go). I think bazelbuild/bazel-gazelle will be helpful for generating and updating BUILD files.

I think the rules that @gravypod listed is a good start. For data validation, I am thinking we can have bazel test rules (perhaps cue_validate_yaml_test or something similar).

I will have to do more reading on Cue's other use cases, but if aren't objections, I can write a short proposal and start working on a proof of concept of building Cue with Bazel and an initial prototype with cue_library, a couple of rules for generating JSON and YAML, and a validation test rule.

WDYT?

cueckoo commented 3 years ago

Original reply by @seh in https://github.com/cuelang/cue/issues/163#issuecomment-573429456

If you didn't feel like arranging for building cue in these rules, you could instead download built executable files using http_archive, since this project publishes such files for three platforms: macOS, Linux, and Windows.

cueckoo commented 3 years ago

Original reply by @gravypod in https://github.com/cuelang/cue/issues/163#issuecomment-573430538

@seh would be an option but http_archive and other workspace rules are a little cumbersome. If build files and rules were in repo you could vendor this project in your company's third_party very easily. You could also use git_repository to refer to any commit.

cueckoo commented 3 years ago

Original reply by @seh in https://github.com/cuelang/cue/issues/163#issuecomment-573436242

I use http_archive like that in four Bazel packages I maintain, for Packer, Terraform, ct, and kubecfg. It works just fine, without much fuss, but I understand that it's not quite "the Bazel way." However, I wouldn't want to hold up development of rules that address using cue on developing rules to build cue.

cueckoo commented 3 years ago

Original reply by @davidzchen in https://github.com/cuelang/cue/issues/163#issuecomment-573800108

bazelbuild/bazel-gazelle can be used to automatically generate BUILD files, and it also adds all the go_repository rules for external dependencies to the WORKSPACE file.

I was able to generate WORKSPACE dependencies and BUILD.bazel files for Cue with Gazelle and successfully build //cmd/cue with the generated BUILD files:

I still need to verify that all the tests run successfully, but I think I can send a PR for this soon.

cueckoo commented 3 years ago

Original reply by @davidzchen in https://github.com/cuelang/cue/issues/163#issuecomment-573803991

FYI the changes are in the bazel branch in my fork: https://github.com/davidzchen/cue/tree/bazel

cueckoo commented 3 years ago

Original reply by @davidzchen in https://github.com/cuelang/cue/issues/163#issuecomment-573815834

Some remaining issues to resolve from this first pass:

Correctly embed testdata files in tests, such as for //cue/load/import_test.
- BUILD.bazel files are being generated in some of the testdata directories. (https://github.com/bazelbuild/rules_go#how-do-i-access-testdata), and these are likely extraneous and should be removed
- Testdata be instead using the data attribute per the [rules_go documentation]
Set up protobuf dependency to WORKSPACE for proto-related tests under //encoding/protobuf
Add the necessary Gazelle directives to correctly generate build files, taking into account the above.

cueckoo commented 3 years ago

Original reply by @davidzchen in https://github.com/cuelang/cue/issues/163#issuecomment-573841830

Most tests are now passing:

cueckoo commented 3 years ago

Original reply by @davidzchen in https://github.com/cuelang/cue/issues/163#issuecomment-589909847

Sorry for the delay. I’ve been busy, but I have resumed working on this. There are some issues with getting some of the tests run by Bazel to correctly find the testdata files. Hoping to get the PR out within the next week.

cueckoo commented 3 years ago

Original reply by @steeve in https://github.com/cuelang/cue/issues/163#issuecomment-611542308

Since Bazel Cue rules will depend on the cue executable, my understanding is that we would need to add BUILD files for building Cue (using bazelbuild/rules_go). I think bazelbuild/bazel-gazelle will be helpful for generating and updating BUILD files.

I think it's perhaps better to just download the cue binaries maybe? That's would remove the dependency to rules_go. ~Although proto import isn't exposed in the CLI.~

If you didn't feel like arranging for building cue in these rules, you could instead download built executable files using http_archive, since this project publishes such files for three platforms: macOS, Linux, and Windows.

~I would +1, but since importing from proto isn't exposed in the CLI, we need to compile it anyhow :(~

Also, I'm not sure adding bazel files in this repo is the way to go unless Cue maintainers actively use them. Worst would be for those build files to be unmaintained.

cueckoo commented 3 years ago

Original reply by @tnarg in https://github.com/cuelang/cue/issues/163#issuecomment-617453128

I've created an initial set of Bazel rules for cue: https://github.com/tnarg/rules_cue. I'd love to get your feedback on them. They include a cue_binary rule which performs a cue export, and a cue_library rule for bundling a cue package. The rules also include a gazelle extension which is used by the repo itself to generate BUILD.bazel files for the examples.

cueckoo commented 3 years ago

Original reply by @steeve in https://github.com/cuelang/cue/issues/163#issuecomment-617719528

@tnarg super nice! saw a few things but overall really nice!

cueckoo commented 3 years ago

Original reply by @steeve in https://github.com/cuelang/cue/issues/163#issuecomment-617735387

@tnarg opened a few issues! very impressed overall! I would have opened PRs but I'm busy on something else at the moment. Great job.

cue-lang / cue

Bazel Rules for Cuelang #163