search

bazelbuild / starlark

Starlark Language
Apache License 2.0
2.44k stars 160 forks source link

Centralize conformance tests in this repository #10

Open damienmg opened 5 years ago

damienmg commented 5 years ago

Hi,

The Go, the Rust and the Java implementation have their own integration tests that test different feature. When writing the integration tests for the rust language I basically took the Java and the Go one and made some variation: the error messages were not the same, the feature set not exactly the same (when importing the Go one) and the test framework is not exactly the same (assert_eq, fails, ....)

It would be great to standardize those test files and have them in this repo so change to the language would be translated here easily.

laurentlb commented 5 years ago

We have this: https://github.com/bazelbuild/bazel/tree/master/src/test/skylark We can make the script ignore the error messages.

damienmg commented 5 years ago

I used a different test runner for rust. I was not speaking about the test runner itself but the tests themselves. The format of tests in Bazel is the one I used in fine, just the error were different.

To have a nice conformance test suite, the format should probably be standardized and have go use the same.

Those tests should also live in this repository in my opinion so everyone could contribute to the same test suite.

Quarz0 commented 5 years ago

Hi,

I’m going to work on ensuring starlark’s implementation are in sync as part of GSoC’19 under @laurentlb mentoring. Regarding the common test suite, I'd like to discuss a few things. First, If we are going to add a test runner then it could be similar to the one for Java for example. It’s a python script that executes the starlark binary as a process and passes the location of the test files as arguments. However, If we did the same for a common test runner for the 3 implementations, then we’d need to add the path of the 3 executables of the go, rust and java implementations. Which I’m not sure if it’s a nice idea.

An alternative I’ve thought of is to use docker or something similar to automatically fetch and build the latest version of the 3 implementations, installing all dependencies in the docker image and then running the test suite inside the docker. This wouldn't require having a local copy of any of the implementations' codes/binaries to run the test cases.

laurentlb commented 5 years ago

I think we can use external repositories to test the 3 implementations.

Quarz0 commented 5 years ago

Concerning the java implementation, wouldn't this require the visibility of the "//src/main/java/com/google/devtools/starlark:Starlark" rule to be public as mentioned here, so that I can use it from another project?

laurentlb commented 5 years ago

Yes, we can do that.