kaskada-ai / kaskada

Modern, open-source event-processing
https://kaskada.io/
Apache License 2.0
352 stars 15 forks source link

feat: Label integration tests; update or eliminate makefile #473

Open bjchambers opened 1 year ago

bjchambers commented 1 year ago

Summary

When running locally, it would be useful if we could rely on the labels to filter integration tests. For instance --query-filter='!pulsar && !redis' to run all of the tests other than Pulsar and Redis tests.

We already have many of the tests properly labeled. For example: https://github.com/kaskada-ai/kaskada/blob/main/tests/integration/api/mat_pulsar_to_obj_store_test.go#L28

However, some of the tests rely on Pulsar without being labeled. For example: https://github.com/kaskada-ai/kaskada/blob/main/tests/integration/api/queries_v1_test.go#L36

In general, we should prefer to label entire functions as we currently do. But based on the Gingkgo docs, there are other options if we really need these tests that use Pulsar internally:

It("can save books remotely", Label("network", "slow", "library query") {
  if Label("performance").MatchesLabelFilter(GinkgoLabelFilter()) {
    exp := gmeasure.NewExperiment()
    // perform some benchmarking with exp...
  }
  // rest of the saving books test
})

(from https://onsi.github.io/ginkgo/#spec-labels)

The other part of this is that the current structure of the makefile makes it difficult to pass arguments -- we need to create additional targets for every change. We should look at passing other arguments through the makefile, or using a shell script or something. Then we could more easily run something like ENV=local-local go run github.com/onsi/ginkgo/v2/ginkgo ./... --label-filter='!pulsar && !redis' (note that this also omits the -vv since the extra verbosity is just spammy in many cases).

epinzur commented 1 year ago

490 adds the remaining pulsar test labels.

epinzur commented 1 year ago

I think we should make a small CLI tool for running the most of the commands currently in the Makefile.

Options on commands would be much more straightforward than enumerating every potential command in the Makefile.

bjchambers commented 1 year ago

I'm 50/50 -- I think it may be nice to just document the commands and people can create aliases if they want? Specifically, it seems easier to "know" how things are being run if we just document "use the ginkgo runner". But :shrug: