Testing Environment for Tilt Extensions

[samwyma]

As we've been migrating to using Tilt for our dev environment management, we've been extending the functionality of Tilt using the load function to import function from other Tiltfiles. As the functionality we started adding became more complex, we started to feel the need for a way to unit test our Tiltfile functions so we could develop faster and feel confident as we made changes. Since there didn't seem like an easy way to test Starlark files, we decided to leverage the fact that Starlark is Pythonic and just run our tests in Python using pytest. Here is an example of what we ended up with. That repo contains the scaffolding we use to run Tiltfiles, as well as some of the custom extensions we've been working on as examples for how we ended up testing them.

We've found that having that simple Tiltfile runner made our lives so much easier when developing extensions. Are there any plans to provide an environment to develop extensions which matches the actual Tilt runtime better? Or maybe a way to run automated tests against a real cluster?

Let me know if you'd like any more information about that example repo or what we're trying to do.

And thanks for the great tool! It's been great fun to tinker around with :)

[landism]

We've taken two steps in this direction:

• tilt alpha tiltfile-result will execute your Tiltfile and spit out json describing the manifests, etc that were created, which looks like it'd take care of a fair amount of your scaffolding, and then you'd be left validating the resulting json.
• tilt ci runs your Tiltfile and tries to simply start everything and then exits with a status code depending on whether everything came up successfully or not. This 1) tests that your Tiltfile can successfully deploy your services and 2) leaves the services running so that you can then run whatever kind of integration tests you want to write using any framework, outside of Tilt.

How do those sound to you?

[samwyma]

Certainly tilt alpha tiltfile-result is a step in the right direction. I'm just having a play with the command now and it looks like we could probably get most of what we wanted by combining the output of the base command and the output of -b. We're mainly looking to capture and interrogate the calls to built-in Tilt functions such as local and k8s_yaml, so -b is likely the most useful for us at this stage.

We've successfully set up a couple of our projects with Tilt now, but at some point we'll migrate from our test scaffolding to using the output of tiltfile-result and see how it compares.

[samwyma]

I've just come back to trying to set up some tests via tilt alpha tiltfile-result -b - when it displays the list of called methods, it doesn't output the arguments which were supplied by keyword.

If I have a call such as local_resource(name="test", cmd="echo", resource_deps=["another"]), the call which tilt alpha tiltfile-result -b outputs is - local_resource()

[landism]

If I have a call such as local_resource(name="test", cmd="echo", resource_deps=["another"]), the call which tilt alpha tiltfile-result -b outputs is - local_resource()

Thanks for the report! We've fixed this for the next release.