Add preview support for Lightweight transforms

[schmelczer]

Summary

Address https://github.com/emdgroup/foundry-dev-tools/issues/45

Add support for running local preview for @lightweight and @transform_polars Transforms which were recently added to the Foundry Transforms library.

• The main contribution is adding local mock implementations for @lightweight, @transforms_polars, LightweightTransformInput, and LightweightTransformOutput.
• However, next to this, this PR also proposes to make Input download the datasets and load the Spark DataFrames lazily. With this, we can avoid initialising Spark when using Lightweight transforms, which shaves off a few seconds from every preview run.

  This also has the added benefit of improving the initial startup overhead in larger repositories.

• A new method, get_local_path_to_dataset, is added to Input to lazily load the underlying files of the Dataset without having to initialise Spark when running Lightweight transforms.

Limitations

• Bring-your-own-container workflows and setting resource limits (as-well-as provisioning GPUs) using @lightweight is not yet supported when running preview locally.
• When running fdt build, the build logs are fetched from spark reporter which is not available in the case of (lightweight) container transform. We could use foundry-telemetry-service's readV3 endpoint to tail logs, however, that has an at least 1 minute latency. Fortunately, a new, real-time log fetching endpoint is soont to be released, so we can transition to using that in a follow-up PR.
• CSV datasets are supported for Lightweight, but the Foundry schema is not loaded (just like for regular transforms)

Checklist

• [x] You agree with our CLA
• [x] Included tests (or is not applicable).
• [x] Updated documentation (or is not applicable).
• [x] Used pre-commit hooks to format and lint the code.

[CLAassistant]

All committers have signed the CLA.

[schmelczer]

I'm coordinating with legal regarding the CLA

[jonas-w]

@schmelczer thank you so much for this great contribution, I'll take a closer look at it tomorrow!

[schmelczer]

We're still actively looking into the CLA

[schmelczer]

The CLA has been cleared, so I've signed it! Feel free to take a look at the PR @jonas-w, I'm happy to discuss the details, especially if there're any counter-arguments against the lazy dataset loading

[nicornk]

Thanks for implementing the laziness in the decorators, something that was on our long list as well.

@schmelczer 🙏 thanks for the contribution.

[schmelczer]

Thanks for giving this a proper review, I've addressed your comments!

I've done some further testing with schema-less datasets for both lightweight & regular transforms and preview seems to be working as expected! As for the logging, there isn't a trivial way of implementing this (we could look at the JobSpec though) but the foundry telemetry service update is just around the corner which will work with both Spark and Lightweight transforms!

[nicornk]

lgtm

[jonas-w]

@schmelczer would it be possible to remove this merge commit d111123 (#47) and instead do a rebase? This way the history commit history would stay a bit cleaner after we merge it into main.

Otherwise it looks good to me!

[schmelczer]

Sure, I've dropped the merge commit @jonas-w!

[jonas-w]

Should be released by now https://pypi.org/project/foundry-dev-tools/

Thanks again @schmelczer!