After numerous design discussions, we've arrived at a new API for running transforms. This commit makes a number of breaking changes at once:
download(ctx) and transform(ds,ctx) special functions are removed
a new dataset module is loaded by default into all transform scripts, the module is scoped to the dataset the transform script is bound to
a new module function dataset.latest() takes no arguments and returns the HEAD dataset version, returns an empty dataset if no history exists
a new module function dataset.commit(ds) requires a dataset argument and must be called by the user exactly once in a transform script to update dataset. The dataset passed to commit is passed directly to save, without merging into the prior version
calling dataset.commit halts the program
calling dataset.commitzero times produces a warning, but is not an error (will error with no changes in practice)
starlark context module is removed
config is an immutable dict-like type replaces ctx.get_config(key) with config.get(key[, default]). config is loaded into the transform script by default. config may inherit more dictionary-type properties in the future.
secrets is an immutable dict-like type, replaces ctx.get_sercret(key) with secrets.get(key[, default]). secrets is loaded into the transform script by default. secrets may inherit more dictionary-type properties in the future.
dataset.TransformStep.Category is no longer used
starlark steps are no longer frozen after execution, allowing symbol manipulation across steps
An example conversion from old to new:
OLD:
load("http.star", "http")
---
def download(ctx):
csvDownloadUrl = ctx.get_config("url")
return http.get(csvDownloadUrl).body()
---
def transform(ds, ctx):
# ctx.download is whatever download() returned
csv = ctx.download
# set the dataset body
ds.set_body(csv, parse_as='csv')
NEW:
load("http.star", "http")
ds = dataset.latest()
---
csvDownloadUrl = config.get("url")
res = http.get(csvDownloadUrl).body()
---
ds.body = res
dataset.commit(ds)
BREAKING CHANGES:
We've overhauled the starlark API to use a cell-based approach. All transform scripts will require updating
@dustmop, after a multi-hour rebase nightmare, I've got most of the changes we discussed re-worked atop the new dataframe api. tests in the cmd package aren't passing, but while I get those fixed up I'd love a review.
After numerous design discussions, we've arrived at a new API for running transforms. This commit makes a number of breaking changes at once:
download(ctx)
andtransform(ds,ctx)
special functions are removeddataset
module is loaded by default into all transform scripts, the module is scoped to the dataset the transform script is bound todataset.latest()
takes no arguments and returns the HEAD dataset version, returns an empty dataset if no history existsdataset.commit(ds)
requires a dataset argument and must be called by the user exactly once in a transform script to update dataset. The dataset passed to commit is passed directly to save, without merging into the prior versiondataset.commit
halts the programdataset.commit
zero times produces a warning, but is not an error (will error with no changes in practice)context
module is removedconfig
is an immutable dict-like type replacesctx.get_config(key)
withconfig.get(key[, default])
.config
is loaded into the transform script by default. config may inherit more dictionary-type properties in the future.secrets
is an immutable dict-like type, replacesctx.get_sercret(key)
withsecrets.get(key[, default])
. secrets is loaded into the transform script by default. secrets may inherit more dictionary-type properties in the future.dataset.TransformStep.Category
is no longer usedAn example conversion from old to new:
OLD:
NEW:
BREAKING CHANGES: We've overhauled the starlark API to use a cell-based approach. All transform scripts will require updating