search

carvel-dev / ytt

YAML templating tool that works on YAML structure instead of text
https://carvel.dev/ytt
Apache License 2.0
1.63k stars 137 forks source link

panic when using template.replace from within a module #162

Open pn-santos opened 4 years ago

pn-santos commented 4 years ago

Not sure if I'm attempting to do something that is not supposed to work but here goes:

With this:

#@ load('@ytt:template', t='template')
#@ load('job.lib.yml',           'job')
#@ load('fn/standard.star',      'std')

#@ def name_bundle(name):
#@   return std.suffix(name, 'config')
#@ end
#@
#@ def name_upload_job(name):
#@   return t.replace(job.config(std.suffix(name_bundle(name), 'upload')))
#@ end

 - _: #@ name_upload_job(bundle.name)

where job.config is bound to default_config:

#@ def default_config(name):
name: #@ name
serial: true
build_log_retention:
  builds: 20
  days: 2
  minimum_succeeded_builds: 1
#@ end

It works as expected.

However if I try to move the t.replace to the function in the job module:

#@ load('@ytt:template', t='template')
#@ load('job.lib.yml',           'job')
#@ load('fn/standard.star',      'std')

#@ def name_bundle(name):
#@   return std.suffix(name, 'config')
#@ end
#@
#@ def name_upload_job(name):
#@   return job.config(std.suffix(name_bundle(name), 'upload'))
#@ end

 - _: #@ name_upload_job(bundle.name)

where job.config is bound to default_config:

#@ def _default_config(name):
name: #@ name
serial: true
build_log_retention:
  builds: 20
  days: 2
  minimum_succeeded_builds: 1
#@ end

#@ def default_config(name):
#@   return t.replace(_default_config(name))
#@ end

then I get a panic:

ytt: Error: 
- runtime error: index out of range [-1] (backtrace: goroutine 1 [running]:
    in default_config
      job.lib.yml:36 | #@   return t.replace(_default_config(name))
    in name_upload_job
      nexus-uploader.yml:22 | #@   return job.config(std.suffix(name_bundle(name), 'upload'))
    in <toplevel>
      nexus-uploader.yml:43 |   - _: #@ name_upload_job(bundle.name)

    reason:
     runtime/debug.Stack(0x917400, 0x966da0, 0xc000318900)
        runtime/debug/stack.go:24 +0x9d
     github.com/k14s/ytt/pkg/template/core.ErrWrapper.func1.1(0xc0001ae3d8)
        github.com/k14s/ytt@/pkg/template/core/errs.go:18 +0x9d
     panic(0x966da0, 0xc000318900)
        runtime/panic.go:679 +0x1b2
     github.com/k14s/ytt/pkg/template.(*CompiledTemplate).TplReplaceNode(0xc0003da000, 0xc0001ce730, 0xc0001863f0, 0xc0004259f0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x30, ...)
        github.com/k14s/ytt@/pkg/template/compiled_template.go:285 +0xf4
     github.com/k14s/ytt/pkg/template/core.ErrWrapper.func1(0xc0001ce730, 0xc0001863f0, 0xc0004259f0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/k14s/ytt@/pkg/template/core/errs.go:25 +0xff
     go.starlark.net/starlark.(*Builtin).CallInternal(0xc0001863f0, 0xc0001ce730, 0xc0004259f0, 0x1, 0x1, 0x0, 0x0, 0x0, 0xc000080000, 0x7f6b18146008, ...)
        go.starlark.net@v0.0.0-20190219202100-4eb76950c5f0/starlark/value.go:653 +0x8b
     go.starlark.net/starlark.Call(0xc0001ce730, 0xae9160, 0xc0001863f0, 0xc0004259f0, 0x1, 0x1, 0x0, 0x0, 0x0, 0xae9120, ...)
        go.starlark.net@v0.0.0-20190219202100-4eb76950c5f0/starlark/eval.go:1028 +0x190
     go.starlark.net/starlark.(*Function).CallInternal(0xc00038f0c0, 0xc0001ce730, 0xc000425700, 0x1, 0x1, 0x0, 0x0, 0x0, 0xc000080000, 0x7f6b18146008, ...)
        go.starlark.net@v0.0.0-20190219202100-4eb76950c5f0/starlark/interp.go:305 +0x3ed8
     go.starlark.net/starlark.Call(0xc0001ce730, 0xae91e0, 0xc00038f0c0, 0xc000425700, 0x1, 0x1, 0x0, 0x0, 0x0, 0xae9b20, ...)
        go.starlark.net@v0.0.0-20190219202100-4eb76950c5f0/starlark/eval.go:1028 +0x190
     go.starlark.net/starlark.(*Function).CallInternal(0xc0003e9540, 0xc0001ce730, 0xc000425680, 0x1, 0x1, 0x0, 0x0, 0x0, 0xc000080000, 0x7f6b18146008, ...)
        go.starlark.net@v0.0.0-20190219202100-4eb76950c5f0/starlark/interp.go:305 +0x3ed8
     go.starlark.net/starlark.Call(0xc0001ce730, 0xae91e0, 0xc0003e9540, 0xc000425680, 0x1, 0x1, 0x0, 0x0, 0x0, 0xae9ae0, ...)
        go.starlark.net@v0.0.0-20190219202100-4eb76950c5f0/starlark/eval.go:1028 +0x190
     go.starlark.net/starlark.(*Function).CallInternal(0xc000217a80, 0xc0001ce730, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x37, 0x38, ...)
        go.starlark.net@v0.0.0-20190219202100-4eb76950c5f0/starlark/interp.go:305 +0x3ed8
     go.starlark.net/starlark.Call(0xc0001ce730, 0xae91e0, 0xc000217a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x9, ...)
        go.starlark.net@v0.0.0-20190219202100-4eb76950c5f0/starlark/eval.go:1028 +0x190
     go.starlark.net/starlark.(*Program).Init(0xc00000ead8, 0xc0001ce730, 0xc000187cb0, 0x0, 0x0, 0xc000217540)
        go.starlark.net@v0.0.0-20190219202100-4eb76950c5f0/starlark/eval.go:358 +0x96
     github.com/k14s/ytt/pkg/template.(*CompiledTemplate).eval(0xc000034f60, 0xc0001ce730, 0xc000187cb0, 0x0, 0x0, 0x0, 0x0, 0x0)
        github.com/k14s/ytt@/pkg/template/compiled_template.go:155 +0x247
     github.com/k14s/ytt/pkg/template.(*CompiledTemplate).Eval(0xc000034f60, 0xc0001ce730, 0xae04e0, 0xc000186bd0, 0xc0000b4500, 0xc0001ce730, 0xc000187aa0, 0x7155ba, 0x7fffe411ff89)
        github.com/k14s/ytt@/pkg/template/compiled_template.go:112 +0x6c8
     github.com/k14s/ytt/pkg/workspace.(*TemplateLoader).EvalYAML(0xc000186bd0, 0xc0000b47d0, 0xc0000b47d0, 0xc0000b4500, 0xc00001ef00, 0x9, 0x10, 0x1)
        github.com/k14s/ytt@/pkg/workspace/template_loader.go:169 +0x614
     github.com/k14s/ytt/pkg/workspace.(*LibraryLoader).eval(0xc0001afad8, 0xc0001c6050, 0x0, 0xc00000e488, 0x1, 0x1, 0xc0001c6050, 0x0, 0x0, 0x1, ...)
        github.com/k14s/ytt@/pkg/workspace/library_loader.go:139 +0xa6c
     github.com/k14s/ytt/pkg/workspace.(*LibraryLoader).Eval(0xc0001afad8, 0xc0001c6050, 0x0, 0x0, 0xc0001c6050)
        github.com/k14s/ytt@/pkg/workspace/library_loader.go:90 +0x50
     github.com/k14s/ytt/pkg/cmd/template.(*TemplateOptions).RunWithFiles(0xc0000cf040, 0xc00001e580, 0x9, 0x10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/k14s/ytt@/pkg/cmd/template/cmd.go:131 +0x3be
     github.com/k14s/ytt/pkg/cmd/template.(*TemplateOptions).Run(0xc0000cf040, 0x0, 0x0)
        github.com/k14s/ytt@/pkg/cmd/template/cmd.go:85 +0x288
     github.com/k14s/ytt/pkg/cmd/template.NewCmd.func1(0xc00011a500, 0xc00011ea10, 0x0, 0x7, 0x0, 0x0)
        github.com/k14s/ytt@/pkg/cmd/template/cmd.go:53 +0x2a
     github.com/cppforlife/cobrautil.WrapRunEForCmd.func1.1(0xc00011a500, 0xc00011ea10, 0x0, 0x7, 0x0, 0x0)
        github.com/cppforlife/cobrautil@v0.0.0-20180924214100-a39a1714c920/misc.go:25 +0xaf
     github.com/spf13/cobra.(*Command).execute(0xc00011a500, 0xc00001e090, 0x7, 0x7, 0xc00011a500, 0xc00001e090)
        github.com/spf13/cobra@v0.0.3/command.go:762 +0x460
     github.com/spf13/cobra.(*Command).ExecuteC(0xc00011a500, 0xc00011a500, 0xe34340, 0x3)
        github.com/spf13/cobra@v0.0.3/command.go:852 +0x2ea
     github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v0.0.3/command.go:800
     main.main()
        github.com/k14s/ytt@/cmd/ytt/ytt.go:17 +0x103
     )

Am I trying to do something that isn't supposed to work or is this a bug?

cppforlife commented 4 years ago

short repro scenario: https://get-ytt.io/#gist:https://gist.github.com/cppforlife/89108d13240490f604fc98e7ac04c236

cppforlife commented 4 years ago

ytt has several "special" functions that need to integrate deeper into structure building stack (vs regular functions that just return structures). template.replace is one of them. will dig in further to see whats going on.

pivotaljohn commented 3 years ago

...10 months later...

While we haven't seen others run into this issue, this falls under the category of "you're using the tool well, it's just a sufficiently advanced case that the current implementation didn't account for it."

This bug is relevant to ytt — we'd like for authors to be able to template.replace() from another module. If a contributor debugged this issue and fixed it, we would definitely accept that contribution.

That said, given the current set of priorities, the Carvel maintainers cannot commit to picking up this work, today.

pivotaljohn commented 3 years ago

As @cppforlife indicated above, this issue requires a more in-depth review to determine an approach.

That would likely result in a design discussion. It indicate that this is in a kind of "research" state, labeling this way and moving to "Carvel Accepted" pipeline.

aaronshurley commented 3 years ago

We're going to move this to the Unprioritized Backlog for now. We're open to contributions but there will likely need to be some design discussions first. Please let us know if you'd be interested in contributing to this work.