Load files with arbitrary names

[mildred]

Currently, load(0 works by detecting the file type depending on the file name suffix. In case of yaml libraries, especially, it must end up with .lib.yml which is not always desired (you might just want to have the standard yaml suffix).

I suggest this limitation could be lifted by explicitly defining the type we want to load at load time. This could be achieved by separate functions such as loadYaml() or loadStarlark().

Another alternative would be to add a parameter to load() but this would be a backwards incompatible change.

[cppforlife]

one of the reasons we use .lib.yml extension is to determine which files are templates and hence should be rendered (included in stdout) vs plain library files that should be discarded. given that we need to identify which files are "root", i dont think we can rely on at-load time indicator (unless we start statically checking whether file uses load, but that's a bit weird imho).

i was considering adding some kind of annotation (e.g. #@file/type "library") that would be used in the file to indicate that file is a library file and should not be considered a template.

[mildred]

If changing the load() statement is not possible because it comes from starlark, peraps adding an annotation on the file name:

• load("$yml:loaded.yaml", ...)
• load("$txt:loaded.txt", ...)
• load("$starlark:loaded.starlark", ...)

I think it's better to define it at the load point (instead of within the loaded file) for different reasons:

• you don't always control the loaded file if it comes from a third party for example
• in some contexts you might want to load a YAML file as text for inclusion as a text object within your document

[cppforlife]

I think it's better to define it at the load point (instead of within the loaded file) for different reasons:

but how should we eliminate library files from being templated on their own?

you don't always control the loaded file if it comes from a third party for example

agreed.

in some contexts you might want to load a YAML file as text for inclusion as a text object within your document

thats possible today via @ytt:data library. https://get-ytt.io/#example:example-load-data-files and https://github.com/k14s/ytt/blob/develop/docs/lang-ref-ytt.md.

[cppforlife]

i do agree that we have special casing for lib.yaml and lib.txt extensions and somehow we should get rid of that.

[mildred]

but how should we eliminate library files from being templated on their own?

I don't think so

---

I tried to look at how files were loaded, and I come to a dead-end. In pkg/workspace/template_loader.go I see that the Load() function is calling Library.FindFile, which suggests the file has already been loaded? Is ytt loading the complete directory structure in the Library before starting up ?

[cppforlife]

which suggests the file has already been loaded?

Load() actually evaluates file. Library.FindFile() is just a way to access in memory representation of the file ([]byte content). all referenced files are read into memory at the beginning.

[nimakaviani]

With #18 finalized, I guess we can expect people to drop their arbitrary .yaml files (with no extra lib.yaml extensions) into _ytt_lib and not have them rendered in the output. This should eliminate the need for the specialized naming of yaml files.

Essentially I am suggesting that yaml files in _ytt_lib should only be used as libraries and don't get rendered to the output.

In other cases though (for text and starlark), I believe the suffix is a good visual clue on the expected content in the file. I believe it is less desirable for people to have to open arbitrarily named files to see what's inside them.