Embedding static resources in Ballerina binaries

[sameerajayasoma]

The current design is suboptimal and works only with jBallerina: anything in the resources/ directory will be copied to the executable .jar file. But there is no API to access these resources at runtime, and one needs to use Java interop to access them.

We need a design that works for jBallerina and nBallerina.

We looked at several approaches. The first one is similar to the //go:embed directive, where we can annotate a module-level variable with the path of the file to be embedded. The file's content will be added as the variable value at compile time.

@embed {pattern:”hello.txt”}
final string s = “”;

@embed {pattern:”hello.txt”}
final readonly & byte[] b = [];

We discussed using compile-time functions to load the content of a file as the second approach.

const string s = foo:read("hello.txt");

Using configurable consts is the third approach. We considered doing compile-time configurable const in the initial stages of the configurable feature, but postponed because there were no real requirements.

configurable const string s = ?

[jclark]

My thoughts on these:

• @embed makes the program differently from how the spec says it should behave
• configurable is mixing two different concepts; we don't necessarily want things that come from a file to be configurable
• const runtime functions is conceptually clean, but only makes sense if we add const runtime functions a bit more generally (i.e. at least allow langlib functions to be const where that makes sense e.g. length function)

[jclark]

C is adding similar functionality - as #embed directive

https://thephd.dev/finally-embed-in-c23

[sanjiva]

Another more first class way to solve this would be to add a concept of a "data" variable: data string s = ("some string" | ?); And then use a const annotation @embed {} to further tell how to populate it.

This way the semantics are maintained - that it's a bit of data that is filled in at compile time with either a constant value or from a compile time resource. Not sure ? is the best way to say that some extra info is expected.

[sameerajayasoma]

I propose going ahead with the @embed approach here. It's similar to C #embed and Go //embed.

@jclark I didn't quite understand your point about the @embed annotation.

[sameerajayasoma]

Here are the proposed semantics of the @embed annotation:

• It can only be attached to module-level variables of type string or readonly & byte[].
• The given pattern should match only one file.
• If the file doesn't exist, the compiler must exit with an error.

[hasithaa]

Here are my thoughts on @sameerajayasoma's Proposal.

• Module-level variables have to be final. Otherwise, it can be mutated. Also, we may want to make them non-public as well. These restrictions are needed to make them secure.
• Another possible type is map for properties files. Or we can allow anydata mapping type if we can do the data binding with @embed annotation.

[jclark]

@sameerajayasoma My concern is that the language already specifies clear semantics for:

final string s = ''';

It is that the s is initialized to the empty string and cannot thereafter be reassigned to. The semantics of the proposed @embed annotation contradict these semantics. Annotations should be augmenting language semantics, not contradicting them.

I'm a strong -1 to the @embed annotation.

[jclark]

Why can't we do this with a regular library function?

[manuranga]

This a good usecase for compile time functions if and when we have them.

[sanjiva]

How about

@embed {pattern: "x.text"}
const string data = ?;

This is declaring a constant and saying the value is coming from somewhere else. The static annotation tells where the value is coming from.

Effectively this desugars to:

const string data = "blah blah blah";

assuming the file x.text contains "blah blah blah".

[chathurace]

Desugaring large text data into strings may fail the compiler. I experienced such compilation failures when very large text values (i.e. encoded EDI mappings) are assigned to strings as constants. Same text loading into a string at runtime does not cause any issue.

I think it's more flexible if we can introduce a library function to load resources at runtime.

[sanjiva]

Chathura you can always do this even now:

final string foo = check io:fileReadString("path-to-file");

But the catch is you have to get the file at the right place at runtime.

We can't do a Java style embedded jar embedded resources because the binary format will not be a zip file at some point in the future.

The compiler should ideally not load the bytes to memory but just transfer it to the output directly .. then it can avoid size issues.

[chathurace]

The problem with io:fileReadString(...) is that it cannot read resources from imported libraries. In my case, for a given collection of EDI mappings, I want to pack EDI mapping json files, generated Ballerina records and some EDI collection specific utility code into a Ballerina library. Then anyone can use this library to parse those EDI files without worrying about mappings, code generation, etc. However, due to this restriction, those EDI mapping json files packed in the library cannot be accessed.

With the proposed solution, if the compiler doesn't load resources at compile time, instead generate code to load resources only when they are referred, we can avoid size issues as you have mentioned.

[sanjiva]

That is not possible .. the source is not available at runtime.

What I was suggesting is the compiler can move the bytes from the source to the output binary file without bringing it into the compiler's memory.

[chathurace]

I was thinking something like keeping the bytes of resources in the output binary file, but load those into (output program's) memory at runtime only when a resource is referred.