search

electricimp / Builder

Preprocessor
Other
21 stars 4 forks source link

readme

Builder

Current version: 4.2.0

Build Status

Contents

About Builder

Builder combines a preprocessor with an expression language and advanced imports.

There are a number of ways in which you can install Builder depending on how you plan to integrate it into your workflow. Once installed on your computer, you can use it to process your Squirrel application and factory firmware before you transfer the code to an impCentral™ Device Group.

Note The Electric Imp VS Code extension already incorporates Builder and can be used to upload code to Device Groups. If you are using the VS Code extension, there is no need install Builder separately to take advantage of its features.

You can use Builder to pull the contents of separate code files into your main source code files. These additional files might contain library code that you make use of across a number of different products, or they might contain confidential data which you don’t want to keep inside source code files that are managed through a software version control system.

You tell Builder which files to import, and where within your main source code they should be inserted, by using the @include command. Builder is able to access additional files that are stored on your computer or held remotely on an external resource (eg. HTTP/HTTPs server or a repository).

While Builder can be used to insert code this way, it can be used in far more sophisticated ways thanks to its integrated expression processor and programming logic. For example, if you need to generate multiple versions of your application firmware for versions of your product which make use of different imp modules, you can use Builder’s conditional execution features, variables and loops to pull your various code components together at build time and output files that are ready to be transferred to impCentral.

To speed up the process, files that are stored remotely which are not expected to change between builds can be cached for quick re-use. Builder's reproducible artifacts feature makes it possible to store references to all files and variables, so that builds can be re-created for future debugging.

For details on the commands that Builder offers, please see the Directives section. This is part of the Builder Syntax section, which also describes how Builder commands are structured.

Builder Installation

Builder requires Node.js 18.14.2 and above. It can be installed and used by two ways:

Command Line Tool Installation

Install Builder:

npm install -g Builder

Now use Builder’s pleasebuild command to configure the newly installed utility:

pleasebuild [-l] [-D<variable> <value>]
    [--github-token <token>] [--azure-user <username> --azure-token <token>]
    [--bitbucket-server-addr <address>] [--bitbucket-server-user <username> --bitbucket-server-token <token>]
    [--lib <path_to_file>] [--use-remote-relative-includes] [--suppress-duplicate-includes-warning]
    [--cache] [--clear-cache] [--cache-exclude-list <path_to_file>]
    [--save-dependencies [<path_to_file>]] [--use-dependencies [<path_to_file>]]
    [--save-directives [<path_to_file>]] [--use-directives [<path_to_file>]]
    <input_file>

where <input_file> is the path to source file which should be preprocessed and the other options are:

Option Synonym Mandatory? Value Required? Description
-l No No Generates line control statements. For a more detailed explanation, please read this GCC page
-D<variable> No Yes Defines a variable. May be specified several times to define multiple variables
--github-token No Yes A GitHub personal access token. Password is not supported.
--azure-user No Yes An Azure Repos username.
--azure-token No Yes An Azure Repos personal access token. Should be specified if the --azure-user option is specified
--bitbucket-server-addr No Yes A Bitbucket Server address. E.g., https://bitbucket-srv.itd.example.com. Note: this option is mandatory to include files from Bitbucket Server
--bitbucket-server-user No Yes A Bitbucket Server username.
--bitbucket-server-token No Yes A Bitbucket Server personal access token or password (not recommended). Should be specified if the --bitbucket-server-user option is specified.
--lib --libs No Yes Include the specified JavaScript file(s) as a library. May be specified several times to include multiple libraries. The provided value may specify a concrete file or a directory (all files from the directory will be included). The value may contain wildcards (all matched files will be included)
--use-remote-relative-includes No No Interpret every non-absolute path in the @include and @include once directives as relative to the location of the source file where it is mentioned. See the Include Files section
--suppress-duplicate-includes-warning --suppress-duplicate No No Do not show a warning if a source file with the same content was included multiple times from different locations and this results in code duplication
--cache -c No No Turn on caching for all files included from remote resources. See the Caching Remote Files section. This option is ignored if the --save-dependencies option is specified (the cache is turned off for all files in this case). If the --use-dependencies option is specified the cache is turned off for the files referenced in the dependency file and is turned on for all other remote files
--clear-cache No No Clear the cache before Builder starts running. See the Caching Remote Files section
--cache-exclude-list No Yes Set the path to the file that lists resources which should not be cached. the Caching Remote Files section
--save-dependencies No No Save references to the required repository files in the specified file. If a file name is not specified, the dependencies.json file in the local directory is used. See ‘Reproducible Artifacts’
--use-dependencies No No Use the specified file to set which repository files are required. If a file name is not specified, the dependencies.json file in the local directory is used. See ‘Reproducible Artifacts’.
--save-directives No No Save Builder variable definitions in the specified file. If a file name is not specified, the directives.json file in the local directory is used. See ‘Reproducible Artifacts’
--use-directives No No Use Builder variable definitions from the specified file. If a file name is not specified, the directives.json file in the local directory is used. See ‘Reproducible Artifacts’

Library Installation

Install Builder:

npm i --save Builder

Now instantiate, configure and execute Builder from within your source code. For example:

const Builder = require('Builder');
const builder = new Builder();

// Specify whether you need line control statements. See the "-l" CLI option.
builder.machine.generateLineControlStatements = <true|false>;

// Cache all files included from remote sources. See the "--cache" CLI option.
builder.machine.useCache = <true|false>;

// Set GitHub credentials. See the "--github-token" CLI option.
builder.machine.readers.github.token = "<PERSONAL_ACCESS_TOKEN>";

// Set Azure Repos credentials. See the "--azure-user" and "--azure-token" CLI options.
builder.machine.readers.azureRepos.username = "<USERNAME>";
builder.machine.readers.azureRepos.token = "<ACCESS_TOKEN>";

// Set Bitbucket Server address and credentials. See the "--bitbucket-server-*" CLI options.
builder.machine.readers.bitbucketSrv.serverAddr = "<ADDRESS>";
builder.machine.readers.bitbucketSrv.username = "<USERNAME>";
builder.machine.readers.bitbucketSrv.token = "<PASSWORD_OR_ACCESS_TOKEN>";

// Path to the file that lists the resources which should be excluded from caching.
// See the "--cache-exclude-list" CLI option.
builder.machine.excludeList = "<PATH_TO_FILE>";

// Interpret a non-absolute path in includes as relative to the current file.
// See the "--use-remote-relative-includes" CLI option.
builder.machine.remoteRelativeIncludes = <true|false>;

// Suppress warning about duplicate includes.
// See the "--suppress-duplicate-includes-warning" CLI option.
builder.machine.suppressDupWarning = <true|false>;

// See the "--save-dependencies" CLI option.
builder.machine.dependenciesSaveFile = <false|"PATH_TO_FILE">;
// See the "--use-dependencies" CLI option.
builder.machine.dependenciesUseFile = <false|"PATH_TO_FILE">;

// See the "--save-directives" CLI option.
builder.machine.directivesSaveFile = <false|"PATH_TO_FILE">;
// See the "--use-directives" CLI option.
builder.machine.directivesUseFile = <false|"PATH_TO_FILE">;

const inputFile = "PATH_TO_YOUR_INPUT_FILE";

// Set the directory of the input file as one of search dirs.
// path.resolve() is used to get the absolute path (requires the path nodejs module)
builder.machine.readers.file.inputFileDir = path.resolve(inputFile);

const result = builder.machine.execute(`@include "${inputFile}"`);
console.log(result);

To understand Builder configuration, please review this source code.

Builder Syntax

Expressions

Types

The following value types are supported in expressions:

Operators

Builder supports the following operators:

Binary

||  &&  ==  !=  <  >  <=  >=  +  -  *  /  %

Unary

+  -  !

Member Expressions

Membership of an object is expressed using any of the following expressions:

Conditional Expressions

Builder provides the standard ternary operator (?:) for evaluating basic conditions:

<condition> ? <if_condition_true> : <if_condition_false>

Variables

Variables can be used in Builder expressions. Variable names can contain $, _, latin letters and digits, however they must not start with a digit. Variables can be defined in the following ways:

All undefined variables are evaluated as null.

Variable Definition Order

When resolving a variable’s value:

  1. Builder looks for its definition among the command line parameters (as -D<variable name> <variable value>) passed to the pleasebuild command.
  2. If no such variable definition is found, the code is scanned for @set directive statements preceding the variable usage.
  3. If no variable definitions are found in the previous steps, Builder looks in the host environment variables.

Environment Variables

There is no special predicate required to make use of environment variables. Builder looks in the host environment variables to try and resolve the expressions if no command line or local variables have been set.

For example, on a Mac:

server.log("Host home path is @{HOME}");

will print the home directory path of the current user of the system where Builder was executed.

Environment variables differ based on OS. If you wish to use environment variables with Builder, a quick internet search will give you details on how to list the variables currently available on your system and also how to set new variables.

Builder Variables

Builder provides the following pre-defined variables:

Builder has two directives @while and @repeat for managing loops. Inside these loops the following variables are available:

Usage examples for these variables can be found in the @while and @repeat directive examples.

Builder Functions

Builder provides the following helper functions:

Builder also comes with some string functions, based on the JavaScript methods of the same names. These functions are available under the namespace S. The first argument of each function is always the string to be operated on. For documentation on the remaining arguments, please see ‘JavaScript String Methods’.

Comments

Lines starting with @ followed by space or a line break are treated as comments and not added to the output. For example:

@ This is a Builder comment and will not appear in the output

Any text following // and extending to the end of the line will be ignored by Builder* and will not appear in the result output. For example:

@set SOME_STRING = "my string" // This is a Builder comment that will not appear in output

Directives

All of Builder’s directives start with the @ symbol. Don’t include a space or line break between the @ and the required directive’s name as this will be interpreted as a comment.

@{...} Inline Expressions/Macros

This directive evaluates the expression enclosed by braces ({ and }) and inserts the result into the output. The enclosure can be the value of a named variable, an expression or a macro.

@{<variable:identifier>}
@{<expression>}
@{macro(a, b, c)}

Example

The line:

The result is: @{123 * 456}.

results in the following output:

The result is: 56088.

@set

This directive assigns a value or the value of an expression to a variable. Variables are defined in a global context. A value can be any supported type or function.

@set <variable:identifier> <value:expression>
@set <variable:identifier> = <value:expression>

Example

In this example, we define a number of variables using @set, then use @{...} to create squirrel log messages with those variables. If the following lines are added to your source code:

@ Set Builder global variables
@set SOME_INT    = 10
@set SOME_STRING = "my string"
@set BOOL_VAL    = (12 > 4)
@
@ Use a Builder function to set a global variable
@set MIN_INT min(1, 2, 3)
@
// Use Builder global variables in squirrel log messages
server.log(@{SOME_INT});
server.log("@{SOME_STRING}");
server.log(@{BOOL_VAL});
server.log(@{MIN_INT});

then during processing, Builder will output:

// Use Builder global variables in squirrel log messages
server.log(10);
server.log("my string");
server.log(true);
server.log(1);

@macro

This directive defines a code block with its own parameters. Macros are declared in a global scope. Macro parameters are only available within the macro scope and override global variables with the same name (but do not affect them).

Define A Macro

@macro <name>(<arguments>)
    <body>
@endmacro

Note @end can be used in place of @endmacro if you prefer.

Use A Macro

Macros can be used either inline with the @{...} directive, or with the @include directive. When macros are used inline no line-control statements are generated for the output inside the macro scope and trailing newlines are trimmed from the macro output.

@{macro(a, b ,c)}
@include macro(a, b ,c)

Inline Example

Define a macro and use the @{...} directive to create a multi-line string to log in squirrel:

@ Define a macro
@macro some_macro(a, b, c)
    Hello, @{a}!
    Roses are @{b},
    And violets are @{c}
@end
@
// Use an inline Builder macro to create a multi-line string
poem <- @"@{some_macro("username", "red", "blue")}";
server.log(poem);

This results in the following output:

// Use an inline Builder macro to create a multi-line string
poem <- @"    Hello, username!
    Roses are red,
    And violets are blue";
server.log(poem);

Include Example

Define a macro and use it with @include to create a multi-line string to log in squirrel:

@ Define a macro
@macro some_macro(a, b, c)
    Hello, @{a}!
    Roses are @{b},
    And violets are @{c}
@end
@
// Use Builder include and macro directives to create a multi-line squirrel string
poem <- @"
@include some_macro("username", "red", "blue")
";
server.log(poem);

This results in the following output:

// Use Builder include and macro directives to create a multi-line squirrel string
poem <- @"
    Hello, username!
    Roses are red,
    And violets are blue
";
server.log(poem);

Optional Parameter Example

Use a function to configure and use a macro with an optional parameter.

@ Define a macro with an optional parameter
@macro some_macro(a, b, c)
    Hello, @{a}!
    Roses are @{b},
    And violets are @{defined(c) ? c : "of undefined color"}.
@end
@
// Use Builder include and macro directives to create a multi-line squirrel string
poem <- @"
@include some_macro("username", "red")
";
server.log(poem);

This results in the following output:

// Use Builder include and macro directives to create a multi-line squirrel string
poem <- @"
    Hello, username!
    Roses are red,
    And violets are of undefined color.
";
server.log(poem);

@include

This directive can be used to include local files, files from remote resources or macros.

@include <source:expression>

The @ character must not be present in the path and name of the file which is being included.

Please see the Include Files section for information about include formats, file searching rules, supported remote resources, examples and other details.

@include once

This directive acts just like @include but has no effect if the specified source has already been included. However, macros are always included.

@include once <source:expression>

@while

This directive invokes a loop which only ends when specified conditions are met. You can access Builder’s loop variables within @while loops.

@while <test:expression>
    // 0-based iteration counter: @{loop.index}
    // 1-based iteration counter: @{loop.iteration}
@endwhile

Note @end may be used in place of @endwhile if you prefer.

Example

The following lines, when added to your source code:

@set myvar = 12

@while myvar > 9
    @set myvar = myvar - 1
var: @{myvar}
    loop.index: @{loop.index}
    loop.iteration: @{loop.iteration}
@end

will output:

var: 11
    loop.index: 0
    loop.iteration: 1
var: 10
    loop.index: 1
    loop.iteration: 2
var: 9
    loop.index: 2
    loop.iteration: 3

@repeat

This directive invokes a loop that repeats for a certain number of iterations. You can access Builder’s loop variables within @repeat loops.

@repeat <times:expression>
    // 0-based iteration counter: @{loop.index}
    // 1-based iteration counter: @{loop.iteration}
@endrepeat

Note @end may be used in place of @endrepeat if you prefer.

Example

The following lines, when added to your source code:

@repeat 3
    loop.index: @{loop.index}
    loop.iteration: @{loop.iteration}

@end

will output:

    loop.index: 0
    loop.iteration: 1

    loop.index: 1
    loop.iteration: 2

    loop.index: 2
    loop.iteration: 3

@if... @elseif... @else

This directive provides conditional branching.

@if <test:expression>
    // Consequent code
@elseif <test:expression>
    // else if #1 code
@else
    // Alternative code
@endif

Note @end may be used in place of @endif if you prefer.

Example

@if __FILE__ == 'abc.ext'
    // include something
@elseif __FILE__ == 'def.ext'
    // include something else
@else
    // do something completely different
@endif

@error

This directive simply emits an error.

@error <message:expression>

Example

@if PLATFORM == "platform1"
    // platform 1 code
@elseif PLATFORM == "platform2"
    // platform 2 code
@elseif PLATFORM == "platform3"
    // platform 3 code
@else
    @error "Platform " + PLATFORM + " is unsupported"
@endif

@warning

This directive simply emits a warning.

@warning <message:expression>

Example

@if PLATFORM == "platform1"
    // platform 1 code
@elseif PLATFORM == "platform2"
    // platform 2 code
@elseif PLATFORM == "platform3"
    // platform 3 code
@else
    @warning "Building for default platform"
    // default platform code
@endif

Filters

The filter operator, |, allows you to pass a value through any of the supported functions.

@{<expression> | <filter>}

This is equivalent to:

@{<filter>(<expression>)}

Example

// Include external HTML piped through the escape processing function
a = "@{include('index.html')|escape}"

// Include an external binary piped through the base64 encoder function
b = "@{include('file.bin')|base64}"

Include Files

This section contains detailed information about usage of the @include and @include once directives to include local or remote files.

Local file is a file on the same system where Builder is running.

Remote file is a file on a remote resource (eg. a repository or a HTTP/HTTPs server). See the Remote Include Files section for the list of supported remote resources.

The file mentioned in the @include or @include once directives may contain a relative or an absolute path.

Examples:
@include "https://example.com/somefile.ext" // absolute path to remote file
@include once "github:electricimp/Promise/promise.class.nut" // absolute path to remote file
@include "c:\somefolder\somefile.ext" // absolute path to local file
@include once "./somefile.ext" // relative path
@include "somefile.ext" // relative path

The next section clarifies how Builder searches the included file.

Searching The Included File

Processed file is a file currently being processed by Builder, it can be local or remote, it contains the @include or @include once directive(s).

Path to the processed file is an absolute path to the processed file.

Path in the include is the path mentioned in the @include or @include once directive, it can be an absolute path to local file, an absolute path to remote file or a relative path.

Full path to the include file is an absolute path to the file which will be included by Builder.

The below rules define how Builder searches the included file:

Absolute Path To Remote File

If the path in the include is an absolute path to the remote file, it is considered as the final path to the include file. If the file is not found there, Builder reports an error.

Examples:
@include "https://example.com/somefile.ext" // absolute path to remote file
@include once "github:electricimp/Promise/promise.class.nut" // absolute path to remote file

Absolute Path To Local File

If the path in the include is an absolute path to the local file, it is considered as the final path to the include file. If the file is not found there, Builder reports an error.

Examples:
@include "c:\somefolder\somefile.ext" // absolute path to local file
@include "/home/user/someuser/somefolder/somefile.ext" // absolute path to local file. But see the note below.

Note: Under some conditions a path in the include which starts from the "/" symbol is processed by the special rule described below.

"/" As Root Of Remote Resource

If

then the final path to the include file is a concatenation of the root of remote resource and the path in the include. If the file is not found there, Builder reports an error. See the Remote Include Files section for the root definition of the supported remote resources.

Examples:
The processed file: "https://example.com/folderA/folderB/somefile.nut"

It contains:
@include "/folderC/anotherfile.nut"

The final path to the include: "https://example.com/folderC/anotherfile.nut"
The processed file: "github:someuser/somerepo/folderA/folderB/somefile.nut@develop"

It contains:
@include "/folderC/anotherfile.nut"

The final path to the include: "github:someuser/somerepo/folderC/anotherfile.nut@develop"

Relative Path To Remote File

If

then the final path to the include file is a concatenation of the path to the processed file and the path in the include. If the file is not found there, Builder reports an error.

Examples:
The processed file: "https://example.com/folderA/folderB/somefile.nut"

It contains:
@include "anotherfile.nut"

The final path to the include: "https://example.com/folderA/folderB/anotherfile.nut"
The processed file: "github:someuser/somerepo/folderA/folderB/somefile.nut@v1.2.3"

It contains:
@include "folderC/anotherfile.nut"

The final path to the include: "github:someuser/somerepo/folderA/folderB/folderC/anotherfile.nut@v1.2.3"

Relative Path To Local File

If

then Builder makes the following steps to find the include file:

  1. Only if the processed file is a local file, the final path to the include file is a concatenation of the path to the processed file and the path in the include. If the file is not found there, moves to the next step.
  2. The final path to the include file is a concatenation of the path to the file specified as the <input_file> parameter of the pleasebuild command and the path in the include. If the file is not found there, moves to the next step. Note: If you use Builder as a library and want this search step to work, make sure you specify the input file path (builder.machine.readers.file.inputFileDir variable) in your source code within the Builder initialization routine. See the Library Installation section to learn more.
  3. The final path to the include file is a concatenation of the path to the directory from where the pleasebuild command has been called and the path in the include. If the file is not found there, Builder reports an error.
Examples:
The processed file: "/home/user/someuser/folderA/somefile.ext"

It contains:
@include "folderX/anotherfile.nut"

The "pleasebuild" command is called from: "/home/user/someuser/folderB"
With the <input_file> parameter: "folderC/initialfile.nut"

The anotherfile.nut file will be searched sequentially in the following locations:
1. "/home/user/someuser/folderA/folderX/"
2. "/home/user/someuser/folderB/folderC/folderX/"
3. "/home/user/someuser/folderB/folderX/"

Remote Include Files

Supported Remote Resources

Builder supports file includes from the following types of remote resources:

HTTP/HTTPs Server

For includes from weblinks.

Format:
@include "http://<server>[:<port>]/<path>"
@include "https://<server>[:<port>]/<path>"
@include once "http://<server>[:<port>]/<path>"
@include once "https://<server>[:<port>]/<path>"

where:

Examples:
@include "https://example.com/somefile.ext"
@include once "http://example.com/folderA/folderB/somefile.nut"
Root of the remote resource:
http://<server>:<port>
https://<server>:<port>

GitHub Repository

For includes from github.com.

Format:
@include "github:<user>/<repo>/<path>[@<ref>]"
@include once "github:<user>/<repo>/<path>[@<ref>]"

where:

Examples:
@include "github:electricimp/Promise/promise.class.nut" // head of the default branch
@include "github:electricimp/Promise/promise.class.nut@develop" // head of the develop branch
@include once "github:electricimp/Promise/promise.class.nut@v3.0.1" // tag v3.0.1
Root of the remote resource:
github:<user>/<repo>
Authentication:

Authentication is optional. It is required to access private repositories only. But please note that when you use authentication, the GitHub API provides much higher rate limits.

For the authentication you need to provide:

Bitbucket Server Repository

For includes from Bitbucket Server. Note, this is not the same as Bitbucket Cloud, includes from Bitbucket.org are not supported.

Builder can work with only one Bitbucket server at a time. Its address must be specified (--bitbucket-server-addr option). Server version 5.3.0 or above is supported.

Format:
@include "bitbucket-server:<project>/<repo>/<path>[@<ref>]"
@include "bitbucket-server:~<user>/<repo>/<path>[@<ref>]"
@include once "bitbucket-server:<project>/<repo>/<path>[@<ref>]"
@include once "bitbucket-server:~<user>/<repo>/<path>[@<ref>]"

where:

Examples:
@include "bitbucket-server:Tools/Promise/promise.class.nut" // head of the default branch
@include once "bitbucket-server:Tools/Promise/promise.class.nut@develop" // head of the develop branch
@include "bitbucket-server:~john/Promise/promise.class.nut@v3.0.1" // tag v3.0.1
Root of the remote resource:
bitbucket-server:<project>/<repo>
bitbucket-server:~<user>/<repo>
Authentication:

Authentication is optional. It is required to access private repositories only.

For the authentication you need to provide:

Azure Repository

For includes from Azure Repos.

Format:
@include "git-azure-repos:<org>/<project>/<repo>/<path>[@<ref>]"
@include once "git-azure-repos:<org>/<project>/<repo>/<path>[@<ref>]"

where:

Examples:
@include once "git-azure-repos:org/project/repo/path/some.class.nut" // head of the default branch
@include "git-azure-repos:org/project/repo/path/some.class.nut@develop" // head of the develop branch
@include "git-azure-repos:org/project/repo/path/some.class.nut@v3.0.1" // tag v3.0.1
Root of the remote resource:
git-azure-repos:<org>/<project>/<repo>
Authentication:

Authentication is optional. It is required to access private repositories only.

For the authentication you need to provide:

Local Git Repository

For includes from Git repositories hosted locally. Note, even as files are local, Builder considers them as files from a remote resource when searching the file to include.

Format:
@include "git-local:<path>[@<ref>]"
@include once "git-local:<path>[@<ref>]"

where:

Examples:
@include once "git-local:/path/to/repo/and/file/some.class.nut" // head of the default branch
@include once "git-local:/path/to/repo/and/file/some.class.nut@develop" // head of the develop branch
@include "git-local:/path/to/repo/and/file/some.class.nut@3.0.1" // tag v3.0.1
@include "git-local:/path/to/repo/and/file/some.class.nut@c13c59e96f3f6a37f75f9e520d0fdc5591e0ba83" // concrete commit
Root of the remote resource:

Is determined by Builder.

Notes:

Caching Remote Files

To reduce compilation time, Builder can optionally cache files included from a remote resource. If the file cache is enabled, remote files are cached locally in the .builder-cache directory. Cached resources expire and are automatically invalidated 24 hours after their addition to the cache.

To turn the cache on, pass the --cache or -c option to Builder. If this option is not specified, Builder will not use the file cache even if the cached data exists and is valid — Builder will continue to query remote resources on every execution.

To reset the cache, use both the --cache and the --clear-cache options.

If a resource should never be cached, it needs to be added to the exclude-list.builder file (see the example below). You can use wildcard characters to mask file names.

Note: Non-identical paths, even if they point to absolutely the same location, are treated as different paths. Eg., if you include /a/b/1.nut, it becomes cached, and then you include /a/b/../b/1.nut, it will not be found in the cache.

Wildcard Pattern Matching

Pattern matching syntax is a similar to that of .gitignore. A string is a wildcard pattern if it contains '?' or '*' characters. Empty strings or strings that starts with '#' are ignored.

A '?' symbol matches any single character. For example, bo?t.js matches boot.js and boat.js, but doesn't match bot.js.

A '*' matches any string, that is limited by slashes, including the empty string. For example, /foo/*ar matches /foo/bar, /foo/ar and /foo/foo-bar, but doesn't match /foo/get/bar or /foo/bar/get.

Two consecutive asterisks ** in patterns matched against full pathname may have special meaning:

Examples:
# Avoid caching a specific file
github:electricimp/MessageManager/MessageManager.lib.nut

# Exclude all electricimp repos
github:electicimp/**

# Exclude all tagged files or files from the specific branches from the cache
github:*/**/*@*

Saving And Reusing Versions Of Remote Files

It is possible to save the versions of all remote files which are used during the current run of Builder, and reuse exactly that versions later. This is applicable for files from repositories only.

See the Repository Files: Dependencies section for the details.

Proxy Access To Remote Files

To specify a proxy that should be used when you are including files from remote resources, set the environment variables HTTP_PROXY/http_proxy and/or HTTPS_PROXY/https_proxy for HTTP and HTTPS protocols respectively.

For example, to operate through a proxy running at IP address 192.168.10.2 on port 3128 for HTTP requests, you should set the environment variable: HTTP_PROXY='http://192.168.10.2:3128'. All of Builder’s HTTP requests will now go through the proxy.

Advanced Builder Usage

This section contains information that will help you work with Builder more effectively, but may not be needed for more basic Builder tasks.

Reproducible Artifacts

It is possible to save the build configuration data used for preprocessing a source file in order to create an identical source file again later with that saved configuration. Builder variable definitions are saved in a ‘directives.json’ file, and references to the concrete versions of repository files and libraries are stored in a ‘dependencies.json’ file.

Builder Variables: Directives

The --save-directives [<path_to_file>] and --use-directives [<path_to_file>] options are used to, respectively, save and reuse Builder variable definitions. The definitions are saved in a JSON file. If a file name is not specified, the directives.json file in the local directory is used. These options are processed the similar way as the --save-dependencies and --use-dependencies options, above.

When the --use-directives [<path_to_file>] option is used, the saved Builder variable definitions are merged with definitions specified by -D<variable> <value> options.

A typical directives.json file looks like this:

{
  "Variable0": "value0",
  "Variable1": "value1"
}

Repository Files: Dependencies

--save-dependencies [<path_to_file>] and --use-dependencies [<path_to_file>] options are used to save and to reuse, respectively, references to concrete versions of repository files and libraries. The references are saved in a JSON file. If a file name is not specified, the dependencies.json file in the local directory is used. Every reference consists of repository file URL and:

For more information, please see the Git Manual and the Git API.

These options are processed the following way:

Note If --save-dependencies is specified, the --cache option is ignored. If --use-dependencies is specified, the --cache option does not affect the files referenced in the dependency file.

A typical dependencies.json file looks like this:

[
  [
    "github:ProjectA/repositoryA/fileA",
    "2ff017dc92e826ad184f9cdeadd1a2446f8d6032"
  ],
  [
    "github:ProjectB/repositoryB/fileB",
    "a01b64f9ce764f226f52c6b9364396d4a8bd550b"
  ],
  [
    "bitbucket-server:projectC/repositoryC/fileC",
    "4bc4024f1f2ad99e8bd2ade73d151912e031d1f5"
  ],
  [
    "git-azure-repos:org/projectD/repositoryD/fileD",
    "d1ccee9ed6e250c6d5e1f052107125659d3ba9d0"
  ],
  [
    "git-local:/path/to/repo/and/fileD",
    "c13c59e96f3f6a37f75f9e520d0fdc5591e0ba83"
  ]
]

Note: Non-identical paths, even if they point to absolutely the same location, are treated as different paths. Eg., if you have saved dependencies for /a/b/1.nut and then you include /a/b/../b/1.nut - the dependencies will not be applied to this file.

Including JavaScript Libraries

Builder can accept JavaScript libraries to add functionality to its global namespace. The library should export an object, the properties of which will be merged into the global namespace. For example, to include a function, upper(), to convert strings to uppercase, define your library file like so:

module.exports = {
    upper: (s) => s.toUpperCase()
};

Now include the function within directives in your input file:

@{upper("warning:")}
@{upper(include("warning.txt"))}

Finally, run Builder with the option --lib path/to/your/lib/file.

Binding The Context Object Correctly

Functions called by Builder will be called with their this argument set to a Builder context object. Within the context object, Builder variables like __FILE__, functions like max(), and other included library functions will be made available at the top level. Variables defined in your input code with @macro or @set will be available under the key globals.

Ignoring the binding of this may cause unexpected behavior, for example when calling methods on objects. Take the following example library:

class MyClass {
    constructor(str) {
        this._str = str;
    }

    getStr() {
        return this._str;
    }
}

myObject = MyClass("my text");

module.exports = {
    myObject
};

Attempting to use this library with the directive @{myObject.getStr()} will not deliver the expected behavior because this in getStr() will be set to a Builder context object and not to myObject. When calling class methods ensure they have been bound to the correct value of this:

class MyClass {
    constructor(str) {
        this._str = str;
    }

    getStr() {
        return this._str;
    }
}

myObject = MyClass("my text");

module.exports = {
    getStr: myObject.getStr.bind(myObject)
};

Testing

When running tests locally, please test on both Windows and macOS. All environment variables are optional. However, if you are working with @includes from GitHub and do not provide GitHub credentials, rate limits imposed by GitHub may cause test failures. The default for SPEC_LOGLEVEL is error.

npm install
SPEC_LOGLEVEL=<debug|info|warning|error>
SPEC_GITHUB_TOKEN=<GitHub personal access token>
npm test

Note 1: The standard set of tests doesn't include Bitbucket Server, Azure Repos and Git Local (but see the Note 2) integration testing. To run Bitbucket Server, Azure Repos or Git Local tests, please see the sections below.

Note 2: The standard set of tests uses Git Local in several tests for testing the overall behavior of Builder. They require the SPEC_GIT_LOCAL_REPO_PATH variable to be set (see the Git Local section below). These are optional tests so they will be skipped if that variable is not set.

Note 3: There are several tests that require access to the root of the filesystem (or to the root of the disk C: on Windows). They will create/remove there a directory named builder_test_g2e5r6uh, so please make sure you don't have some important data in such directory.

Bitbucket Server

Prerequisites:

  1. A running instance of Bitbucket Server
  2. A clone of Builder repo placed on this server
  3. If the server / repo are not public, an account (username and password / token) with permissions to access the repo
npm install
SPEC_LOGLEVEL=<debug|info|warning|error>
# E.g., "https://bitbucket-srv.itd.example.com"
SPEC_BITBUCKET_SERVER_ADDRESS=<Bitbucket Server address>
# Format: "<project>/<repo>". E.g., "myProj/BuilderClone"
# If the repo belongs to a user (not to a project), the format is: "~<user>/<repo>". E.g., "~john/BuilderClone"
SPEC_BITBUCKET_SERVER_REPO_PATH=<Path to the cloned Builder repo on the server>
SPEC_BITBUCKET_SERVER_USERNAME=<Bitbucket Server username>
SPEC_BITBUCKET_SERVER_TOKEN=<Bitbucket Server password/access token>
npm run test:bitbucket-server

Azure Repos

Prerequisites:

  1. A clone of Builder repo placed at Azure Repos
  2. An account (username and token) with permissions to access the repo
npm install
SPEC_LOGLEVEL=<debug|info|warning|error>
# Format: "<org>/<project>/<repo>". E.g., "myOrg/myProj/BuilderClone"
SPEC_AZURE_REPOS_REPO_PATH=<Path to the cloned Builder repo at the Azure Repos>
SPEC_AZURE_REPOS_USERNAME=<Azure Repos username>
SPEC_AZURE_REPOS_TOKEN=<Azure Repos access token>
npm run test:azure-repos

Git Local

Prerequisites:

  1. A clone of Builder repo placed locally
npm install
SPEC_LOGLEVEL=<debug|info|warning|error>
SPEC_GIT_LOCAL_REPO_PATH=<Path to the root of the cloned Builder repo>
npm run test:git-local

License

Builder is licensed under the MIT License.