Feature request: Add imports to other Bolt files

[kermankohli]

My bolts file is getting pretty large and it's getting pretty messy with heaps of different entities. It'd be really cool if you could do something like #import "users.bolts" and all your other bolts files into your "rules.bolts".

[mckoss]

Great idea. I've also been thinking about way to make application schema more modular for complex apps.

I can see two reasons for splitting up a Bolt file:

• Sharing types and functions for use as a "library".
• Embedding applications as a "child" in a larger database.

Here are some syntax proposals - I could use some help coming to a final design.

import would define a "namespace" for pre-defined types and functions:

import util;

path /users is util.User[] {
  validate() = util.isCurrentUser(this.id);
}

type Post {
  validate() = util.isRecent(modified);

  message: String;
  modified: util.CurrentTime;
}

For the case of embedding a complete application definition:

import chat at /apps/chat;

This would have the effect of importing the chat application, but making all paths relative to /apps/chat.

I see a problem with this approach in that multiple apps might want to share some common structure - like user accounts, but segregate their data otherwise - would have to figure out how to share some paths but not others....

[mckoss]

Maybe if the import statement does not have an at clause - then that just uses the global (common) location. Each app could then import users, and reference the same schema a paths...

[kermankohli]

Hmm that could definitely work! Or maybe there could be a public and private section in each file? Also is there any way to define optional properties? The current way of saying path /users/$userid is User then having to define every single other optional property by /path/users/$userid/age is String is pretty meh. Should be something simpler like

Type User {
age: String <optional>
name: String
}

[mckoss]

The way Bolt does this now, is:

type User {
  age: String | Null;
  name: String;
}

[mckoss]

Check out the freshly written guide for this and other tips.

[kermankohli]

Thanks! The new guide would have been really helpful before I began, but it's okay now, the Bolts rules are really easy and cool to work with :) Also how would something like newData.val().matches(/^[A-Z0-9,_%+-]+@[A-Z0-9.-]+.[A-Z]{2,4}$/i) be written in Bolts? I tried adding apostrophes before and after but the $i gave me a bit of trouble. Any ideas??

[mckoss]

Bolt uses "test" instead of "matches". Also - the parse does not currently support inline regexp in the syntax - but you can put it in a string (include the whole thing, /i and all), and it will generate the correct regexp in the JSON output.

path / is T;

type T extends String {
  validate() =  this.test('/^[A-Z0-9,_%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}$/i');
}

generates:

{
  "rules": {
    ".validate": "newData.isString() && newData.val().matches(/^[A-Z0-9,_%+-]+@[A-Z0-9.-]+.[A-Z]{2,4}$/i)"
  }
}

[brewsoftware]

Hi guys, getting a little off track above. I've created PR #142 with a proposed syntax for imports. It's not perfect and uses some different conventions but I'm basically going for a subset of the Javascript convention.

This would be

import {'./foo'}; a local file called 'foo.bolt' in the same directory as the current directory. All functions/schema/path variables are considered global.

import {'foo'}; a global module from the node_modules directory with a package name of 'foo' and a default index.bolt

import {'foo/bar'} - Not supported

import {'foo'} as bar - Import a module with all functions/schema/paths with alias bar.

So the above PR is just for the AST parser. If you agree to push it through I'd like to look at implementing the actual JSON processor which is a little more complex. Currently it assumes a Synchronous pipeline and needs to be done asynchronously. Maybe PM me if you have an suggestions but I'm almost in the boat of creating a new gulp-bolt plugin to support streams/async file operations.

Cheers, Jason

[mckoss]

I look forward to checking it out. Sorry, it's going to take me a couple weeks, as I'm spending most of my time preparing for Google IO this time of year.

[cdcarson]

I think imports would be a good option, as would @brewsoftware's idea for a gulp plugin. Here's what I'm currently hacky way of addressing the issue, via gulp. It's slower than running firebase-bolt from the command line, but preserves my sanity.

gulp.task('bolt', function () {
    'use strict';
    var bolt = require('firebase-bolt');
    var plumber = require('gulp-plumber');
    var concat = require('gulp-concat');
    var through = require('through2');
    var rename = require('gulp-rename');
    var gutil = require('gulp-util');
    var PluginError = gutil.PluginError;

    var onError = function (err) {
        var gutil = require('gulp-util');
        gutil.log( gutil.colors.magenta(err.message));
        gutil.beep();
        this.emit('end');
    };
    // modified from https://github.com/firebase/bolt/blob/master/bin/firebase-bolt
    var translateRules = function(input) {
        var symbols;
        var rules;

        try {
            symbols = bolt.parse(input);
        } catch (e) {
            throw(e);
        }

        try {
            var gen = new bolt.Generator(symbols);
            rules = gen.generateRules();
        } catch (e) {
            throw(e);
        }

        return JSON.stringify(rules, null, 2);
    };

    // order the patterns according to what you want first in the concatenation
    // pretty sure this does not matter to the parser (i.e. a function or type can be 
    // declared after it's used), but anyway... 
    return gulp.src(['./rules/functions.bolt', './rules/**/*.bolt'])
        .pipe(plumber({errorHandler: onError}))
        .pipe(concat('rules.bolt'))
        .pipe(through.obj(function (file, enc, callback) {
            var raw = file.contents.toString();
            var parsed;
            try {
                parsed = translateRules(raw);
                file.contents = new Buffer(parsed);
                callback(null, file);
            } catch (e) {
                this.emit('error', new PluginError('bolt', e.message));
            }

        }))
        .pipe(rename('rules.json'))
        .pipe(gulp.dest('./'));

});

[brewsoftware]

Thanks cdcarson, I've pushed a plugin that should be able to handle basic imports and published it in the npm registery:

npm install gulp-firebase-bolt.

I will extend out the use cases over the next week or two but for now it just requires that all .bolt files are in the same directory. If you have any suggestions or feature requests the project is hosted under https://github.com/brewsoftware/gulp-firebase-bolt.

PR's and comments welcome... I'm also waiting on the official update of my PR in this project and have skirted around a large number of integration issues for now..

Cheers, Jason

[mckoss]

Continuing the discussion from (closed) PR #42....

There are several approaches to this problem, with different levels of complexity.

Treat import as file concatenation:

import * from './functions.bolt';

This could be used to factor function and type definitions into multiple files. Disadvantage is that you could have name collisions with local functions and types.

Import with an alias (borrowing TypeScript import syntax):

import * as util from './util.bolt';

path /users is util.User[];

Any imported functions and types would be prefixed with an alias name.

Import individual symbols from a file (syntactic sugar - not essential):

import {User} from './util.bolt';

Adjusting paths

import * as users from './users.bolt' at /users;

This could prefix all path statements in the imported file with '/users' - in this way a bolt file could describe a self-contained application. Note that this won't work universally for the reasons stated at the beginning of this post (although using relative vs. absolute paths can be used to differentiate the two types of uses).

I would recommend starting out with support for either:

import * from './functions.bolt';
OR
import * as util from './util.bolt';

[brewsoftware]

Hi Mike, Just touching base on this functionality (I've been pre-occupied for a couple of weeks now). I'd like to have another stab at the PR for the import syntax with a couple of update (Specifically to match the ES6 syntax properly). https://github.com/firebase/bolt/pull/142 Is mostly complete except for a couple of syntax changes but I can open a new PR with those updates when ready.

There is a secondary issue where the current compiler works synchronously to compile the bolt files. This more related to a re-write of the core compile so I think the debate around how the compiler works for processing import statements can be done separately.

For any example of the re-write I have a gulp plugin that works on the AST using through2 to load the various external modules. How to integrate this back into the BOLT compiler is another discussion that we haven't had yet (Let me know if you want to include it in a Larger PR or if we should aim for 2 smaller PR's).

Also, I agree that ES6/TS is the way to go and I like the suggestion on the re-homing of paths.

Let me know what you think would work for getting this all the way through.

Cheers, Jason

[RyanWarner]

Any updates on this? Would love a way to split my rules across multiple bolt files.

[brewsoftware]

Hey, Apologies for the stagnant contribution. I was ready to go then we have a life event (9 months worth).

I learnt a few things from the Gulp implementation, namely:

• Syntax needs to change to more ES6 style
• Paths are a bit of an issue
• The parser needs to be contextual (i.e. TypeA -> Import TypeB -> Import TypeC, where TypeC should not be directly referable in the TypeA context).

So I have 1 more update to do to the parser which is just the context part above. The compiler is a little trickier to implement properly. Once I have the basic version done I will post back here with a link to the reference implementation. It's been a while so my repo is a little messy and also needs to be cleaned up before merging...

Cheers, Jason

[brewsoftware]

Bump!

I have opened a new PR at https://github.com/firebase/bolt/pull/208 for early comment. Please let me know if you have any comments on the implementation.

The current PR is only for Type Imports and does not cover paths and functions. I will do those in a seperate PR.

Cheers, Jason