support modules in pjs environment

[kevinbarabash]

@ChrisJPhoenix @MikaalSky and I were discussing this on our brand new gitter channel https://gitter.im/Khan/live-editor. I've tried to summary as well as add some additional concerns and information about future integration with webapp.

We'd like to eventually allow users to import code from other scratchpads. As a first step we'd like to support modules within a scratchpad. This will allow users to break their programs into one or more application and or library modules.

Modules would export code using the CommonJS standard module.exports = { } and exports.property = ... patterns and would import code using var A = require("./lib_a.js");.

Out of scope for initial support would be importing code from another scratchpad. This is because we'd want to be able to pin the version of a scratchpad that a person is importing from so that the code doesn't change from underneath them if the original author makes a change. This is also necessary for importing code from your own scratchpads because we'll probably want "spin off" to only spin off a project and not all its dependencies because that would get messy quite quickly. Before we can handle any of this we will need a way to surface information about versions to users as well as allow users to switch between versions.

For the UI we could have some sort of tab interface where each tab would have the "filename" of the module. If there are too many modules to display all of the tabs we could use a list modal to switch to a file that is out of view. That new tab would come into view and the least used tab would be moved out of view.

For the data storage we'll have to change how scratchpads are stored. There is an internal server end point which returns a Scratchpad entity as JSON. There is a "revision" property which contains a ScratchpadRevision entity as JSON which contains a "code" property which is a string. I think it makes sense to update this to be a JSON object where the keys are filenames and the values are the contents of each of the files.

Although we can make changes to live-editor to handle "code" that's a JSON object and display multiple tabs, we'll have to put this functionality behind a flag until webapp can be updated. It's probably easiest to add a new property to ScratchpadRevision to store the new "modularCode". This change will require updating webapp in a lot of places. This work will require someone at KA to make those changes but it's hard to say when resources will be available to make those changes. Simply keeping stuff on a branch until we're ready to make those changes doesn't work either because that branch will die. That's why hiding the code behind a flag is necessary.

[ChrisJPhoenix]

I'll be out of town the rest of the week so can't really engage till next week.

One quick thought: Seems like defining a magic comment would allow a pure-front-end to break up a single text buffer into multiple tabs, making it really easy to develop the UI separately from any other code dependencies. So work on the tab UI doesn't have to wait on anything.

Second quick thought: Seems like there's some UI design necessary - e.g. how to let the user name the tabs. See you Monday (or maybe Wednesday...)

[bytorbrynden]

I think it makes sense to update this to be a JSON object where the keys are filenames and the values are the contents of each of the files.

While that might be the easier way to do this, I think it'll start to consume too many resources over time... What if you could just require a Scratchpad ID (this could be changed at some point, I'm just brainstorming), and then, when the program is saved, include those in the ScratchpadRevision object, as an object, called dependencies, then, whenever the scratchpad begins to load, import that code (invisibly), into the current scratchpad.

The only thing with my idea, is the live-editor will have to make a n more asynchronous requests; where n is the number of dependencies...

[kevinbarabash]

@ChrisJPhoenix if we used magic comments as module separators it means more work for us because now we have to write a parser to do the splitting. The nice thing about using JSON is that there's already parse/stringify methods that we can use. I'm not sure how it would make development easier because we'd still need to implement "require" before the code would even run. Work on the UI won't have to wait on anything as long as we put it behind a URL parameter flag.

@Gigabyte-Giant If you're importing code from another Scratchpad that makes sense, but if you're creating a program and you want to have separate modules, the overhead of describing those modules within a JSON structure will be dwarfed by the size of the code in most cases.

[petercollingridge]

This would be very useful, importing from external scratchpads even more so. Personally I don't think it's necessary to pin version of scratchpads. You should just encourage people to spin-off from the code they want and then import that instead (or only allow people to import their own code). That way people still get the Inspiration badge when someone uses their code. It might be useful if spin-offs showed whether they were different from the code they were spun-off from, so you see when a new version of the code you copied was released.

[benburrill]

At the expense of some storage, you could vastly simplify the whole version problem by giving users the ability to either create their own module for the project or fork an external module for use in their project. This has the added benefit of allowing users to modify the module for their specific project. In the UI, forked module tabs could include a way to update them to the latest version along with some form of attribution stating the original author(s) + the projects author if they modified it.

I'm not sure what to do with the scratchpad API. If you separated the entry point of the program from the modules, you might be able to keep the current structure, just with the addition of a modules object.

Anyway, the module idea would be incredibly useful if it was implemented. I have wanted something like that for a while (coincidentally, I created a simple placeholder module system for the pjs environment around the time this issue was opened)

[codeHusky]

Save as Module and then loadModule functions would make sense as well.