biojs / biojs3

Draft of BioJS 3: Web components
BSD 3-Clause "New" or "Revised" License
18 stars 2 forks source link

How should we manage dependencies? #4

Open wilzbach opened 9 years ago

wilzbach commented 9 years ago

If you haven't done so, please read this excellent article as an intro to the problem of dependency management in web components by @Tjvantoll:

http://tjvantoll.com/2014/08/12/the-problem-with-using-html-imports-for-dependency-management/

@Tjvantoll discusses the following options - for convenience shortly summarized here:

Option 1: Use a CDN

<script src="//cdnjs.cloudflare.com/ajax/libs/moment.js/2.7.0/moment.min.js"></script>

The Polymer project is using this approach - here is the distribution guide for 0.5 written by @addyosmani.

Option 3: Feature detection

Summary: using a tiny module loader to detect whether the script has already been loaded

Summary: no reference in the component's code - the dependency is only referenced in the documentation


This also has been discussed at the Polymer project.

Here is my opinion to start the discussion: Option 2 (enforcing a common folder structure) seems to be the lesser evil.

However for dependency conflicts:

I am very much looking forward to your thoughts :)

herkulano commented 9 years ago

HTML Imports are also being contested by Mozilla and they will not implement them. They propose the use of ES6 modules instead.

"Mozilla will not ship an implementation of HTML Imports. We expect that once JavaScript modules — a feature derived from JavaScript libraries written by the developer community — is shipped, the way we look at this problem will have changed." https://hacks.mozilla.org/2014/12/mozilla-and-web-components/

ECMAScript 6 provides the following ways of importing:

    // Default exports and named exports
    import theDefault, { named1, named2 } from 'src/mylib';
    import theDefault from 'src/mylib';
    import { named1, named2 } from 'src/mylib';

    // Renaming: import named1 as myNamed1
    import { named1 as myNamed1, named2 } from 'src/mylib';

    // Importing the module as an object
    // (with one property per named export)
    import * as mylib from 'src/mylib';

    // Only load the module, don’t import anything
    import 'src/mylib';

ES6 Modules Final - Article

Hypercubed commented 9 years ago

I'm hoping for an ES6 module solution. Currently SystemJS and Web Components are very tangential.

Hypercubed commented 9 years ago

@greenify, If I understand correctly the four options above are regarding html imports. But html imports is not a necessity of web components. From https://hacks.mozilla.org/2015/06/the-state-of-web-components/:

We’ve been working with Web Components in Firefox OS for over a year and have found using existing module syntax (AMD or Common JS) to resolve a dependency tree, registering elements, loaded using a normal <script> tag seems to be enough to get stuff done.

Using SystemJS might be something to look into.

herkulano commented 9 years ago

SystemJS as a module loader

Added a new branch for this test: https://github.com/herkulano/biojs3-webcomponent-example/tree/dependency-systemjs

Loading module in .js

System.import('../../d3/d3.min.js').then(function() {
  Polymer({
  ...

https://github.com/herkulano/biojs3-webcomponent-example/blob/dependency-systemjs/biojs-component.js#L1

Loading SystemJS in index.html

<script src="../../systemjs/dist/system.js"></script>

https://github.com/herkulano/biojs3-webcomponent-example/blob/dependency-systemjs/demo/index.html#L10

Dependencies listed in bower.json

  "dependencies": {
    "polymer": "Polymer/polymer#^1.0.0",
    "d3": "~3.5.6",
    "systemjs": "https://github.com/systemjs/systemjs.git#0.18.4"
  },

https://github.com/herkulano/biojs3-webcomponent-example/blob/dependency-systemjs/bower.json#L23

It still depends on bower to install all dependencies, i.e., the end-user still has to use bower to install the dependencies.

Any thoughts?

wilzbach commented 9 years ago

I'm hoping for an ES6 module solution. Currently SystemJS and Web Components are very tangential.

I also find the ES6 module specification quite tempting - especially if we have a loader like SystemJS that supports AMD, CommonJS and ES6 modules. However we still would require a common structure to be able to avoid duplications when loading modules?

It still depends on bower to install all dependencies, i.e., the end-user still has to use bower to install the dependencies.

I have created an example that gets all internal dependencies and Polymer from SystemJS.

https://github.com/greenify/polymer-systemjs-loading

Unfortunately it also doesn't solve the problem that we have if someone imports third-party dependencies like System.import('biojs-core'); as SystemJS would need to have a way to figure out where the module is. So maybe the Polymer approach of using a common structure isn't that bad at all?

Hypercubed commented 9 years ago

@greenify we are working on the same thing.

SystemJS knows where the modules are based on the System.config.js. Using jspm this is automatic.

jspm install d3=github:github:mbostock/d3@3.5.5

then in the module you can:

import d3 from 'd3';
Hypercubed commented 9 years ago

It is additionally complicated if you want to use bundling. SystemJS builder won't bundle into html files.

Hypercubed commented 9 years ago

Here is one way to bootstrap a Web Component using SystemJS: http://plnkr.co/edit/w3L3FB

I used commonjs because I couldn't get traceur working in plunker.

herkulano commented 9 years ago

@Hypercubed That's a really cool example! :+1:

It also shows that developers can use whatever flavour they want and still comply with Polymer > Web Components (in the near future).

Hypercubed commented 9 years ago

Thanks. Here is one more demo before I call it a night: http://plnkr.co/edit/JJp6jp

Here I made a dumb systemjs plugin to do the HTML importing. This allows you to use mappings in the system.config.js (and all module loading goodness). I haven't tested it much so I don't know how robust it is.

I can imagine that it could be a lot smarter and process inline css and js for you. I'm surprised it hasn't been done before.

Hypercubed commented 9 years ago

FYI... https://github.com/Hypercubed/systemjs-plugin-html

herkulano commented 9 years ago

@Hypercubed Amazing! :+1:

wilzbach commented 9 years ago

@Hypercubed I like your approach - it is more secure than mine ;-)

Here a few remarks that I have by looking through your code:

Injecting html via systemjs-plugin-html

AFAIK using this approach is problematic when you bundle everything, as address won't be available them - that is the advantage of the systemjs-text plugin.

exports.fetch = function(load) {
  return importHref(load.address);
};

Injecting css via relative links

We probably can't bundle bar-char-component.css easily if we just use a relative link - or can we?

<link rel="import" type="css" href="bar-chart-component.css">

using systemjs plugins

I guess that we need to agree on all the custom plugins of systemjs that we want to use as for a development setup as we probably want to avoid requiring a bundled version in other packages - we would have hard time to filter out duplicates.

In summary

The biggest difference between our two approaches is that you (1) dynamically add <link> tags for the html and css on the document, whereas I (2) use innerHTML. Your approach (1) is probably faster and more secure, whereas (2) requiring ressources probably is better for a bundling process. I will have a look at best practices about the bundling process - see you soon :)

For convenience I have put my current version onto plunkr, too: http://plnkr.co/edit/CbZXA1?p=preview

Hypercubed commented 9 years ago

You are correct. Bundling wont work. The css plugin has it's own mechanism for bundling that I did not implement. The inline css and js are NOT processed at all by systemjs. At this point, as shown in my last plunker, I'm using the plugin as a promise generator. But maybe with some work...

herkulano commented 9 years ago

@greenify Also amazing! :+1:

Hypercubed commented 9 years ago

Found this... https://github.com/nevir/html-exports/tree/master/dist/sysjs-plugin

Haven't had a chance to look in detail.

herkulano commented 9 years ago

@greenify nice

I've forked @greenify's plunk and added some abstraction as a bio.js library to simplify the module loaders for developers: http://plnkr.co/edit/mEKZ2r?p=preview

Some problems of this solution:

Are we planning to have several versions: ES5, ES6, HTML? Should we have a recommended way and then add flavors for other developers?

UPDATE: I've merged your ideas in this plunker http://plnkr.co/edit/g54ESw?p=preview

Hypercubed commented 9 years ago

For me, as a potential BioJS3 module consumer, I am happy using jspm/SystemJS/ES6. I think, however, npm/Browserify/CommonJS has more traction currently and may be more familiar to BioJS2 users and developers. The Web Components/Polymer way seams to be to forget modules, throw everything into bower and reference via relative path. That approach of course limits what modules you can consume. Would it even allow you to import an existing BioJS2 io module from npm?

That said, I think it is more than possible to create a BioJS3 template that provides all three. Write your code in ES6 and html, generate an ES5/UMD module, generate a HTML Import file that uses relative paths. Provide a package.json file for npm that points to the UMD/CommonJS module, include jspm additions that point to the ES6 module, and a bower.json file that points to the HTML import file.

I think that would be amazing.

Hypercubed commented 9 years ago

@herkulano I don't think we want to be using the System.import directly, this wont work for bundling.

I took your plunker and added and es6 module, a generated (by hand) es5 commonjs file, and a pure web component html import file. Only the ES5 module is being used but should show a way we can provide all three. http://plnkr.co/edit/gwnVq9

Hypercubed commented 9 years ago

Relevant discussion: http://www.codequora.com/questions/25005623/web-components-polymer-and-systemjs

Wondering if we need/want Polymer/HTML imports at all.

wilzbach commented 9 years ago

It is absolutely great to work with you on this :)

For me, as a potential BioJS3 module consumer, I am happy using jspm/SystemJS/ES6. I think, however, npm/Browserify/CommonJS has more traction currently and may be more familiar to BioJS2 users and developers.

Have you seen this great talk by @guybredford (the author of jspm) on how jspm can be used to manage {es6,es5} dependencies from {github,npm} - he also showcases a cool demo with web components!

how does the end-user install local dependencies of a component?

I do imagine two ways:

1) script-tags for the end-user: bundled (optionally minimized) version in /dist for independent inclusion 2) bower/npm components for a developer: unfortunately we can't expect everyone to use es6 and neither to use SystemJS as @Hypercubed pointed out

That said, I think it is more than possible to create a BioJS3 template that provides all three. Write your code in ES6 and html, generate an ES5/UMD module, generate a HTML Import file that uses relative paths. Provide a package.json file for npm that points to the UMD/CommonJS module, include jspm additions that point to the ES6 module, and a bower.json file that points to the HTML import file.

Wow :+1: - I am only a bit afraid that

loss of the css scope by Polymer. @Hypercubed's version solves this and the css as a link tag inside the element doesn't seem to be a problem and it would also be easier to package for distribution. [...] moving the add html outside of the javascript file

(: , but if we decide to go for html imports we would need to find a way to run vulcanize on top of the systemjs bundle. That is why I personally would prefer if the JavaScript version would require the html and css -I need to find a better way than innerHTML tough ;-)

Wondering if we need/want HTML imports at all.

:+1: - wondering about the same!

Hypercubed commented 9 years ago

Yeah, keeping the version numbers in sync between bower.json and package.json is already a hassle. I imagine someone could write a tool that maps jspm package names down to bower relative paths. Maybe some fancy gulp scripts? I think I saw someone made a bower -> jspm upconverter... I'll have to find it again. I would guess jspm -> bower -> html imports would be easier than the reverse. Not sure if it is worth all the tooling. Vulcanizing a SystemJS bundle sounds better. I wonder what @guybedford thinks of all this.

herkulano commented 9 years ago

@Hypercubed looks great!

herkulano commented 9 years ago

Some findings on using JSPM with Polymer

I've added another branch, wanted to try jspm, so couldn't use plunker for this. https://github.com/herkulano/biojs3-webcomponent-example/tree/jspm-test

Polymer Elements
It works well as long as you don't use polymer elements.
When you want to add a polymer element, for instance, a simple paper-button element, because of the structure they have for elements and reliance on bower, you can't use them with jspm anymore.

From paper-button.html

<link rel="import" href="../polymer/polymer.html">
<link rel="import" href="../paper-material/paper-material.html">
<link rel="import" href="../paper-ripple/paper-ripple.html">
<link rel="import" href="../paper-behaviors/paper-button-behavior.html">

Bower
I managed to use bower with jspm with a jspm registry endpoint for bower

Some problems:

wilzbach commented 9 years ago

Hey,

I used our work @herkulano & @Hypercubed to create a jspm example app that uses the following web components as dependencies:

In particular bio-container depends on bio-element, other javascript dependencies like Polymer or lodash (as example) are also injected via the SystemJS loader. For simplicity I avoided ES6 code, but feel free to add it. Also the web component polyfill is now conditionally loaded (needs still some work)

Currently

I guess we only

Feel free to modify the bio-element, bio-container and the example app.

Happy hacking!

Hypercubed commented 9 years ago

Hi @greenify... Looks amazing. Does this line https://github.com/biojs/bio-container/blob/master/main.html#L9 work? If someone imports main.html via HTML import the script tags are treated by the browser, not by SystemJS, therefore I think require will be undefined when ./bio-container.js is loaded. There is a work around of adding a global copy of System.amdRequire but this is AMD not CJS (see https://github.com/systemjs/systemjs/wiki/Module-Format-Support#requirejs-support).

wilzbach commented 9 years ago

Looks amazing. Does this line https://github.com/biojs/bio-container/blob/master/main.html#L9 work? If someone imports main.html via HTML import the script tags are treated by the browser, not by SystemJS, therefore I think require will be undefined when ./bio-container.js is loaded

Yes you are right (and I knew about this), but I couldn't think of a proper way around. Even if we use AMD - the normal Polymer setup doesn't include an AMD module loader like RequireJS - devs would have to manually configure the path :(

herkulano commented 9 years ago

@greenify awesome! :+1:

Hypercubed commented 9 years ago

@herkulano Actually I think using the jspm-bower-endpoint works fine. This sets up a bower_components directory and internal references work, the same as using just bower. The only trouble is you need to ensure that your app is not including Polymer twice. If it does you will get an error. This is working for me:

https://github.com/Hypercubed/biojs3-webcomponent-example/commit/a6d473bfda18116f8fcecb80531c4b19d6848fc7

Hypercubed commented 9 years ago

Sorry, I take that back. I had a bower_components directory leftover from previous tests.

herkulano commented 9 years ago

@greenify some thoughts...

1 There is a problem with encapsulation in your example.

For clarity, let's consider that there is only the container in the https://github.com/biojs/biojs3-jspm-example

<div class="hello">Loading a web component with SystemJS</div>
<bio-container>container</bio-container>

https://github.com/biojs/biojs3-jspm-example/package.json should not have any dependency other than the bio-container. It should not be concerned with the dependencies or inner parts of the bio-container, that should be encapsulated in bio-container.

biojs3-jspm-example/package.json

"dependencies": {
   "bio-element": "github:biojs/bio-container@0.0.1",
 },

So bio-container/package.json should look like this (removed lodash for clarity)

"dependencies": {
  "bio-element": "github:biojs/bio-element@0.0.2",
  "html": "github:Hypercubed/systemjs-plugin-html@master",
  "polymer": "github:greenify/polymer-js@^1.0.5",
  "webcomponentsjs-lite": "github:webcomponents/webcomponentsjs@0.7.2"
},

All elements need to have the polymer and webcomponents dependencies as they can be implemented on their own. so https://github.com/biojs/bio-element/package.json should look like this:

"dependencies": {
  "polymer": "github:greenify/polymer-js@^1.0.5",
  "webcomponentsjs-lite": "github:webcomponents/webcomponentsjs@0.7.2"
},

2 Another thing is the main.html and main.js, they are interesting ideas, but there are some conflicts.

As far as I understand the idea of the https://github.com/biojs/bio-element/main.html is for compatibility with bower, which is great. But then you have a duplication of loading Polymer both in the main.html and bio-container.js

main.html

<link rel="import" href="../polymer/polymer.html">
<link rel="import" href="../bio-element/bio-element.html">
<link rel="import" href="bio-container.html">
<script src="bio-container.js"></script>

bio-container.js

var Polymer  = require('polymer');

This can be easily solved by moving Polymer.js loading to main.js and managing all loading dependencies there.

main.js

require('webcomponents');
var Polymer  = require('polymer');

require('bio-element');

require('./bio-container.html!html');
module.exports = require("bio-container");

3 Does this work if it's loaded from main.html? I suspect it doesn't, but it should be tested.

var BioContainer = Polymer({
...
});
module.exports = BioContainer;
herkulano commented 9 years ago

@greenify

I've forked your examples and made the changes mentioned in the previous comment here: https://github.com/herkulano/biojs3-jspm-example https://github.com/herkulano/bio-element https://github.com/herkulano/bio-container

Everything works as expected, jspm takes care of sub-dependencies and all elements work on their own. I created an index.html to test the components in isolation.

One thing that I still haven't tested yet is the main.html compatibility with bower and other polymer components.

wilzbach commented 9 years ago

I've forked your examples and made the changes mentioned in the previous comment here

This is great - do you want to merge your changes back to the biojs components?

The only thing that I would change back is the dependency on the polyfill, including it in the components makes it very hard to remove the polyfill for browsers like chrome. Imho it should be only included in the bundling app.

One thing that I still haven't tested yet is the main.html compatibility with bower and other polymer components.

As discussed above we doubt it works without some hacks.

requiring Polymer in main.js

I do see your point that you want to have these calls in the main.js file. However loading Polymer there only works because Polymer is also a global once it is loaded. It won't work for other, normal dependencies :(

herkulano commented 9 years ago

The only thing that I would change back is the dependency on the polyfill, including it in the components makes it very hard to remove the polyfill for browsers like chrome. Imho it should be only included in the bundling app.

Makes sense, hopefully it will disappear over time.

I do see your point that you want to have these calls in the main.js file. However loading Polymer there only works because Polymer is also a global once it is loaded. It won't work for other, normal dependencies :(

If we don't care about working well with bower and other polymer elements we can then remove the main.js altogether and do everything in the element.js

I'll make the changes, happy to merge back the changes after you OK :)

herkulano commented 9 years ago

I've made the changes and created the PRs.

This still has the question of how to build / vulcanize it for production?

I tried to build it and it throws this error

SystemJS Builder - Plugin for github:herkulano/bio-element@0.0.9/bio-element.html!github:Hypercubed/systemjs-plugin-html@master does not support sfx builds

Does jspm try to build also html, text, css, etc dependencies through its plugin? It might be worth investigating it.

UPDATE: jspm does support building for plugins, like css https://github.com/systemjs/plugin-css#builder-support

wilzbach commented 9 years ago

I've made the changes and created the PRs.

Highly appreciated! Just three quick questions - I hope you don't mind:

1) Why do why need this in the index.html?

// Use Shadow DOM if available
window.Polymer = window.Polymer || {};
window.Polymer.dom = 'shadow';

2) Why do we need the host selector in the CSS?

:host .hello

3) I assumed you accidentally pushed the build.js, so I removed it - I hope I am not mistaken ;-)

Does jspm try to build also html, text, css, etc dependencies through its plugin? It might be worth investigating it.

Yes that it does e.g. for the css, or html - this why I initially used the html plugin and innerHTML ;-)

If we don't care about working well with bower and other polymer elements we can then remove the main.js altogether and do everything in the element.js

Unfortunately this is seems to be our general consensus for the time being ...

wilzbach commented 9 years ago

Update: I started to create a small Web Component bio-element-vanilla to showcase that non-Polymer Web Components just work fine. As our discussion has grown quite long I created a new issue.

Hypercubed commented 9 years ago

The html plugin contains code to skip building the html imports. Possibly doesn't work when sfx builds. The plugin probably does need to use something like innerHTML eventually, especially if we want SystemJS process the JS. I should look into how wc works. It doesn't just append the html to the DOM, it creates and adds a new sub document.

Hypercubed commented 9 years ago

I'm beginning to have the feeling biojs should do one of the following:

A) Adopt polymer methods, use only html imports and bower (losing all systemjs goodness)

Or

B) Skip polymer and create a custom js polymer like biojs web component lib. (Maybe what @greenify was getting at in his vanilla example)

herkulano commented 9 years ago

@Hypercubed

I agree as I said before it's getting convoluted and the tech is still immature.

Having said that, I've augmented your SystemJS plugin with support for SystemJS Builder.

Have a look here: https://github.com/herkulano/systemjs-plugin-html

herkulano commented 9 years ago

@Hypercubed @greenify

SystemJS Plugin HTML now also minifies the css and injects it into the dom-module, so there are no longer css dependencies after the build command. All html and css files are packaged in one single file: elements.html.

https://github.com/Hypercubed/systemjs-plugin-html/pull/1

Consideration: the build doesn't inject HTML import links inside the html, it only looks for css imports. So it doesn't work with Polymer as it is distributed currently, it only works with greenify/polymer-js, using Polymer without HTML Imports.

Hypercubed commented 9 years ago

I think for building individual components Polymer/vulcanize is a better option. systemjs-plugin-html is still useful for importing into an application and can work using bower paths.

herkulano commented 9 years ago

@Hypercubed

AFAIK, vulcanize works only with HTML Imports, so it will not work with the current setup of JSPM and SystemJS, thus the need for a build step inside the plugin itself.

B) Skip polymer and create a custom js polymer like biojs web component lib. (Maybe what @greenify was getting at in his vanilla example)

I agree, there's actually no need to use Polymer if we're not going to use any of its components.

herkulano commented 9 years ago

Also made some comments to this here: https://github.com/Hypercubed/systemjs-plugin-html/pull/1#issuecomment-120789590

herkulano commented 9 years ago

We should decide which way to go:

  1. Polymer HTML + Bower + Polymer/vulcanize (Polymer's official way)
  2. Vanilla + SystemJS + JSPM Build
  3. Polymer JS + SystemJS + JSPM Build

I'm sure we can find a lot more ways of doing this and we are still learning and testing these things, but it would be good to focus on one particular way, get that one right, and then add other possible solutions as we go along.

AntonPetrov commented 9 years ago

From the options outlined by @herkulano I would personally go for the official Polymer way, as it is likely to be well documented/supported and many people are already familiar with Bower.

Also, some Polymer components can be useful for building widgets, for example, iron-ajax.

herkulano commented 9 years ago

We should also take this to the wider community and talk to the people that are actually going to use BioJS.

Hypercubed commented 9 years ago

As a developer I currently prefer jspm and SystemJS but there is nothing that prevents me from using bower and jspm simultaneously in an application. In fact systemjs-plugin-html would work just fine in this setup. What I think you lose is that many of the existing BioJS modules are currently npm commonjs (node) modules. What about BioJS IO modules? Are you going browser only?

wilzbach commented 9 years ago

I don't see a reason why 2 & 3 should conflict with each other. As long as polymer and our plugins are defined as dependencies we should be able to use jspm to bundle it. However at the moment we won't be able to include the polymer components.

My experience over the last year tells that scientist are super lazy to update their code if they ever do so - it's already a huge step for them to share their code with others. That is why I think having proper dependency management (what jspm does and bower sucks) is an essential requirement for us.

So that is why I personally vote for focussing on jspm way as main dependency manager.

On July 13, 2015 12:31:23 PM GMT+02:00, Herculano Campos notifications@github.com wrote:

We should decide which way to go:

  1. Polymer HTML + Bower + Polymer/vulcanize (Polymer's official way)
  2. Vanilla + SystemJS + JSPM Build
  3. Polymer JS + SystemJS + JSPM Build

I'm sure we can find a lot more ways of doing this and we are still learning and testing these things, but it would be good to focus on one particular way, get that one right, and then add other possible solutions as we go along.


Reply to this email directly or view it on GitHub: https://github.com/biojs/biojs3/issues/4#issuecomment-120886715

Hypercubed commented 9 years ago

We should also consider that polymer team is likely going to change their strategy soon. There is talk of using npm3: https://github.com/Polymer/polymer/issues/326