Publish sub-projects on npm, add them to package.json.

[timoxley]

Problem: The build process for polymer is clunky.

Solution: Publish individual components on npm, set up dependencies through each component's package.json, let npm manage gathering and building dependencies through npm's prepublish scripts.

This would get rid of this pull-all.sh hackery, allow pieces to be used in isolation, consolidate dependency management (you're already using npm to manage dev dependencies) as well as make it a more idomatic js project.

[samccone]

@nwwells please remember that is an open source effort run by other humans, so please be respectful when communicating.

Any brainstorming or even prototypes of ways to solve this, such as what @addyosmani put together would be immensely helpful.

Thanks for your interest, and looking forward to any future ideas you may have :v:

[justinfagnani]

@nwwells I'm sorry that you're reading this as saying that we don't want to support you. We do.

From my point of view the problem here is exactly what to support. As far as I can tell, here's no standard plug-and-play standard npm configuration that supports browsers like there there is with Bower. Every npm project uses npm + other tools to get the job done, and there are a great many variations on those other tools. It seems like the required practice is to use npm to get the bits to disk, then build steps to massage and bundle the bits to be loadable and usable in a browser.

In my projects where I'm a user of Polymer and npm libraries, I just add Bower to the mix to get the Polymer (and some other frontend packages) bits, and roughly the same build steps to prepare things for the browser.

Here's how I've thought of the various users of Bower and npm, before and after adopting Polymer:

Before Polymer: bower -> usable project npm + build steps -> usable project

After Polymer: bower -> usable project npm + bower + build steps -> usable project

The observation here is that npm users already have and understand a more complex checkout and build process, and presumably can integrate Bower to pull in Polymer. Maybe it's the assumption that adding Bower to an npm project (that is already integrating several tools by necessity) is acceptably small extra step that's wrong. If so, I hear you, and we've been hearing this much louder and more frequently recently. This is why it's on our short list (besides having wanted to figure out how to best support npm already) to figure out.

Do you have any opinions on the option for starting of just publishing our packages, without necessarily having the whole workflow figured out? That would at least let npm users get rid of Bower and have a single dependencies list. I do think that the perfect has been the enemy of the good enough here, and this could be a way to make progress and get feedback.

[jokeyrhyme]

I'd suggest publishing to a scope, and asking the npm registry folks to show all public packages in search results, regardless of scope.

[ericelliott]

Do you have any opinions on the option for starting of just publishing our packages, without necessarily having the whole workflow figured out?

I think this is a good strategy. Also, I don't believe that Bower was ever necessary. I believe that a simple, plug and play solution could have easily been built on the foundation that npm provides.

npm allows you to create custom fields for any metadata that might have been required, and task runners for any build steps necessary to shuffle things to where they need to be. You could even create a simple CLI to hide the details from users who don't want to peek under the hood and see that maybe there's a "build:polimer" npm script getting run on install.

I also agree that refusing to do something because you think most of your users won't understand it does a great discredit to most of the potential users of Polymer.

"Front-end" engineers are not simply a large group of idiots incapable of figuring things out.

The key is to make it as easy as possible, but not so easy as to make it tricky to integrate Polymer into a typical JS project using ES6 modules & npm.

Let's just call it like it is: Polymer made a bet that "front-end" modules had special needs that were not required of Node modules, and that it would be difficult to built front-end software using npm as the package manager. The success of tens of thousands of projects using npm for both front-end and universal JavaScript projects proves that the bet was wrong.

In the Universal JavaScript ecosystem, there is no front-end/backend. There is only universal.

[ericelliott]

I'd suggest publishing to a scope, and asking the npm registry folks to show all public packages in search results, regardless of scope.

I agree with this. They should filter search based on public vs private, not based on scope.

[oleersoy]

With everything that has been said so far, there is still a very tight bottleneck ahead for anyone who wants to build / prototype / experiment / extend polymer.

Suppose you design a new Calendar element / component. It pulls in a bunch of other polymer components. You publish it.

I try to use it in a broader more aggregated project and after running: npm i your-calendar

I get several warnings that the component and several of the sub components are incompatible with some of my main components and their sub components. This is a guaranteed frequent scenario and it's going to put a massive drag on the entire ecosystem. There's a reason why Eclipse runs on OSGi.

[jokeyrhyme]

I get several warnings that the component and several of the sub components are incompatible with some of my main components and their sub components.

@oleersoy: this crops up with any component-first framework, not just Polymer. Polymer has a leg up here because it has such an enormous first-party component library, compared to any other framework. I think component version compatibility is the least of Polymer's problems currently.

[nwwells]

@samccone I'm sorry if I came off as disrespectful. My intent was to be respectful, an intent I thought I had accomplished. I hope you read the following message as respectful also. I think what you're calling disrespectful is simply me expressing frustration. As noted at the top of my comment, I have nothing but respect and admiration for what the polymer team has accomplished.

@justinfagnani The problem with the npm vs bower debate is that it's comparing apples to oranges. Npm is a package manager with a little bit of tooling to make it easy to run scripts, including build tools. Bower is a package manager + a build tool. Many people choose npm (including myself) because it is independent of the target environment (browser or server). It is this, in fact which makes me value npm so highly: it maintains the SRP, mostly.

Bower doesn't. That's not to say it's a bad tool or the people that use it or made it are bad. It's a purpose-built tool, which is great. There are many reasons I could be persuaded to use bower, in fact. Be that as it may, it isn't always the right tool, and even if it is, someone who's myopic about npm may simply have the power to override everyone's best judgement and pick it. That's not right, and it's not good, but it's often reality.

With this (my perspective) as backdrop, let me walk you through my reaction to your statements:

As far as I can tell, here's no standard plug-and-play standard npm configuration that supports browsers like there there is with Bower.

That's right, because if there was, it would be a violation of the SRP.

It seems like the required practice is to use npm to get the bits to disk, then build steps to massage and bundle the bits to be loadable and usable in a browser.

Yes. Because the appropriate choice of build system is specific to each project.

The observation here is that npm users already have and understand a more complex checkout and build process, and presumably can integrate Bower to pull in Polymer.

Here's where you lose me. Just because I understand a more complex build process means that I am required to have two package managers if I want to use Polymer?

Maybe it's the assumption that adding Bower to an npm project (that is already integrating several tools by necessity) is acceptably small extra step that's wrong.

There you go. yes. Here's the difference: new clones of my project used to just run

git clone $project
npm install
npm start

Now they have to run

git clone $project
npm install
npm start
# build failure. Maybe if the build system has really good error messaging
#      it tells them, "hey, did you install bower?"
# confusing debugging time because aforementioned error messaging probably doesn't exist.
# eventually give up and ask someone else, which causes _them_ to context switch
# "Oh, yeah. You've got to bower install"
# "What's bower?"
# "Oh, it's a package manager"
# "You mean like npm?"
# "Yeah"
# "Well, why do we have 2?"
# "Well, polymer, which is this awesome web component driven framework 
#         requires that we use bower, hopefully just for a little while longer."
# "That's dumb"
# "Yeah, but we deal with it."
# "OK. w/e"
npm install -g bower
bower install

If so, I hear you, and we've been hearing this much louder and more frequently recently.

Great! I hope you don't hold it against me that I'm belaboring the issue. I feel bad about doing it actually, but there still seems to be some confusion about what an ideal workflow on npm + (insert build tool here) looks like. I'm happy to provide examples and answer questions about it.

Do you have any opinions on the option for starting of just publishing our packages, without necessarily having the whole workflow figured out?

Yes, that's a great start. In fact, that solves this issue from my perspective. Ideally you wouldn't have to use a scope, but that's a problem that can be worked on in parallel.

After you can get any polymer package on npm, then we can solve integrating with build systems, whether that's browserify, webpack, or whatever. If the polymer project governance (i.e. "You people") decide that browserify (or w/e) is a pain, or can't be supported in a general way, then great. We (the devs) can handle that on our own.

Read on for rants about the definition of "community". You have been warned.

---

Finally, (totally off-topic for this issue) I wish there was a lot less "us" and "them". It seems really off-putting to me. I'm a member of the community, which means I'm part of the team. I'm one of those "other humans". The React community does this very well. In fact, I was politely, but firmly corrected on one issue I commented on for implying that "they" (meaning the Facebook devs who maintain/support React) weren't paying attention to it. A member of the community (non-Facebook employee) said something like, "Well, if it bothers you so much, submit a PR. These people don't owe you anything."

So, in the spirit of community cooperation (:D), am I clear about my use case here? Any questions about the role of npm?

[miztroh-zz]

I don't see a problem offering npm as another delivery method, but I'd hate to see it replace bower. npm installs / updates always seem to take forever. I get the difference between flat / nested dependencies, but still. That's been my experience as someone who rarely needs more than what bower provides.

[nwwells]

@miztroh, I totally agree with you.

[nwwells]

OK. I have a confession to make. I just did a huge npm land grab. I cloned all the elements currently in the catalog, converted the bower.json into something compatible with npm's package.json, and then published them. Ran into the aforementioned conflicts:

• google-analytics
• google-calendar
• google-feeds
• google-sheets
• paper-button
• paper-checkbox
• paper-ripple
• paper-slider
• paper-tabs

After seeing that some of those projects are just node utilities that probably won't sacrifice their nice real estate on the altar of polymer, I wondered if I could simply prepend all the package names with polymer-. Turns out none of those project names are taken!

Well, they are now.

So, I think the polymer- prefix is probably the best move. It namespaces, but doesn't have the negative side effects of scoping (search works... yay!). I'm sure that we can automate the generation of the package.json (... I just did), so I don't feel like that should be a huge concern.

Thoughts? I am, of course, willing to hand over any/all of these modules to the right people. Heck, I'll even delete them if that's what the community wants. I just didn't want to let any more of the names get into the wrong hands.

[oleersoy]

@oleersoy: this crops up with any component-first framework

@jokeyrhyme the first question is does it need to?

[jokeyrhyme]

@oleersoy we can avoid dependency version conflicts by never exposing a public API surface, and/or never using modules. I think we're already past this point though. :P

[robdodson]

Thank you @nwwells for this thoughtful followup.

One bit jumped out at me:

After you can get any polymer package on npm, then we can solve integrating with build systems, whether that's browserify, webpack, or whatever. If the polymer project governance (i.e. "You people") decide that browserify (or w/e) is a pain, or can't be supported in a general way, then great. We (the devs) can handle that on our own.

I wrote a thing a few weeks ago to integrate Polymer with browserify and figured I should put it out there in case anyone in the community finds it interesting. It's called spock-cli and it lets you use ES2015 or CommonJS modules inside of your Polymer elements. It is by no means a fully baked, production-ready thing, but it's fun to play around with the ideas :) @justinfagnani, @samccone, @addyosmani, and myself have been kicking around a plan to take some of the features of spock and roll them into a first-class build tool for Polymer. That work will start soon on GitHub so everyone can participate.

Finally, (totally off-topic for this issue) I wish there was a lot less "us" and "them". It seems really off-putting to me. I'm a member of the community, which means I'm part of the team. I'm one of those "other humans".

We totally agree and there's been a lot of discussion recently within the team to figure out how to fix this. Our plan for 2016 and beyond is to give more ownership and control back to the community. So when someone opens an issue, instead of letting it just sit there, we ask the creator to open a PR and offer to help them through the process. I think Sam and Addy have set a good example with Polymer Starter Kit, which is largely driven and maintained by community contributions at this point. We want that same level of community ownership for pretty much everything that falls under the Polymer org.

So yeah, big +1 to everything you said!

[chuckh]

@nwwells thanks for setting up npm polymer- elements. I tried to install few different polymer- elements using:

• npm i polymer-iron-icon
• npm i polymer-paper-input
• npm i polymer-paper-button
• npm i polymer-paper-toolbar

But on all of them getting the below errors: (using node v5.4.1 and npm v3.5.3)

npm ERR! git rev-list -n1 %5E1.0.0: fatal: ambiguous argument '%5E1.0.0': unknown revision or path not in the working tree.
npm ERR! git rev-list -n1 %5E1.0.0: Use '--' to separate paths from revisions, like this:
npm ERR! git rev-list -n1 %5E1.0.0: 'git <command> [<revision>...] -- [<file>...]'
...
npm ERR! Command failed: git rev-list -n1 %5E1.0.0
npm ERR! fatal: ambiguous argument '%5E1.0.0': unknown revision or path not in the working tree.
npm ERR! Use '--' to separate paths from revisions, like this:
npm ERR! 'git <command> [<revision>...] -- [<file>...]'

Has anyone been able to install any of these ok?

Doing npm i npm-polymer-elements worked fine.

[ChadKillingsworth]

I didn't know this discussion existed - I was watching #2578.

The Polymer project has continually felt like it caters to the least common developer at the expense of the advanced user. Reading this issue continues to reinforce that idea.

Publish to npm. Bower causes real heartache. Let me list the ways:

• Uses github as a CDN
• Requires built files to be committed back to the repo (yuck)
• Offers a "flat" dependency structure - without telling you of the hell that will ensue maintaining resolutions.
• Has been known to randomly corrupt downloads
• No good way to have an internal registry
• Has only 1 core dev that has no plans to address these issues

I will second the motion that you publish first, figure out the build story later. I can figure out my own build story - I already do. From my other thread:

Because Polymer is only distributed as an HTML import via bower, I'm having to run through a build tools gauntlet to convert it to a standard JS script: vulcanize for single import, crisper to factor out into separate script, custom rule to re-add license, etc.

I can offer you an easy migration path: write a custom bower resolver. It will allow your bower installs to at least work off the npm registry. I'm the author of the bower-sinopia-resolver because of this exact same problem. I'll even be willing to help.

[nwwells]

@chuckh I need to apologize! I had replied on Friday, but apparently the message didn't get through to github. Here's basically what I said:

Yeah, they won't work until we properly convert the bower.json files into npm package.json files. I think that's the next step, some sort of bower2npm translator. Hopefully that's not too difficult, and I've got kind of a mental start on it:

1. prepend polymer- to the name.
2. remove any "private" key, if there is one.
3. remove the "main" key, as it's relatively useless in this context (bower main and npm main simply mean different things
4. translate dependencies to npm dependencies rather than bower repo references.

That last, I think, is the reason why the current ones fail.

[chuckh]

@nwwells np :smile:. Your plan makes sense. Let me know once you have given it a shot and I will try testing it.

[justinfagnani]

We've started publishing packages to npm under the @polymer scope. Polymer core is here: https://www.npmjs.com/package/@polymer/polymer

I'll have the elements up tonight or tomorrow.

[justinfagnani]

Ok, Polymer and all the elements are up (except for paper-listbox for some reason). You can see the list here: https://www.npmjs.com/~polymer

You can install packages with npm i @polymer/paper-input etc., and use them in html like:

<!doctype html>
<html>
  <head>
    <link rel="import" href="node_modules/@polymer/paper-input/paper-input.html">
  </head>
  <body>
    <paper-input label="Testing"></paper-input>
  </body>
</html>

This works because all the Polymer-team packages are in the same scope. We'll need some polyserve and/or gulp goodness to make relative imports work across different scopes and unscoped packages.

[ericelliott]

I would suggest npm scripts rather than gulp. Not everybody wants to use gulp, but all npm users have npm. ;)

[justinfagnani]

Oh, and a quick note on versions.

I accidentally published @polymer/polymer with some bad dependencies, so I unpublished it, not knowing that I wouldn't be able to re-publish it with the same version. So that burned v1.2.4.

Then I decided to publish every package with a v0.0.1 so that I could rev packages quickly if the bower.json -> package.json translation had problems. @polymer/polymer had already been published though, so I set its version to 1.2.5-npm-test.1 so that we don't stomp on the 1.2.5 version. Once we try out a few more tests, push the package.json commits, and release Polymer 1.2.5 or 1.3.0, I'll republish all the elements with their real version.

I updated our repo management tool to generate package.json from bower.json, using bower.json as the source of truth, but I published with some simple shell commands. Soon I'll update the tool to tag a release in github and npm publish at the same time so that we can guarantee that we have the exact same versions available in bower and npm.

[justinfagnani]

@ericelliott users will be able to use whatever they want. Building files for loading by browsers won't be prescriptive, but we'll have some demos showing some of the myriad ways it could be done.

[jokeyrhyme]

Now that HTML Imports is basically dead (only natively implemented in Chrome and likely to be this way forever due to ES2015 modules having more vendor favour), I wonder if we could just bake a custom module resolver into the HTML Imports polyfill, and use the polyfill even in Chrome?

What if instead of

<link rel="import" href="node_modules/@polymer/paper-input/paper-input.html">

we used

<link rel="polymer-import" href="@polymer/paper-input">

and let our custom HTML Imports resolver figure out the right thing to do. This could check both node_modules and bower_components, it could check relative paths, anything that makes sense.

This could make life simpler for both npm and bower users.

[justinfagnani]

@jokeyrhyme HTML Imports are far from dead. They were discussed at the latest standards meeting, and now that <script type="module"> and JS module loading specs are progressing, there's movement on unifying the loaders.

Regardless, you don't want the browser checking multiple paths for a module like node does. The resolution of a module name or path should be done purely on the client.

[rictic]

Yeah, HTML Imports are quite compatible with ES Modules. They've had a low priority in the standards meetings so far mostly because they're super easy to polyfill and everyone's been waiting for ES Modules to get sufficiently hashed out. I believe they're on the docket for the next web components standards meeting.

[ericelliott]

@justinfagnani

@ericelliott users will be able to use whatever they want.

Great news. =) Thanks for your work on this. It's much appreciated! :+1:

[jokeyrhyme]

Regardless, you don't want the browser checking multiple paths for a module like node does. The resolution of a module name or path should be done purely on the client. -- @justinfagnani

Previously:

After Polymer: bower -> usable project npm + bower + build steps -> usable project -- @justinfagnani

I only suggested having client-side check multiple paths because you seem to value having the shortest path possible to "usable project", e.g. clone, npm install, open in browser

I would hope that nobody is deploying non-vulcanised Polymer projects in production, and I would want the production-mode resolver to short-circuit any guesswork. But performing multiple checks seemed like it would be okay for developer-mode.

In any case, it's great news that there is a unification effort behind the scenes. :D

[ChadKillingsworth]

This is awesome news!

[TimvdLippe]

Given the complexity of the task to publish all projects on NPM as well as the various arising problems to successfully complete the transition, would it maybe better to create a separate repository for the issue management of all problems? I have been keeping an eye on this thread and I think keeping it in just 1 thread makes it hard to keep a complete overview of the progress and accompanying issues. Moreover, in such a repository, a guide or usage manual can be created on how to use Polymer with just NPM. My 2 cents.

[justinfagnani]

@TimvdLippe at least at first, I think additional repos bring discoverability issues. We can open general npm issues on the main repo, any package specific ones on the respective package repo, and any tools we develop to help with npm will have their own repos as well.

I'm sure we're open to different organizations in the future if this doesn't work.

[nwwells]

So, obviously some decision was made against the polymer- naming. Shall I just delete those packages, then?

[justinfagnani]

I think that would be best. Thanks for your help and cooperation @nwwells !

As for the decision to use a scope, @polymer/ gives us the ability to add new packages using the prefixes we already have (paper-, iron-, gold-, platinum-, etc.) without the risk of future name conflicts. Prefixes only do that by convention, yet have the same path problems that scopes have. We're a little bt on the bleeding edge by using scopes, but I hope that npmjs.com scope support ramps up quickly. We're following issues like: https://github.com/npm/download-counts/issues/23

[justinfagnani]

I pushed a few new versions of the elements to fix some issues I had came across while making a demo.

The demo: @notwaldorf's awesome <emoji-selector> converted to use npm, can be seen here: https://github.com/notwaldorf/emoji-selector/pull/5

I preserved the ability to use the package from Bower, since there's no hard-coding of the components directory, and matched the layout of bower_components/ in browser_components/. To serve the demo I used a patched version of polyserve that allowed me to set the component directory to something other than bower_components/, in this case browser_modules/. I'll submit that patch asap.

One general class of problems that will be common are packages that are laid out differently in Bower vs npm, which I fixed case-by-case.

Note, that this is only one way to use npm packages. Again, this is not prescriptive, just a demo.

If anyone here wants to try converting other projects it will be very helpful in finding holes in our packages and related packages.

[nwwells]

@justinfagnani done.

[justinfagnani]

Thanks @nwwells!

[justinfagnani]

polyserve 0.6.0 is now published, with the -c and -n options to specify the component dir and package name which were previously only read from bower.json. You can run the npm version of <emoji-selector> with polyserve -c browser_modules -n emoji-selector. Then navigate to http://localhost:8080/components/emoji-selector/demo/index.html

To load the documentation you currently have to run both npm and bower install because the npm version of hydrolysis doesn't include all the files needed for the browser.

[abdonrd]

Nice work @justinfagnani!

[stramel]

:+1:

[MeTaNoV]

+1

[leodido]

+1

[jordanaustin]

When will you be publishing v1.3.1 to npm? I noticed the package.json version was bumped but still seeing v1.2.x on npm.

[addyosmani]

@justinfagnani is Tedium currently setup to publish sub-projects to npm or does this require a manual publish to address the above request? Would be great if we could keep this automated so versions stay in sync everywhere.

[ChadKillingsworth]

ping - 1.2.5-npm-test.2 is still the latest version on npm.

[jordanaustin]

I can see the package.json file is being kept up-to-date but you aren't publishing the updates to npm. Bower has 1.4.0 but npm only has 1.2.5-test.2.

Can you publish the same versions on both bower and npm, we need to keep these in sync.

[robdodson]

Yep, just spoke with @tjsavage who said we'll add it to our launch checklist

[nschipperbrainsmith]

@robdodson Is there any time indication for when you expect the new version to be published to NPM? Or should it be done shortly?

[tjsavage]

ASAP - @justinfagnani WDYT?

[jordanaustin]

@tjsavage @justinfagnani since you've already taken the steps to get it on npm it seems like keeping npm in sync with polymer releases should be pretty trivial. As soon as you made these packages available on npm our company (and I'm sure many others) moved to installing polymer from npm, however we've been stuck on v1.2.x due to no updates on the npm package for 2 months.

Do you foresee this being an issue moving forward or is the plan to maintaining the same versions across bower and npm in the near future?

Thanks for your attention on this!

[justinfagnani]

Unfortunately nothing's trivial across 150 repos and a dozen developers. Since we use bower, our release process consists bumping version in one file and making a release in Github. Adding npm means we need to keep package metadata in sync and perform two different release steps. We're writing a tool to automate this, along with test integration, so that they don't go out of sync in the future.

The current set of packages was pushed mostly to see if things would work and find issues in related packages. When we're ready to actually support the packages fully going forward, I think we'll make a loud announcement about it. I'll work on updating the current asap, hopefully today, and keep this issue update on the progress of our release tool.