search

OpenUserJS / OpenUserJS.org

The home of FOSS user scripts.
https://openuserjs.org/
GNU General Public License v3.0
857 stars 305 forks source link

Establish a build/exec/deploy system #249

Open Martii opened 10 years ago

Martii commented 10 years ago

Fork a command line minifier for dev tools on the OpenUserJS account

This will be needed in a while to minify what we want to before deploying along with a deploy.sh that is kept with sizzle and maybe here... depends on the secrets that need to be kept.

Dev and GH source should remain unminified at all times... only time minified should be existing is on deployment/testing... this may require some extra time with making sure minification doesn't clobber anything... usually they are quite good at what they do by now but to be safe an extra testing measure should be done.

This is probably a long term issue since it's going to take some time to undo what's been done.

We could also use, instead, a shell script with git directly to a folder and then use some .sh scripts to move things to the appropriate locations... keeping in mind .gitignore . e.g. we really need a build.sh system and exec.sh type system.

This section may change constantly...

cletusc commented 10 years ago

This is where something like #97 Grunt or gulp.js would come in. As far as I'm concerned, all tools (including deploy scripts) should be in the repo in some sort of build or deploy folder to store temp files (auth info or other files). Either build tool can concatenate and minify JS, CSS, or precompile mustache templates.

Martii commented 10 years ago

Thanks Cletus... I'll start reading up on these, including grunts repo. We definitely seem to have calls over many CDNs. This really should stop and be centralized. A build system could manage some of this... hopefully without merge conflicts... but I see multiple .css/.js files from different CDNs with the same naming. While I'm sure those sites might be credible it's impending doom if they change something and kill OUJS for some reason.

cletusc commented 10 years ago

We can also use something along the lines of Bower in conjuction with these. You can set it up to pull a specific URL or repo, e.g. a known working version from a CDN or github, and then the build system can pull from that (just like anything in node_modules from npm install ...). When we need to update a version, it's as simple as changing the Bower info, then testing again. For this particular site, Grunt with Bower is probably the ideal combo.

joeytwiddle commented 10 years ago

CloudFlare seems to be a pretty popular CDN: https://www.cloudflare.com/features-cdn

They have this collection for rarer scripts: http://cdnjs.com/

At least if they go down, many other websites will be broken too. ;)

Obviously Google has their own CDN: https://developers.google.com/speed/libraries/devguide

Some advantages of using a CDN over self-hosting:

Disadvantages:

Martii commented 10 years ago

I understand the only perk is for speed but with recent experiences these CDNs are more of a pain than a plus.

Disadvantages of a non-owned/managed CDN Advantages of a non-owned/managed CDN
Lack of consistency in the code base. e.g. we are referencing CSS files that have the same name and are just slightly different but presumably the same purpose... this is bad. Occasional and intermittent speed improvement
Lack of security esp. with .js hosted files
Lack of privacy. CDNs have always had this issue
On every machine in my nets the cache is cleared on browser close and open... occasionally during if it's initiated. Setting expiry dates isn't that difficult provided web standards are adhered to. ;)
Cloudflare is where we currently get our icons from and sometimes they never show up.
Ace editor has cross-domain issues when used on dev and occasionally on pro.
Their site goes down... we could be foo'd
Add-on Incompatibility of known blacklists

In general it's a bad practice if we don't own the management rights to the hosting space without some control of versioning. The weight of this merits hosting things ourselves or implementing project build controls. The files are relatively small so I don't think it's going to be much of an issue. The solution currently is to put these on our controlled site and have some sort of management to keep them up to date. I'd much rather we kill the site accidentally than someone else out there.

Martii commented 10 years ago

@cletusc Hmm grunt requires a CLA to be signed if we ever need to contribute upstream. This is probably not a wise thing to mix with GPL... GPL will always take precedence in our project and usurping that clearly defined authority could be deemed as unsupported. There is also GHs licensing which this can conflict with. I'll still look into it some more but the gulp repo might be more viable but don't know if it's Bower compatible. Do you know off hand? Also if I was grunt I wouldn't be advertising that Adobe is one of their users... imeo they have some of most buggy closed source software around and we all get to pay for it usually. ;)

There's still always the option of doing this as natively as possible with git. Bash shell scripts can do just about anything and has a longer mtbf since it's been around since the dawn of terminal shells.

Related chatter:

Martii commented 10 years ago

@cletusc Cc: @sizzlemctwizzle Silly question... why would we want to use Bower which has a wrapped package for their system instead of $ npm directly which supports GH repo's? So far jQuery and ACE are on Bower but they are also on GH (jQuery on npm however ACE is not that I can find but I searched on GH for #281)... so couldn't we just use the npm package.json to do the downloading/updating/versioning? Does nodejitsu support GH downloads in package.json? We'll still need something to minify and I'm liking the docs for gulp since it doesn't use intermediate files as much. I assume that we can pull from the node_modules folder anything we want to a build or test folder, and deploy folder. (and/or reuse the dev folder)

$ npm install -h
npm install
npm install <pkg>
npm install <pkg>@<tag>
npm install <pkg>@<version>
npm install <pkg>@<version range>
npm install <folder>
npm install <tarball file>
npm install <tarball url>
npm install <git:// url>
npm install <github username>/<github project>
cletusc commented 10 years ago

so couldn't we just use the npm package.json to do the downloading/updating/versioning?

We could; the only downside is we wouldn't be able to do single files as with Bower.

I assume that we can pull from the node_modules folder

Or serve directly from the node_modules folder, if we ever wanted to go that route, but yes, npm/bower basically just downloads the package to node_modules/bower_components and you can deal with it just like any other file/folder.

Martii commented 10 years ago

the only downside is we wouldn't be able to do single files as with Bower.

It's unlikely that any dev would want skeleton code I think... deploying and testing could use gulp to scoot individual files around because they should be minified/uglified anyhow. I'd rather not bruise the original source when trying to track down a bug. Loading a minified file in takes FOREVER in any editor (save for VIM which is console based...even UltraEdit under Win has some issues sometimes) and I'm developing on a very new machine with a ton of processing power and memory... not to mention trying to search that. GH has some limitations too when the diff is too big but with smaller steps I think this can be managed. Thanks for your feedback.

Bower also creates a lot of weird hidden files... and somehow it created the same filename on my ext4 filesystem (not a symlink either)... so I'm a bit leery on that part. I also had to fill in the entire project information that is already in package.json and tell Bower that it was private (assume this needed to be done). So I'm leaning towards a -1 for Bower I think.

cletusc commented 10 years ago

I'd rather not bruise the original source when trying to track down a bug.

I didn't mean we pick and choose files that we need. I meant that some repos out there have their source in a single file, without a package.json; along the same lines as having a userscript in a single file, some projects do the same.

Loading a minified file in takes FOREVER in any editor

Sublime Text 3 on windows opens a 575562 character minified file in way under a second, and prettifies it in < 3 seconds to 731301 characters (I do this once every couple of days). Time to upgrade :P

Bower also creates a lot of weird hidden files

Not on my setup (Win7). I run npm install -g bower, then add a bower.json. Doing a bower install doesn't add any hidden files that I'm seeing.

I also had to fill in the entire project information that is already in package.json and tell Bower that it was private

If you wanted to add "private": true in there, you can, but we aren't publishing to the bower registry; FWIW we don't have private in our package.json, so would could technically publish the project to the npm registry. Our only commands with bower should be bower install though, so it shouldn't be an issue. This is the absolute bare minimum bower.json; just the name is needed and then your deps.

{
  "name": "openuserjs",
  "dependencies": {
    "jquery": "2.0"
  }
}

Whatever we do, we could use npm to manage the packages that will work with it (e.g. packages with a package.json), and Bower for cases where it doesn't.

Martii commented 10 years ago

Little short there for Bower config... bare bones is:

{
  "name": "OpenUserJS.org",
  "version": "0.1.3",
  "homepage": "https://github.com/OpenUserJs/OpenUserJS.org",
  "description": "An open source user scripts repo built using Node.js",
  "main": "./app.js",
  "moduleType": [
    "node"
  ],
  "keywords": [
    "user",
    "script",
    "userscript"
  ],
  "authors": [
    "sizzlemctwizzle"
  ],
  "license": "GPL-3.0+",
  "private": true,
  "ignore": [
    "**/.*",
    "node_modules",
    "bower_components",
    "test",
    "tests"
  ],
  "dependencies": {
    "ace-builds": "~1.1.5",
    "jquery": "~2.1.1"
  }
}

Having to maintain two separate configs is going in the opposite direction of what this issue is intended. I'm also thinking none of them are going to address rereferencing things that get suffixed e.g. filename.min.js vs filename.js in our actual source.

Let's not get into a debate on editors. Besides I have no intention of reading minified code for any reason. ;) :)

I meant that some repos out there have their source in a single file

The only time this currently applies that I've seen is with some CSS that we pull, for no apparent reason. I would think we would want to handle this ourselves. The google fonts were a prime example of what not to do. Fonts almost never change. I'm using some ttf's from the early 90's and they still work like a charm on every system... granted wrappers have changed a bit for open fonts but they still don't change.

cletusc commented 10 years ago

Little short there for Bower config

Not if we are just consuming deps. Absolute bare minimum is what I posted--see the bower.json spec; all the other stuff is if you are publishing to the Bower registry, of which we are not.

Besides I have no intention of reading minified code for any reason.

Unfortunately, I have to for a userscript; gotta see how the site does certain things to prevent conflicts :D

Martii commented 10 years ago

without a package.json;

There's also this. So the rule would be download whatever is there instead of a versioned item. This seems like a recipe for disaster with multiple file management.

I'm still curious why we have CSS files that are nearly the same but slightly different referenced from cloudflare... I would think that our CSS would be the same and we would want to track changes here on GH instead of blindly following some CDN out there that only has one file on it that may or may not be available at the time of querying.

gotta see how the site does certain things to prevent conflicts

Do you mean if OUJS clobbers something with minification? Those projects that minify are using the same things we could be using. The minified code shouldn't be obfuscated or different. It was not easy determining the Ace version we have locally since those comment headers were stripped out.

Absolute bare minimum is what I posted

Well Bower install started what looked like a TCL/TCK script and starting asking me questions. New devs to the project would need to be instructed on how to use Bower and what not to pick. I think it could be an option chosen by a dev... we could locally ignore those folders but I don't think we should rely on CDNs unless we have to. In most cases it's best to track the changes here on GH and figure things out for ourselves. We can of course preview what others are doing that's why it's called a SCM repo here. :) We're still going to need some accelerator shell scripts too.

cletusc commented 10 years ago

Do you mean if OUJS clobbers something with minification?

No, I mean I have to dig through this and this on a regular basis for this. Was off-topic.

Martii commented 10 years ago

UPDATE

As I've been commenting and referencing/alluding galore for this issue this is a huge undertaking as well as a work in progress one step at a time.

With the recent discovery that $ npm currently has issues with GH urls as dependencies I'm tacking on tracking upstream because that portion needs to be fixed in npm

Target test: $ npm install git://github.com/github/octicons.git That stream added package.json in a PR.

Also bumping the source tree down one level to src is probably what needs to be done in which @sizzlemctwizzle will need to be directly involved with this since deployment is affected as well.

This is where I am at.

Some rambling notes that are probably going to be changing in the comment to reduce the noise level a bit:

Martii commented 8 years ago

@cletusc Cc: @sizzlemctwizzle and anyone else too

Does anyone have a preference on the file cache for the release version of express-minify? This affects dev as well as pro.

Possible locations are:

  1. ./.cache/express-minify... (static or dynamically created?)
  2. SystemTempSpecialFolder/openuserjs.org/cache/express-minify... (dynamically created... optional oujs short name instead of full?)
  3. HomeDirectorySpecialFolder/.cache/openuserjs.org/express-minify... (dynamically created... optional oujs short name instead of full?)
  4. Other physical disk location... (please specify)

If no response something will be chosen... of course it can be changed at a later date.