search

pilwon / ultimate-seed

{MEAN Stack on Steroids} The ultimate full-stack AngularJS + Node.js/Express seed (batteries included!) that makes web developers insanely productive. (Angular, AngularUI, Barbeque, Bootstrap, Bower, Browserify, Docker, Express, Font Awesome, Grunt, Handlebars, jQuery, JSHint, Karma/Mocha, LESS/LESSHat, Livereload, Lodash/Underscore, Modernizr, MongoDB/Mongoose, Passport for Facebook/Google/Twitter, Redis, SocketIO, Source Maps, Uglify, Winston)
852 stars 179 forks source link

Request all dependency package owners to add .npmignore #71

Open pilwon opened 10 years ago

pilwon commented 10 years ago

It's taking too long to install all dependencies. It appears that most of dependency packages do not contain .npmignore or files specified in package.json to cherry pick what files gets blacklisted or whitelisted in the published npm packages. It will be nice if we inform each one of them about this technique and reduce the overall size and time to install all dependencies.

pilwon commented 10 years ago

All modules that has a real size > 1mb...

where real size = total module directory size - sum of its sub-module directories

Real Size (MB) Size (MB) Name Version Path
10.8832960129 49.9334955215 barbeque 0.2.1 node_modules/barbeque
9.21323013306 9.59646987915 phantomjs 1.9.7-1 node_modules/karma-phantomjs-launcher/node_modules/phantomjs
8.07612419128 9.8537569046 less 1.6.3 node_modules/barbeque/node_modules/less-middleware/node_modules/less
7.89259338379 7.89259338379 zeparser 0.0.5 node_modules/barbeque/node_modules/socket.io/node_modules/socket.io-client/node_modules/active-x-obfuscator/node_modules/zeparser
7.89257717133 7.89257717133 zeparser 0.0.5 node_modules/karma/node_modules/socket.io/node_modules/socket.io-client/node_modules/active-x-obfuscator/node_modules/zeparser
7.89257717133 7.89257717133 zeparser 0.0.5 node_modules/ultimate/node_modules/socket.io/node_modules/socket.io-client/node_modules/active-x-obfuscator/node_modules/zeparser
5.59165477753 7.38679122925 less 1.5.1 node_modules/grunt-contrib-less/node_modules/less
3.54654598236 3.54654598236 esprima 0.9.9 node_modules/barbeque/node_modules/browserify-middleware/node_modules/browserify/node_modules/syntax-error/node_modules/esprima
3.54654598236 3.54654598236 esprima 0.9.9 node_modules/grunt-browserify2/node_modules/browserify/node_modules/syntax-error/node_modules/esprima
3.413438797 3.5336265564 multiparty 2.2.0 node_modules/barbeque/node_modules/express/node_modules/connect/node_modules/multiparty
2.40671348572 2.40671348572 dojo 2.0.0-dev node_modules/intern/node_modules/dojo
2.35101032257 2.35101032257 moment 2.4.0 node_modules/moment
2.21811771393 2.24058628082 useragent 2.0.7 node_modules/karma/node_modules/useragent
1.86213493347 11.7248077393 socket.io-client 0.9.16 node_modules/ultimate/node_modules/socket.io/node_modules/socket.io-client
1.861992836 11.7363100052 socket.io-client 0.9.16 node_modules/barbeque/node_modules/socket.io/node_modules/socket.io-client
1.86197662354 11.7244939804 socket.io-client 0.9.16 node_modules/karma/node_modules/socket.io/node_modules/socket.io-client
1.70005321503 1.70005321503 selenium-webdriver 2.37.0 node_modules/protractor/node_modules/selenium-webdriver
1.30572986603 1.30572986603 uglify-js 1.2.5 node_modules/barbeque/node_modules/socket.io/node_modules/socket.io-client/node_modules/uglify-js
1.30572986603 1.30572986603 uglify-js 1.2.5 node_modules/karma/node_modules/socket.io/node_modules/socket.io-client/node_modules/uglify-js
1.30571365356 1.30571365356 uglify-js 1.2.5 node_modules/ultimate/node_modules/socket.io/node_modules/socket.io-client/node_modules/uglify-js
1.25746059418 1.25746059418 esprima 1.1.0-dev node_modules/rfileify/node_modules/falafel/node_modules/esprima
1.21668815613 1.21668815613 bson 0.1.8 node_modules/ultimate/node_modules/connect-mongo/node_modules/mongodb/node_modules/bson
1.20090389252 1.20090389252 lodash 1.3.1 node_modules/lodash
1.16223812103 1.16223812103 esprima 1.0.2 node_modules/barbeque/node_modules/browserify-middleware/node_modules/browserify/node_modules/insert-module-globals/node_modules/lexical-scope/node_modules/astw/node_modules/esprima
1.16223812103 1.16223812103 esprima 1.0.2 node_modules/barbeque/node_modules/browserify-middleware/node_modules/browserify/node_modules/module-deps/node_modules/detective/node_modules/esprima
1.16223812103 1.16223812103 esprima 1.0.2 node_modules/grunt-browserify2/node_modules/browserify/node_modules/insert-module-globals/node_modules/lexical-scope/node_modules/astw/node_modules/esprima
1.16223812103 1.16223812103 esprima 1.0.2 node_modules/grunt-browserify2/node_modules/browserify/node_modules/module-deps/node_modules/detective/node_modules/esprima
1.06831932068 1.06831932068 bson 0.2.5 node_modules/winston-mongodb/node_modules/mongodb/node_modules/bson
1.06285095215 1.06285095215 requirejs 2.1.11 node_modules/requirejs

Python script used:

#!/usr/bin/env python

import json
import os
from operator import itemgetter

ROOT_DIR = '<absolute-path-of-ultimate-seed-root-dir>'
DATA = []

def get_dir_size(dir):
    dir_size = 0
    for (path, dirs, files) in os.walk(dir):
        for file in files:
            filename = os.path.join(path, file)
            dir_size += os.path.getsize(filename)
    return dir_size / 1024.0 / 1024.0

for root, dirnames, filenames in os.walk(ROOT_DIR):
    if not root.endswith('node_modules'): continue
    for dirname in dirnames:
        module_name = dirname
        module_path = os.path.join(root, dirname)
        pkg_json_path = os.path.join(module_path, 'package.json')
        if dirname == '.bin': continue
        if not os.path.isfile(pkg_json_path): continue
        module_size = get_dir_size(module_path)
        submodules_size = get_dir_size(os.path.join(module_path, 'node_modules'))
        real_size = module_size - submodules_size
        pkg_json_file = open(pkg_json_path)
        pkg_json = json.load(pkg_json_file)
        pkg_json_file.close()
        if 'version' not in pkg_json: continue
        module_version = pkg_json['version']
        DATA.append((real_size, module_size, module_name, module_version, module_path))

print '| Real Size (MB) | Size (MB) | Name | Version | Path |'
print '| --------- | ---- | ---- | ------- | ---- |'
for row in sorted(DATA, key=itemgetter(0), reverse=True):
    print '| ',
    print row[0], ' | ',
    print row[1], ' | ',
    print row[2], ' | ',
    print row[3], ' | ',
    print row[4].replace(ROOT_DIR + '/', ''),
    print '|'