Better processing of packages

[beruic]

I have a rather large packman.json file, and saving it or restoring packages takes ages. The UI (VS 2015) also becomes unresponsive.

First of all, I think something should be placed in a background thread so the whole UI doesn't freeze.

Next, I imagine the following could speed the whole process it up:

1. When installing a package, add a file in the install directory, with the installed package name and version number.
2. Upon saving packman.json, when running through the packages, check for correct package name and version number in existing install directories, and if it doesn't match, delete the directory content and copy.

I believe this would reduce the number of necessary file operations drastically.

[madskristensen]

Can you please share the packman.json file? Also, what project type (MVC, Cordova, or...) are you seeing this in?

[beruic]

{
  "packages": {
    // Required
    "animate.css": {
      "path": "lib/animate",
      "version": "3.5.1"
    },
    "twitter-bootstrap": {
      "path": "lib/bootstrap",
      "version": "3.3.6"
    },
    "font-awesome": {
      "path": "lib/fontawesome",
      "version": "4.5.0"
    },
    "jquery": {
      "path": "lib/jquery",
      "version": "2.2.0"
    },
    "metisMenu": {
      "path": "lib/metisMenu",
      "version": "2.4.0"
    },
    "pace": {
      "path": "lib/pace",
      "version": "1.0.2"
    },

    // Plugins
    "blueimp-gallery": {
      "path": "lib/blueimp-gallery",
      "version": "2.18.1"
    },
    "bootstrap-datepicker": {
      "path": "lib/bootstrap-datepicker",
      "version": "1.6.0"
    },
    "bootstrap-markdown": {
      "path": "lib/bootstrap-markdown",
      "version": "2.10.0"
    },
    "bootstrap-tour": {
      "path": "lib/bootstrap-tour",
      "version": "0.10.3"
    },
    "bootstrap-touchspin": {
      "path": "lib/bootstrap-touchspin",
      "version": "3.1.1"
    },
    "Chart.js": {
      "path": "lib/chartjs",
      "version": "1.0.2"
    },
    "clipboard.js": {
      "path": "lib/clipboardjs",
      "version": "1.5.9"
    },
    "codemirror": {
      "path": "lib/codemirror",
      "version": "5.12.0"
    },
    "c3": {
      "path": "lib/c3",
      "version": "0.4.9"
    },
    "datatables": {
      "path": "lib/datatables",
      "version": "1.10.11"
    },
    "d3": {
      "path": "lib/d3",
      "version": "3.5.9"
    },
    "jeditable.js": {
      "path": "lib/jeditablejs",
      "version": "1.7.3"
    },
    "jqcloud": {
      "path": "lib/jqcloud",
      "version": "1.0.4"
    },
    "jqgrid": {
      "path": "lib/jqgrid",
      "version": "4.6.0"
    },
    "jqueryui": {
      "path": "lib/jqueryui",
      "version": "1.11.4"
    },
    "jQuery-slimScroll": {
      "path": "lib/jquery-slimScroll",
      "version": "1.3.7"
    },
    "jquery-sparklines": {
      "path": "lib/jquery-sparklines",
      "version": "2.1.2"
    },
    "jquery-validate": {
      "path": "lib/jquery-validate",
      "version": "1.14.0"
    },
    "nestable": {
      "path": "lib/nestable",
      "urls": [
        "https://raw.githubusercontent.com/dbushell/Nestable/4f93032cfafe2002f24ed6bd7dc1510931498503/jquery.nestable.js"
      ]
    },
    "peity": {
      "path": "lib/peity",
      "version": "3.2.0"
    },
    "flot": {
      "path": "lib/jquery-flot",
      "version": "0.8.3"
    },
    "flot.tooltip": {
      "path": "lib/jquery-flot-tooltip",
      "version": "0.8.5"
    },
    "flot-spline": {
      "path": "lib/jquery-flot-spline",
      "urls": [
        "https://raw.githubusercontent.com/johnpozy/flot-spline/4d9c93dd081b219f17e7e062e7b1223d4d5a7290/js/jquery.flot.spline.js",
        "https://raw.githubusercontent.com/johnpozy/flot-spline/4d9c93dd081b219f17e7e062e7b1223d4d5a7290/js/jquery.flot.spline.min.js"

      ]
    },
    "iCheck": {
      "path": "lib/icheck",
      "version": "1.0.2"
    },
    "markdown.js": {
      "path": "lib/markdownjs",
      "version": "0.5.0"
    },
    "morris.js": {
      "path": "lib/morrisjs",
      "version": "0.5.1"
    },
    "raphael": {
      "path": "lib/raphael",
      "version": "2.1.4"
    },
    "rickshaw": {
      "path": "lib/rickshaw",
      "version": "1.5.1"
    }

    // Other stuff
  }
}

[madskristensen]

Oh man, that slows everything down. I can repro and I'm working on a fix. Thanks

[beruic]

I can send you a longer one whenever you'd like it. Think my final file will be twice as long from the looks of the bundles I have to make :)

[beruic]

I can see that you have Added option to exclude libraries from project. How does this work, and will you release it before releasing a final fix?

[madskristensen]

The includeInProject property in packman.json is being released soon. I'm still trying to figure out how to make it fast to add many files to the project. It's not as simple as it sounds. The "includeInProject": false mitigates the issue for now

[beruic]

Awesome.

Am I correct to assume that the includeInProject option set to false prevents the added files from being added to the .csproj file? In that case, would it perhaps speed things up just to postpone this step until all the files are in place, so they are bulk added, and processed once by VS, instead of being processed for each item?

Also, not adding the files to the .csproj file, will that prevent files from being served on [host]/lib/[package]/[whatever]? This would be an issue for packages where you bundle styles or scripts that need to access static files in the package directory structure, like e.g. font-awesome that needs to access its font files, references in the css by the path ../fonts.

[madskristensen]

The fix was to batch add the files using a kinda strange API. It takes about ~30 seconds to do a package restore using the repro packman.json file from above with all 1093 files, but only if none of the files where already in the project. It should be a lot faster for the 80% use case.

Here are some of the changes:

1. New API for batch adding the files to the project
2. Only copying files if they don't exist or are different
3. Only adding files to project that has been updated

I hope this fixes the issue though it isn't perfect

[beruic]

Well, it's a start :) I assume it will release sometime soon? Or should I pull a nightly to test?

[madskristensen]

You can always pull the nightly. I'll release it soon though.