mwaylabs / generator-m-ionic

Advanced workflows and setup for building rock-solid Ionic apps
MIT License
672 stars 134 forks source link

update functionality for the generator #158

Open gruppjo opened 9 years ago

gruppjo commented 9 years ago

read .yo-rc.json contents, so it works similar to reapeating yo m --skip-prompts in the same directory. Be careful since .yo-rc.json contents get overwritten without any warning. This might lead to complications when version controlling cordova platforms and plugins via .yo-rc.json

gruppjo commented 8 years ago

usual procedure:

  1. always document changes to generator core files (readme is a good place)
    • gulp tasks, gitignore, eslintrc, index.html, app.js ...
  2. create new folder
  3. setup new project (same name, bundle ids, ..)
  4. cp .git/ folder to new project
  5. copy custom files and folders if you have any (artifacts, doc, ...), that definitely won't have conflicts
  6. remove modules in app/ and test/karma and test/protractor
  7. copy your own modules into the respective folders (spare .eslintrc files and such)
  8. replace res/ folder
  9. git diff
  10. reproduce documented changes to generator files and resolve conflicts
  11. git add one at a time
  12. esp. bower components, config.xml configuration, ...
  13. bower install && npm install && gulp --cordova 'prepare'
  14. gulp watch
  15. review changes, commit

possible issues:

test:

gruppjo commented 8 years ago
gruppjo commented 8 years ago

first draft: unfortunately some weird issues with using npm programatically. Upon calling install of the second generator version npm throws this error, no matter what I do: timeout, loading npm again, ...

npm ERR! install trying to install 1.7.0 to /Users/jonathan/Projects/generator-m-ionic-demo/node_modules/generator-m-ionic
npm ERR! install but already installed versions [ '1.6.0' ]

npm i --save-dev yeoman-test in project

gulp/update.js

'use strict';
// gulp
var gulp = require('gulp');
var options = gulp.options;
var $ = require('gulp-load-plugins')();
// node core
var npm = require('npm');
var fs = require('fs');
var path = require('path');
// other
var chalk = require('chalk');
var yeomanTest = require('yeoman-test');
var answers = require('../.yo-rc.json')['generator-m-ionic'].answers;

var help = {
  loadNpm: function (cb) {
    npm.load({}, function (err) {
      if (err) {
        console.log(chalk.red('error loading npm\n'), err);
      }
      else {
        console.log(chalk.green('npm loaded successfully\n'));
      }
      cb();
    });
  },

  npmCmd: function (command, packageName, cb) {
    npm.commands[command]([packageName], function (er) {
      if (er) {
        console.log(chalk.red('error running: ') + command + ' ' + packageName, er);
      }
      else {
        console.log(chalk.green('successfullly ran: ') + command + ' ' + packageName);
      }
      cb();
    });
  },

  runGeneratorVersion: function (version, cb) {
    this.loadNpm(function () {
      this.npmCmd('install', 'generator-m-ionic@' + version, function () {
        process.chdir('../');
        fs.mkdirSync(help.projectDirVersion(version));
        process.chdir(help.projectDirVersion(version));

        var generatorBase = '../' + help.projectDir + '/node_modules/generator-m-ionic/generators/';
        var ctx = yeomanTest.run(path.resolve(generatorBase + '/app'))
          .withGenerators(help.getDirectoryDirs(generatorBase))
          .withOptions({ // execute with options
            'skip-install': true, // don't need to install deps
            'skip-sdk': true // for some reason won't install cordova properly, so just leave it
          })
          .withPrompts(answers)  // answer prompts
          .on('end', function () {
            process.chdir('../' + help.projectDir);
            this.npmCmd('uninstall', 'generator-m-ionic', cb);
          }.bind(this));
        ctx.settings.tmpdir = false; // don't run in tempdir
      }.bind(this));
    }.bind(this));
  },

  projectDir: (function () {
    var projectDir = process.cwd().split('/');
    projectDir = projectDir[projectDir.length - 1];

    return projectDir;
  })(),

  projectDirVersion: function (version) {
    return this.projectDir + '@' + version;
  },

  getDirectoryDirs: function (dirPath) {
    return fs.readdirSync(dirPath)
    .filter(function (file) {
      return fs.statSync(path.join(dirPath, file)).isDirectory();
    })
    .map(function (dir) {
      return path.resolve(dirPath, dir);
    });
  }
};

// UPDATE
gulp.task('update', [], function (done) {
  if (!options.from || !options.to) {
    console.log(chalk.red('error: ') + 'supply proper generator versions with --from and --to');
    return;
  }
  help.runGeneratorVersion(options.from, function () {
    help.runGeneratorVersion(options.to, function () {
      done();
    });
  });
});
gruppjo commented 8 years ago

Some related research:

use npm programatically http://stackoverflow.com/questions/20686244/install-programmatically-a-npm-package-providing-its-version http://stackoverflow.com/questions/15957529/can-i-install-a-npm-package-from-javascript-running-in-node-js

run generators programatically http://stackoverflow.com/questions/25233234/using-yeoman-programmatically-inside-nodejs-project

get the difference between two repos http://stackoverflow.com/questions/1968512/getting-the-difference-between-two-repositories

node cache invalidation http://stackoverflow.com/questions/9210542/node-js-require-cache-possible-to-invalidate https://nodejs.org/docs/latest/api/globals.html#globals_require_cache

Tips for Writing Portable Node.js Code https://gist.github.com/domenic/2790533 http://shapeshed.com/writing-cross-platform-node/

gruppjo commented 8 years ago

version 2: with child_process.exec, creates two new folders with a fresh setup according to answers. Applying some gitmagic for updates.

'use strict';
// gulp
var gulp = require('gulp');
var options = gulp.options;
// node core
var fs = require('fs');
var path = require('path');
var exec = require('child_process').exec;
// other
var yeomanTest = require('yeoman-test');
var chalk = require('chalk');
var answers = require('../.yo-rc.json')['generator-m-ionic'].answers;

var help = {
  projectDir: (function () {
    var projectDir = process.cwd().split('/');
    projectDir = projectDir[projectDir.length - 1];

    return projectDir;
  })(),

  projectDirVersion: function (version) {
    return this.projectDir + '@' + version;
  },

  getDirectoryDirs: function (dirPath) {
    return fs.readdirSync(dirPath)
    .filter(function (file) {
      return fs.statSync(path.join(dirPath, file)).isDirectory();
    })
    .map(function (dir) {
      return path.resolve(dirPath, dir);
    });
  },

  installAndRunGen: function (version, cb) {
    // install
    this.install(version, function () {

      // change into new dir
      process.chdir('../');
      fs.mkdirSync(this.projectDirVersion(version));
      process.chdir(this.projectDirVersion(version));
      console.log(chalk.green('change dir: ') + process.cwd());

      // execute generator
      var generatorBase = '../' + this.projectDir + '/node_modules/generator-m-ionic';
      var generatorGenerators = generatorBase + '/generators';
      // delete require cache to make node reload the new version
      this.deleteCache();
      console.log(chalk.green('running: ') + require('../' + generatorBase + '/package.json').version);

      var ctx = yeomanTest.run(path.resolve(generatorGenerators + '/app'));
      ctx.settings.tmpdir = false; // don't run in tempdir
      ctx
      .withGenerators(this.getDirectoryDirs(generatorGenerators))
      .withOptions({ // execute with options
        'skip-install': true, // don't need to install deps
        'skip-sdk': true // for some reason won't install cordova properly, so just leave it
      })
      .withPrompts(answers)  // answer prompts
      .on('end', function () {
        // git
        exec('git init && git add . && git commit -m \'init\'');
        // uninstall
        this.uninstall(version, cb);

      }.bind(this));
    }.bind(this));
  },

  uninstall: function (version, cb) {
    process.chdir('../' + this.projectDir);
    console.log(chalk.green('uninstalling: ') + version + ' ' + process.cwd());
    exec('npm uninstall generator-m-ionic', function (error) {
      if (error) {
        console.log(chalk.red('error: ') + 'uninstalling ' + version + '\n', error);
      }
      cb();
    });
  },

  install: function (version, cb) {
    console.log(chalk.green('installing: ') + version + ' to ' + process.cwd());
    exec('npm i generator-m-ionic@' + version, function (error) {
      if (error) {
        console.log(chalk.red('error: ') + 'installing ' + version + '\n', error);
      }
      cb();
    });
  },

  deleteCache: function () {
    for (var key in require.cache) {
      if (key.indexOf('node_modules/generator-m') > -1) {
        delete require.cache[key];
      }
    }
  }
};

gulp.task('update', [], function (done) {
  if (!options.from || !options.to) {
    console.log(chalk.red('error: ') + 'supply proper generator versions with --from and --to');
    return;
  }

  help.installAndRunGen(options.from, function () {
    help.installAndRunGen(options.to, function () {
      // add remotes to from-project
      process.chdir('../' + help.projectDirVersion(options.from));
      var gitCmds = 'git remote add -f update ../' +
        help.projectDirVersion(options.to) +
        '&& git remote add -f project ../' +
        help.projectDir +
        '&& git remote update';
      exec(gitCmds, function () {
        done();
      });
    });
  });
});

Additional git-magic use in project@<from-version>:

git diff master remotes/update/master --stat --diff-filter=M > update-diff
git diff master remotes/project/master --stat --diff-filter=M > project-diff
diff update-diff project-diff

OR:

# merge update branch into original clean state
git checkout -b magic
git merge remotes/update/master -X theirs
# then merge the project branch
git merge remotes/project/master
# resolve conflicts

=> not the same repo, could delete .git and copy the one from original project. Rather complicated though.

gruppjo commented 8 years ago

version 3: applies update on-top of current repo, overwriting files

'use strict';
// gulp
var gulp = require('gulp');
var options = gulp.options;
// node core
var fs = require('fs');
var path = require('path');
var exec = require('child_process').exec;
// other
var yeomanTest = require('yeoman-test');
var chalk = require('chalk');
var answers = require('../.yo-rc.json')['generator-m-ionic'].answers;

var help = {
  projectDir: (function () {
    var projectDir = process.cwd().split('/');
    projectDir = projectDir[projectDir.length - 1];

    return projectDir;
  })(),

  projectDirVersion: function (version) {
    return this.projectDir + '@' + version;
  },

  getDirectoryDirs: function (dirPath) {
    return fs.readdirSync(dirPath)
    .filter(function (file) {
      return fs.statSync(path.join(dirPath, file)).isDirectory();
    })
    .map(function (dir) {
      return path.resolve(dirPath, dir);
    });
  },

  updateWithYo: function (version, cb) {
    this.install(version, function () {
      // execute generator
      var generatorBase = '../' + this.projectDir + '/node_modules/generator-m-ionic';
      var generatorGenerators = generatorBase + '/generators';
      console.log(chalk.green('running: ') + require('../' + generatorBase + '/package.json').version);

      var ctx = yeomanTest.run(path.resolve(generatorGenerators + '/app'));
      ctx.settings.tmpdir = false; // don't run in tempdir
      ctx
      .withGenerators(this.getDirectoryDirs(generatorGenerators))
      .withOptions({ // execute with options
        'skip-install': true, // don't need to install deps
        'skip-sdk': true, // for some reason won't install cordova properly, so just leave it
        'force': true
      })
      .withPrompts(answers) // answer prompts
      .on('end', function () {
        // uninstall
        this.uninstall(version, cb);
      }.bind(this));
    }.bind(this));
  },

  uninstall: function (version, cb) {
    process.chdir('../' + this.projectDir);
    console.log(chalk.green('uninstalling: ') + version + ' ' + process.cwd());
    exec('npm uninstall generator-m-ionic', function (error) {
      if (error) {
        console.log(chalk.red('error: ') + 'uninstalling ' + version + '\n', error);
      }
      cb();
    });
  },

  install: function (version, cb) {
    console.log(chalk.green('installing: ') + version + ' to ' + process.cwd());
    exec('npm i generator-m-ionic@' + version, function (error) {
      if (error) {
        console.log(chalk.red('error: ') + 'installing ' + version + '\n', error);
      }
      cb();
    });
  }
};

gulp.task('update', [], function (done) {
  if (!options.to) {
    console.log(chalk.red('error: ') + 'supply proper generator version with --to');
    return;
  }

  help.updateWithYo(options.to, function () {
    done();
  });

});
gruppjo commented 8 years ago

@MathiasTim, @lordgreg, @DrMabuse23 and team. I invested some (quite a lot of) time into this and found out (as expected) that this is a very delicate and complex task.

However I have an initial proposal that I marked as experimental. Please let me know what you think and if this is similar to what you had in mind and if you have any ideas on how to improve on this.

Before this goes live tomorrow you can find a guide here: https://github.com/mwaylabs/generator-m-ionic/blob/dev/docs/guides/generator_update.md

lordgreg commented 8 years ago

@gruppjo I will probably have to wait for upcoming update (1.8.0). What I've did is:

Here's what I get:

Gregor@HomePC /c/Development/genm-upgrade (update)
$ gulp experimental-update --to=1.7.0
[20:00:17] Using gulpfile C:\Development\genm-upgrade\gulpfile.js
[20:00:17] Starting 'experimental-update'...
installing: 1.7.0 to C:\Development\genm-upgrade
module.js:341
    throw err;
    ^

Error: Cannot find module '../../C:\Development\genm-upgrade/node_modules/generator-m-ionic/package.json'
    at Function.Module._resolveFilename (module.js:339:15)
    at Function.Module._load (module.js:290:25)
    at Module.require (module.js:367:17)
    at require (internal/module.js:16:19)
    at Object.<anonymous> (C:\Development\genm-upgrade\gulp\update.js:41:46)
    at C:\Development\genm-upgrade\gulp\update.js:77:7
    at ChildProcess.exithandler (child_process.js:193:7)
    at emitTwo (events.js:100:13)
    at ChildProcess.emit (events.js:185:7)
    at maybeClose (internal/child_process.js:850:16)
    at Socket.<anonymous> (internal/child_process.js:323:11)
    at emitOne (events.js:90:13)
    at Socket.emit (events.js:182:7)
    at Pipe._onclose (net.js:477:12)

That was, however, after waiting approximately 30 seconds in the terminal without knowing if there's any progress or not.

$ npm --version
3.7.3

Gregor@HomePC /c/Development/genm-upgrade (update)
$ node --version
v5.9.1
DrMabuse23 commented 8 years ago

So what about migrations? Maybe with a diff From your git repo from Tag you can find out which files are habe to be changed. And maybe we can add a migration json Form Every Version to Version its simple but strong

Sent from my Cyanogen phone

Am 28.04.2016 8:06 nachm. schrieb Gregor notifications@github.com:

@gruppjo I will probably have to wait for upcoming update (1.8.0). What I've did is:

cloned a project with generator-m 1.3.3.installed yeoman-test with npm i yeoman-test --save-dev created new branch git checkout -b update copied update.js from gulp directory of dev branch of generator-m and pasted it into my current project folder. ran eperimental-update task with version 1.7.0 since 1.8.0 isn't available yet.

Here's what I get:

Gregor@HomePC /c/Development/genm-upgrade (update) $ gulp experimental-update --to=1.7.0 [20:00:17] Using gulpfile C:\Development\genm-upgrade\gulpfile.js [20:00:17] Starting 'experimental-update'... installing: 1.7.0 to C:\Development\genm-upgrade module.js:341 throw err; ^ Error: Cannot find module '../../C:\Development\genm-upgrade/node_modules/generator-m-ionic/package.json' at Function.Module._resolveFilename (module.js:339:15) at Function.Module._load (module.js:290:25) at Module.require (module.js:367:17) at require (internal/module.js:16:19) at Object. (C:\Development\genm-upgrade\gulp\update.js:41:46) at C:\Development\genm-upgrade\gulp\update.js:77:7 at ChildProcess.exithandler (child_process.js:193:7) at emitTwo (events.js:100:13) at ChildProcess.emit (events.js:185:7) at maybeClose (internal/child_process.js:850:16) at Socket. (internal/child_process.js:323:11) at emitOne (events.js:90:13) at Socket.emit (events.js:182:7) at Pipe._onclose (net.js:477:12)

That was, however, after waiting approximately 30 seconds in the terminal without knowing if there's any progress or not.

$ npm --version 3.7.3 Gregor@HomePC /c/Development/genm-upgrade (update) $ node --version v5.9.1

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

volkrass commented 8 years ago

@lordgreg, please use at least npm 3.8.3 or higher. This version fixed some of the performance issues of npm.

gruppjo commented 8 years ago

@lordgreg, it seems like the paths are not windows-safe. Wops. I'll try to fix that. The wait is normal, since the installation takes a while and I don't output the buffer, so you don't see the usual installation progress. Which I'd like to have in a future version but I'm not familiar with buffers and streams in JS so much and I really want to release this version soon.

@DrMabuse23, I like the idea of providing a diff. We could do that in every release for the previous version and mention in the doc that you can create the diffs between the versions you need yourself. I'm not sure what you mean by migration json?

gruppjo commented 8 years ago

@lordgreg, please try the new version of update.js. It should now be windows safe!

lordgreg commented 8 years ago

@gruppjo will do roger wilco (aka today evening). Will report my result afterwards. 👍

gruppjo commented 8 years ago

I will leave this issue open so we can improve on this in future versions together.

lordgreg commented 8 years ago

So, here's how my update just went. No errors:

Gregor@HomePC /c/Development/genm-upgrade (update)
$ gulp experimental-update --to=1.8.0
[19:13:14] Using gulpfile C:\Development\genm-upgrade\gulpfile.js
[19:13:14] Starting 'experimental-update'...
installing: 1.8.0 to C:\Development\genm-upgrade
running: 1.8.0

Gregor@HomePC /c/Development/genm-upgrade (update)
$ git status
On branch update
Untracked files:
  (use "git add <file>..." to include in what will be committed)

        gulp/update.js
        jsconfig.json

nothing added to commit but untracked files present (use "git add" to track)

Gregor@HomePC /c/Development/genm-upgrade (update)
$ git log
commit e7d3ef57c07514bc6fa7be375962c97b82fa7661
Author: Gregor <lordgreg@gmail.com>
Date:   Thu Apr 28 19:54:58 2016 +0200

    initial commit

Checking README.md, there's still version 1.3.3 listed. diff shows no other things were updated, which leads me thinking... did it worked? :8ball:

gruppjo commented 8 years ago

@lordgreg, it most likely did not. The output is not complete. This is how it should look:

$ gulp experimental-update --to=1.8.0
[15:45:30] Using gulpfile ~/Projects/generator-m-ionic-demo/gulpfile.js
[15:45:30] Starting 'experimental-update'...
installing: 1.8.0 to /Users/jonathan/Projects/generator-m-ionic-demo
running: 1.8.0
uninstalling: 1.8.0 /Users/jonathan/Projects/generator-m-ionic-demo
[15:45:59] Finished 'experimental-update' after 29 s

You're missing the second half.

I'm a little confused. It seems like the task is failing but silently. It's very inconvenient that I cannot test this on windows myself. You could insert some console.logs to see where it fails. It appears to fail somewhere along these lines. https://github.com/mwaylabs/generator-m-ionic/blob/master/generators/app/templates/gulp/update.js#L32-L50

gruppjo commented 7 years ago

Addy Osmani over at Yeoman, mentions the "emit" feature that has been coming out of the CLI ecosystems. https://github.com/yeoman/yeoman/issues/1265#issuecomment-285806051

gruppjo commented 7 years ago

Removed

As part of #481, we're removing this feature from the master.

Alternatives

1 Manually upgrade

You can manually upgrade the generator by creating a new project and moving your old files to the new project. A good step by step guide I have written down here for your consideration: https://github.com/mwaylabs/generator-m-ionic/issues/158#issuecomment-168699072

2 Run yo m-ionic

When running yo m-ionic again in your project's directory and it will run the generator again, using your old answers (except for a few ones that you will prompted to answer again). Then use yeoman's diff feature to look for changes.

3 CLI and "emit"

Maybe the future of generators is a CLI/Generator hybrid that can emit the config upon request as mentioned above.

Addy Osmani over at Yeoman, mentions the "emit" feature that has been coming out of the CLI ecosystems. https://github.com/yeoman/yeoman/issues/1265#issuecomment-285806051

The current implementation for reference

'use strict';
// gulp
var gulp = require('gulp');
var options = gulp.options;
// node core
var fs = require('fs');
var path = require('path');
var exec = require('child_process').exec;
// other
var yeomanTest = require('yeoman-test');
var chalk = require('chalk');
var answers = require('../.yo-rc.json')['generator-m-ionic'].answers;

var help = {
  getDirectoryDirs: function (dirPath) {
    return fs.readdirSync(dirPath)
    .filter(function (file) {
      return fs.statSync(path.join(dirPath, file)).isDirectory();
    })
    .map(function (dir) {
      return path.resolve(dirPath, dir);
    });
  },

  updateWithYo: function (version, cb) {
    this.install(version, function () {
      // execute generator
      var generatorBase = path.resolve('node_modules/generator-m-ionic');
      var generatorGenerators = path.join(generatorBase, 'generators');
      console.log(chalk.green('running: ') + require(path.join(generatorBase, '/package.json')).version);

      var ctx = yeomanTest.run(path.join(generatorGenerators, '/app'));
      ctx.settings.tmpdir = false; // don't run in tempdir
      ctx
      .withGenerators(this.getDirectoryDirs(generatorGenerators))
      .withOptions({ // execute with options
        'skip-install': true, // don't need to install deps
        'skip-sdk': true, // for some reason won't install cordova properly, so just leave it
        'force': true
      })
      .withPrompts(answers) // answer prompts
      .on('end', function () {
        // uninstall
        this.uninstall(version, cb);
      }.bind(this));
    }.bind(this));
  },

  uninstall: function (version, cb) {
    console.log(chalk.green('uninstalling: ') + version + ' ' + process.cwd());
    exec('npm uninstall generator-m-ionic', function (error) {
      if (error) {
        console.log(chalk.red('error: ') + 'uninstalling ' + version + '\n', error);
      }
      cb();
    });
  },

  install: function (version, cb) {
    console.log(chalk.green('installing: ') + version + ' to ' + process.cwd());
    exec('npm i generator-m-ionic@' + version, function (error) {
      if (error) {
        console.log(chalk.red('error: ') + 'installing ' + version + '\n', error);
      }
      cb();
    });
  }
};

gulp.task('experimental-update', [], function (done) {
  if (!options.to) {
    console.log(chalk.red('error: ') + 'supply proper generator version with --to');
    return;
  }

  help.updateWithYo(options.to, function () {
    done();
  });
});

The current docs:


Generator Update (experimental)

This feature is highly experimental!
We're exploring ways that can make upgrading your project to a new version of the generator easier. We're very happy to get feedback from you, but please be careful with what you are doing. The full discussion with many different ideas on how to achieve an update can be found in this issue.

Considerations

When you are trying to upgrade your project there's many things to consider that stem from a complex technology stack, how you work on your project, whether and how you use git and many other factors. So depending on these topics there might be better ways of how to perform an update. Please feel free to explore them and share them with us.

Additional considerations that play a role on how easy an update is:

Additional/alternative measures

Versions <1.8.0

If you're upgrading from an version of the generator that is smaller than 1.8.0. You'll need to do two things:

# install yeoman-test as a devDependency (in your project folder)
npm i yeoman-test --save-dev

And save the following gulp file as gulp/update.js in your project.

Perform experimental update

First of all we recommend switching to a new git branch (just to be sure).

git checkout -b update

Before you perform the next step.

Read this carefully. This will do the following:

=> It's up to you to make sense of these changes and differentiate between changes we made and possible changes you made, and reapply them when necessary.

This will not:

There's still some things that could go wrong:

If you are lucky and additionally:

... updating might be very simple.

No promises!

Then, after careful consideration of all of this, run:

# update to version 1.8.0
gulp experimental-update --to=1.8.0
# wait
git diff

Good luck and please give some feedback in this issue!