Reduce dependencies

[icebob]

Due to the recent event-stream vulnerability issue, I'm thinking to reduce the used NPM dependencies in the Moleculer core. Currently it is the current dependency tree (29 modules):

`-- moleculer@0.13.4
  +-- args@5.0.0
  | +-- camelcase@5.0.0
  | +-- chalk@2.4.1 deduped
  | +-- leven@2.1.0
  | `-- mri@1.1.1
  +-- bluebird@3.5.2
  +-- chalk@2.4.1
  | +-- ansi-styles@3.2.1
  | | `-- color-convert@1.9.3
  | |   `-- color-name@1.1.3
  | +-- escape-string-regexp@1.0.5
  | `-- supports-color@5.5.0
  |   `-- has-flag@3.0.0
  +-- es6-error@4.1.1
  +-- eventemitter2@5.0.1
  +-- fastest-validator@0.6.12
  +-- glob@7.1.3
  | +-- fs.realpath@1.0.0
  | +-- inflight@1.0.6
  | | +-- once@1.4.0 deduped
  | | `-- wrappy@1.0.2
  | +-- inherits@2.0.3
  | +-- minimatch@3.0.4
  | | `-- brace-expansion@1.1.11
  | |   +-- balanced-match@1.0.0
  | |   `-- concat-map@0.0.1
  | +-- once@1.4.0
  | | `-- wrappy@1.0.2 deduped
  | `-- path-is-absolute@1.0.1
  +-- ipaddr.js@1.8.1
  `-- lodash@4.17.11

There are 3 libs which have too much dependencies:

1. args (4 other deps) It is used in Moleculer Runner to parse arguments
2. chalk (6 other deps) It is used to coloring some log messages
3. glob (12(!) other deps) It is used in Moleculer Runner & ServiceBroker to load services from folders.
4. vorpal It's a dev dependency but it is not maintained & contains some vulnerabilities.

Possible alternatives:

Important to keep the current functionality, so the changes don't cause breaking changes!

args

• [ ] mri (no deps)
• [ ] cac (no deps)
• [ ] tinyargs (no deps)

  chalk

• [x] kleur (no deps)

glob

• [ ] tiny-glob (2 deps)

vorpal

• ?

[icebob]

@lukeed do you have any awesome (fast & no deps) lib to replace the huge glob lib? :smile:

[lukeed]

Hey, haha, I helped work on tiny-glob. It's the one I use

[tinchoz49]

https://github.com/vadimdemedes/ink could be a replacement for vorpal

[lukeed]

Ink is good. I forgot to mention before that sade is also an option for your CLI builder. Its only dependency is mri, which is already in this list

[joshmanders]

Hello, we use Moleculer at my company and vorpal's vulnerabilities are a concern to us and we'd like to get this resolved. What can I do to help with this? I can fork and start a PR to replace vorpal with something else.

Thanks.

[icebob]

vorpal is more than the ink. It has a REPL mode with command & argument handling and autocomplete feature. Ink (as soon as I can see) is a react based GUI for console apps. The better option would be to fix vulnerable packages in vorpal. I've created a fork here

[joshmanders]

Yeah I was looking over alternatives and none really seem to do what moleculer uses vorpal for.

@icebob do you plan to utilize the fork in moleculer-repl by publishing a package or directly to the github repo in the package.json? I can start working on getting the vulnerabilities patched and submit a PR to the fork.

[tinchoz49]

This is another one more close to the vorpal concept: https://github.com/drew-y/cliffy

But I agree:

The better option would be to fix vulnerable packages in vorpal. I've created a fork here

[icebob]

@joshmanders good news. It would be good if you could fix it. I plan to publish the fixed vorpal under the org like: @moleculer/vorpal

[icebob]

@tinchoz49 great, yes, cliffy can be an alternative. Btw, I've starred it for a while :P

[joshmanders]

@joshmanders good news. It would be good if you could fix it. I plan to publish the fixed vorpal under the org like: @moleculer/vorpal

Excellent. I have a PR waiting right now, just have noticed some tests are timing out so I'm trying to figure that out.

[GrosSacASac]

I heard that bluebird is no longer needed

[abdavid]

@icebob Could we add lodash to the list of libraries that Moleculer could live without? I dissected the framework a while back while playing around with making it a first-class typescript citizen and noticed a lot of lodash usage that could be replaced by simple native js. The reason I particularly am mentioning this lib is because of the bad pratices like prototype pollution it often entails.

https://github.com/moleculerjs/moleculer/search?q=isObject&unscoped_q=isObject

[abdavid]

Latest tree btw

moleculer@0.14.6 /Users/davborre/Source/moleculer
├─┬ args@5.0.1
│ ├── camelcase@5.0.0
│ ├─┬ chalk@2.4.2
│ │ ├─┬ ansi-styles@3.2.1
│ │ │ └─┬ color-convert@1.9.1
│ │ │   └── color-name@1.1.3
│ │ ├── escape-string-regexp@1.0.5
│ │ └─┬ supports-color@5.4.0
│ │   └── has-flag@3.0.0
│ ├── leven@2.1.0
│ └── mri@1.1.4
├── es6-error@4.1.1
├── eventemitter2@6.4.0
├── fastest-validator@1.4.1
├── fn-args@5.0.0
├─┬ glob@7.1.6
│ ├── fs.realpath@1.0.0
│ ├─┬ inflight@1.0.6
│ │ ├── once@1.4.0 deduped
│ │ └── wrappy@1.0.2
│ ├── inherits@2.0.3
│ ├─┬ minimatch@3.0.4
│ │ └─┬ brace-expansion@1.1.11
│ │   ├── balanced-match@1.0.0
│ │   └── concat-map@0.0.1
│ ├─┬ once@1.4.0
│ │ └── wrappy@1.0.2 deduped
│ └── path-is-absolute@1.0.1
├── ipaddr.js@1.9.1
├── kleur@3.0.3
├── lodash@4.17.15
├─┬ lru-cache@5.1.1
│ └── yallist@3.0.3
└── node-fetch@2.6.0

[icebob]

Yes, it would be good to drop lodash lib from the core repo. The most important method is the _.defaultsDeep. It is used in many places, so it's the most important to find an exact same implementation or copy out from the lodash.

[joshmanders]

Yes, it would be good to drop lodash lib from the core repo. The most important method is the _.defaultsDeep. It is used in many places, so it's the most important to find an exact same implementation or copy out from the lodash.

Could just depend on lodash.defaultsDeep which is just the dependencies needed for that function.

[icebob]

Hmm, good idea!

[icebob]

By the way, I've just found cac which is same as args but no dependencies.

Only need to change the glob to avoid another big branch

[abdavid]

_.get used in the cacher implementation is pretty important aslo I believe. It would be great to firstly get an overview of all lodash deps to determine which could easily be switched out with vanilla js and which we need to consider keeping or replace. I’d be happy to contribute on some concrete issues like «replace usage of isObject» etc to we can gradually migrate lodash out.

• David

21. mai 2020 kl. 18:32 skrev Josh Manders notifications@github.com:

 Yes, it would be good to drop lodash lib from the core repo. The most important method is the _.defaultsDeep. It is used in many places, so it's the most important to find an exact same implementation or copy out from the lodash.

Could just depend on lodash.defaultsDeep which is just the dependencies needed for that function.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[lukeed]

cac has dependencies (including mri) – it's just all inlined into one big pre-built file, which prevents any cross-dep code reuse

[icebob]

Ohh, thanks @lukeed, I didn't see it.

[lukeed]

It's actually only mri - I thought there were more. This means cac basically the same as sade...good upgrade either way

[icebob]

@abdavid Here is a list about the used Lodash methods:

[abdavid]

Nice! I see a couple of low hanging ones. I think we can break them down pr function.

@icebob should we maybe create an epic and materialize tasks for each function? That way we will have the oppertunity to discuss replacements contextually. Like Josh commented we could just import the complex ones.

• David

22. mai 2020 kl. 18:37 skrev Icebob notifications@github.com:

 @abdavid Here is a list about the used Lodash methods:

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[icebob]

Ok, go ahead :)

[abdavid]

I do not have the necessary access to create anything like that. For github an epic equivalent would maybe be a project with multiple subtasks? #jiranerd

[icebob]

Here there are some native implementation: https://github.com/you-dont-need/You-Dont-Need-Lodash-Underscore#quick-links

[abdavid]

Here there are some native implementation: https://github.com/you-dont-need/You-Dont-Need-Lodash-Underscore#quick-links

Sweet! Nice find

[abdavid]

@icebob Do we have any docs / guides on performance testing? Would be necessary to have before and after measurements to ensure we keep things speedy

[icebob]

Yes, you can with npm run bench. It executes common performance benchmark tests, so you can measure before & after with it.

[Wallacy]

@icebob Another option to replace Vorpal: https://github.com/f/omelette

Has zero dependencies;

[icebob]

Thanks @Wallacy but I think it's not a replacement of vorpal. It's just a part of it.

[AndreMaz]

One possible replacement for vorpal in moleculer-repl is shargs.

Some features:

• Highly customizable command-line arguments parser and usage documentation generator.
• 35+ opt-in features, e.g. (multiple) subcommands, spelling mistake detection, default values, and (best guess) casting.
• Synchronous and Promise-based asynchronous mode with async/await support.
• Automatic usage documentation generation with fine-grained control over layouts and styles.
• Easily extensible with your own custom parser stages and custom usage layouts.
• Extensively documented and very well tested (750+ unit and integration tests).
• Modular library layout with zero runtime dependencies.

and @Yord is currently working on REPL mode and autocompletion

[icebob]

It sounds great. it has no dependency.

[Yord]

Hey all, shargs is meant to replace the command-line parser of another project of mine because it had way too many dependencies (27!). Seems our causes are aligned.

The REPL mode is still very fresh, but I am interested in spending more time.

[icebob]

@Yord Do you know vorpal? Are there any other missing parts in shargs? We are using it in our moleculer-repl

[Yord]

@icebob I have never worked with vorpal.

The command-line parser parts of shargs are almost at 1.0.0, error message ergonomy is left (and bash autocomplete).

As for REPLs, shargs has basic REPL functionality based on the Node REPL and autocomplete for commands. These two parts are not tested, yet.

I could imagine shargs REPL misses text colors and formatting (aka tables) unless the action output does that.

Is there anything apparent to vorpal users that I miss?

Edited to add: Shargs REPL misses pipes and prompts.

[AndreMaz]

I could imagine shargs REPL misses text colors and formatting (aka tables) unless the action output does that.

You are right, the coloring and formatting are done directly in action (e.g., service )

Edited to add: Shargs REPL misses pipes and prompts.

I think that we don't use either of those features. We only use "a small portion" of vorpal

[icebob]

Would be good to create a branch in moleculer-repl and trying to add Shargs instead of vorpal. @AndreMaz do you have some capacity to work on it? Maybe @Yord can help if something is missing :)

[Yord]

@icebob, @AndreMaz I will gladly ship in if I can be of help.

[AndreMaz]

@icebob I will look at it during weekend. In theory it shouldn't be very complex. I just need to find the equivalent for vorpal's syntax. E.g.:

vorpal
  .command('foo <requiredArg> [optionalArg]')
  .autocomplete({
    data() {
        return // Return autocomplete array
  })
  .option('-v, --verbose', 'Print foobar instead.')
  .description('Outputs "bar".')
  .alias('foosball')
  .action(function(args, callback) {
    if (args.options.verbose) {
      this.log('foobar');
    } else {
      this.log('bar');
    }
    callback();
  });

@Yord do you have any tutorial for newbies? Your docs are very good but I didn't found any introductory tutorial.

[Yord]

@AndreMaz Fortunately I wrote one just yesterday:

https://github.com/Yord/shargs-tutorial-git/blob/master/README.md

[AndreMaz]

Sweet 😄 Thanks

[Yord]

@AndreMaz Just wanted to let you know I have incorporated recent updates into shargs-example-repl.

To elaborate on this: The API has only slight changes, the updates concern mainly autocomplete internals.

[AndreMaz]

@Yord thanks for the info. Can you add the changelog to your repos? It helps to track the changes

[Yord]

@AndreMaz Yes, I will do that. Sorry for the sloppy work. I usually start a changelog with the first stable version, but here, it definitely makes sense to start right now.

Changelogs:

• shargs-example-repl 0.4.0
• shargs-repl 0.2.0
• shargs 0.24.6

[gautaz]

Hello, sorry to barge in but I think that what I raised here might be related to this issue.

My main concern is related to the fact that some of us (I have no idea how many) are using the Moleculer broker without the runner. I was wondering if moving the broker to something like moleculer-core (or the runner to moleculer-runner) would be an acceptable solution.

The runner is great but it also comes with its cost in term of dependencies and I guess that part of the dependencies raised in this issue is in fact due to the runner (another batch is certainly due to the REPL). :warning: This last one is an assumption, I did no check the source code so I may be totally wrong (which would be far from being the first time :laughing:).

[icebob]

I plan to split this repo to multiple packages in v1.0 and release them under @moleculer/....