Debate: Monorepo

[jesus-seijas-sp]

We are near to publish the version 2.0.0, with async process, transformations, slot filling... and so many features. How to evolve to the version 3.0.0? From my point of view, evolving to a monorepo with lerna can be positive. Pros:

• Import only what you need
• As there will be integration with duckling, not only Microsoft Recognizers, that means that you can choose which one to use, without having the other into node_modules.
• There are features that works perfectly out of the NLP environment, example the SimilarSearch, with levenshtein distance, is faster than the existing in NPM due to the use of a fast normalizetion instead the INTL collation.

Contras:

• More libraries into the npm ecosystem
• Harder to mantain and publish
• More noise to the developer environment.

What do you think?

@FKSI @franher @vvscode @poveden @nulltoken

[franher]

You already know my answer @jesus-seijas-sp :) I love lerna monorepos <3

But as you said not always it fits well with all the projects. How many packages do you think will be?

[jesus-seijas-sp]

Well, it will be:

• Similar Search: The util to do levenshtein distance, and search substrings inside strings with accuracy
• XTables: Util for extracting tables from excel
• Sentiment Analysis
• NLP: Main manager
• NER: Entity extraction
• NLG: Language generation
• Recognizer: Microsoft Bot Framework Recognizer
• NER-Microsoft: the builtin entity extraction using microsoft
• NER-Duckling: builtin entity extraction using duckling

Like this, perhaps also split by language because each language has files with big size.

[franher]

So for me monorepo is a valid solution seeing all the kind of packages you have in mind 👍

In my experience, this pattern encourages developer to decoupling features, better set of test and better documentation.

[FKSI]

Coming a bit late in the discussion, the list of Pros and Cons are fair. My only concern right now is if we "split" would we have traction to make each project grow enough? In other terms is it the good time (marketing wise) to introduce such big change? 🤔

[jesus-seijas-sp]

IMHO the product will be still one: nlp.js. You have examples in github of other lerna or monorepo projects. One example is the botbuilder-js from Microsoft: when a developer wants to develop a bot, botbuilder is one option, even if internally is a monorepo. Also, the objective is not the marketing, but giving value back to the open source community. Having it monorepo adds several values to the project:

• Less size (currently full project uncompressed is 100MB)
• You use only what you need
• More easy to develop side components made by other developers
• Capability of making it compatible with web not only node, with less size, importing only the languages that you use (language data is what is consuming more space)

My vote is for monorepo.

[jesus-seijas-sp]

Invoking @davidknezic ... it's possible to publish under organization?

[jesus-seijas-sp]

Closing: v4 is a monorepo