search

Ladder99 / ladder99

Ladder99 - connects factory devices to database and visualizations using MTConnect, an open standard.
https://mriiot.com
Apache License 2.0
27 stars 10 forks source link

Revise the repository structure #189

Open tukusejssirs opened 1 year ago

tukusejssirs commented 1 year ago

You seem to have multiple packages in a single repository, but the repository structure is a bit confusing. Below is a suggested structure:

If we want to use Nx, it has a distinction between apps and libraries. AFAIK, most of the apps (like drivers) we have, are libraries actually, therefore those would reside in libs/ folder under Nx. Other apps (or microservices) would live under apps/.

I really like Nx monorepo. I understand that this change is quite fundamental and @bburns would need to learn a new tool (actually, a few of them: ESLint, Nx, potentially Nest), however, I really think it would ease are work in the long-term. Currently, Brian does all the hard work, however, why should we do something that is already created for us. Moreover, this tooling helps us generate some boilerplates, helps with testing, structuring, validating the code, building CRUD methods, APIs, etc. Currently, it is a bit of a mess (structurally and codewise too). On the other, Brian seem to do well, which means he’s an experienced programmer. :wink:

tukusejssirs commented 1 year ago

@MRIIOT, just a quick note: moving to use Nx monorepo is much easier than using Nest (obviously). There are some guides to adopt Nx, from which the one that we might want to use is Manual migration of existing code bases. The first and basic task is simple create a new Nx workspace within the repo and move apps and libs to the workspace. That should be a trivial task, however, there might be some changes required.

MRIIOT commented 1 year ago

@bburns describe the impact on setup scripts, setup folder structure, general installation and usage workflow.

bburns commented 1 year ago

services/ seem to be separate apps, therefore I suggest to rename that folder to apps/, probably using Nx;

they're all running at same time and interacting like services, not independent apps, so i liked that term - that's how they're referred to in the docs also. an app to me is more like something user-facing. so would prefer to keep as services.

nx looks good - their default structure is

foo/
├── packages/
├── nx.json
└── package.json

should we stick with packages? or i guess can call it whatever we want?

might also want a services/common folder for common code, though not sure how that would work with building docker images, eg the adapter image, which would need to install this common package code

shell/ is just another app (or a collection of apps?), thus it (they all?) should be moved into separate folder(s) within apps/:

i do think shell should stay top-level, eg for discoverability

consider moving from BASH to JS/TS; consider using Commander.js;

would love to do that - they started as tiny shell scripts and grew to ugly bash code which i don't like. it's a matter of time/budget.

'setups'

  • I am not sure if setups/ should be part of the codebase, maybe we should move it to docs/ and there might not be needed example/ folder at all;
  • I think we should remove/move this folder;

sounds good - i'd left the example setup in there for when user tries out l99, and as a base for new setups (using 'l99 init foo' command), but could pull it out to a setup-example repo, and add the instruction to do 'l99 download example' to the docs, and the l99 init command could make sure example setup is downloaded as a base to copy from.

volumes/:

  • I am not sure what is volumes/ used for, however, it seems to be just some Grafana config, thus consider moving it in that app (in apps/) that needs this.
  • I think we should remove/move this folder;

good idea - can move to services/grafana.

If we want to use Nx, it has a distinction between apps and libraries. AFAIK, most of the apps (like drivers) we have, are libraries actually, therefore those would reside in libs/ folder under Nx. Other apps (or microservices) would live under apps/.

have you developed plugins with nx? is that how you would do it?

i kept the plugins in the relevant service for simplicity, eg https://github.com/Ladder99/ladder99/blob/main/services/adapter/src/drivers/modbus.js

if a client wants to develop custom drivers they can keep them in their setup folder (code to search there not yet implemented tho).

so - would need further exploration for this - can leave where they are for now.

I really like Nx monorepo. I understand that this change is quite fundamental and @bburns would need to learn a new tool (actually, a few of them: ESLint, Nx, potentially Nest), however, I really think it would ease are work in the long-term.

it all sounds good!

describe the impact on setup scripts, setup folder structure, general installation and usage workflow.

i don't think it would have too much impact, other than what was noted above -

scripts - prefer to leave in shell folder setup folders - can stay same place, same structure - just move example setup out of this repo installation - don't know yet - otto? usage - should be same

tukusejssirs commented 1 year ago

services/ seem to be separate apps, therefore I suggest to rename that folder to apps/, probably using Nx;

they're all running at same time and interacting like services, not independent apps, so i liked that term - that's how they're referred to in the docs also. an app to me is more like something user-facing. so would prefer to keep as services.

In Nx and Nest terminology, an app can be any kind of app, be it a CLI app, web server, microservice, …

nx looks good - their default structure is

foo/
├── packages/
├── nx.json
└── package.json

Well, in practice, we could name the src/apps/libs folder(s) whatever we want, however, both Nest and Nx uses src/ folder name as the source folder containing all data of a single app.

When there is only a single app in a Nest or Nx workspace, the src/ folder is in the repo root folder.

However, when there are at least two apps (or one app and one library) in either Nest or Nx workspace, the folder structure changes so that there is apps folder for all apps and libs folder for all libraries.

apps
├── app1
│   └── src
└── app2
    └── src
libs
├── lib1
│   └── src
└── lib2
    └── src

should we stick with packages? or i guess can call it whatever we want?

I suggest to keep the folder structure with default folder names.

might also want a services/common folder for common code, though not sure how that would work with building docker images, eg the adapter image, which would need to install this common package code

As I have already said above, services are apps too from Nx/Nest point of view.

What is common folder for? I presume I’d contain some libraries. In Nx, we can have import libraries from within the monorepo under a predefined prefix, like in @ladder/libName, the prefix is @ladder. We could have multiple prefices in a monorepo, although it is usually only one. Is that the use case for your suggested common folder?

As for building docker images and compiling code, it is usually all built/compiled into dist/ folder (it could be changed though). Every app has a separate subfolder (e.g. apps/ladder app is compiled into dist/ladder). By default, Nx uses Webpack, thus all libraries are ‘copied’ (built) into the the apps (all apps that import that library) unless we configure the library to be buildable. However, until an app/lib is moved into Nx workspace, it could be built whatever way we want (i.e. we do the steps, like build/start/publish, manually, ideally using a script that is defined in package.json).

shell/ is just another app (or a collection of apps?), thus it (they all?) should be moved into separate folder(s) within apps/:

i do think shell should stay top-level, eg for discoverability

Scripts could stay in a scripts/ folder, however, we might want to port the BASH scripts (esp the CLI app) into JS (e.g. using Nest + Nest Commander), then is should be moved to apps/cli and it might be installable via @ladder/cli from NPM.

consider moving from BASH to JS/TS; consider using Commander.js;

would love to do that - they started as tiny shell scripts and grew to ugly bash code which i don't like. it's a matter of time/budget.

Yeah, I actually dislike BASH (apart of some quick, short-term stuff), as it is not maintainable in the long run and untestable (at least I don’t know of any BASH test framework).

'setups'

  • I am not sure if setups/ should be part of the codebase, maybe we should move it to docs/ and there might not be needed example/ folder at all;
  • I think we should remove/move this folder;

sounds good - i'd left the example setup in there for when user tries out l99, and as a base for new setups (using 'l99 init foo' command), but could pull it out to a setup-example repo, and add the instruction to do 'l99 download example' to the docs, and the l99 init command could make sure example setup is downloaded as a base to copy from.

Yeah, from example point of view, example/ folder makes sense, however, IMHO all examples should be in docs/.

IMHO l99 init should generate new configuration from code snippets stored in the l99 init script itself (or a BASH/JS file it sources/imports). I understand l99 download command and it is useful, however, IMHO there should a single default config used by l99 init and then the user either provide the values using command options (e.g. --grafana-ip 1.2.3.4) or env vars (e.g. GRAFANA_IP=1.2.3.4) or l99 init should prompt for the config values or the default should be used (using l99 init -y where -y is to assume yes). That said, I still have no idea about how it actually works in l99.

If we want to use Nx, it has a distinction between apps and libraries. AFAIK, most of the apps (like drivers) we have, are libraries actually, therefore those would reside in libs/ folder under Nx. Other apps (or microservices) would live under apps/.

have you developed plugins with nx? is that how you would do it?

No, I haven’t developed a Nx plugin yet, but I use some pre-built plugins like @nrwl/nest:application to generate a Nest app or @nrwl/nest:library to generate a Nest library (and @nrwl/nest:remove to remove a Nest app or library). Here is a list of Nx plugins.

However, I think you talk about libraries (in Nx/Nest terminology), e.g. a Modbus driver. And yes, I have build some libraries. :wink:

I call your drivers (and other, non-app stuff in l99) libraries. Yes, from some point of view, they are l99 plugins, however, in general I’d call them libraries, although they are not to be used by other apps.

if a client wants to develop custom drivers they can keep them in their setup folder (code to search there not yet implemented tho).

I am sort of lost: what custom drivers you mean a client could develop? IMHO a l99 plugin system could be developed for this purpose if we wanted to, however, I see no (current) use case for this (I might be wrong and I might misunderstant your comment).

scripts - prefer to leave in shell folder setup folders - can stay same place, same structure - just move example setup out of this repo installation - don't know yet - otto? usage - should be same

I’d say, for now, we should keep everything as it is, then later (when we find some time for this) decide which (single) app/library to move to Nx workspace, setup the build system, etc, and only then will @bburns actually see what does it mean to work in an Nx monorepo. :wink: After than, we could move apps and libraries one by one, modifying the config as needed, potentially port the shell scripts (a sort of l99 CLI app) to JS (or TypeScript? :wink:).