[Rant/Suggestions] Source control, packages, agents, etc

[JustTNE]

Meshcentral is software that I and my small (hobbyist) team use every day, but I feel like Meshcentral development is following some disadvantageous practices that could slow down development or lead to security issues.

1. Binary files don't belong in source control Meshcentral has this weird practice of having all binary files (agents, router, etc) as part of its git source control. This leads to issues like insanely long download times of the Meshcentral git repository and large file sizes. For example, the current Meshcentral ".git" folder alone is almost 650 mb large when performing a full clone.

2. Build scripts The way you would solve the problem 1 and make it easier for new contributors to join AND make it easier for Meshcentral releases to be released is to use build scripts. Ideally, you should just be able to run one command to be able to build a package almost identical to what you would get from NPM. Of course, this is difficult with different architectures, but the binary files should be part of the .gitignore file.

3. Use NPM packages and webpack for the client side This is a big one and in my opinion a very important issue. Meshcentral decides to include third party modules in its source code instead of using NPM. This prevents NPM updates with improvements and security fixes that are being rolled out by the package maintainers from ever reaching Meshcentral, because all updates would have to be performed manually instead of using something like "yarn upgradeInteractive --latest". An example for this would be xterm.js or most things in the public/scripts folder. A tool like webpack makes it easy to include third party modules and combine them into one single JS file that gets downloaded by the browser. Webpack also allows you to use nodejs require syntax in the browser and even easily allows for advanced techniques like dynamic loading.

\

Those are the main things I've noticed and I'd of course be happy to help out when implementing improvements since we use all of these methods in production.

[ryanblenis]

While those aspects never occurred to me they do sound reasonable, and if you're able to help implement them and improve the project, all the better! My two cents (in addition) would be:

4. I can't count the times (especially since stable / nightly releases were implemented) that I've wanted to fork the repo to branch / rebase the commits into stable and nightly channels (backporting fix-based commits into "stable" while leaving new functionality in "nightly") to show how this could be improved as well, but honestly I'm not familiar enough with all the changes happening and underlying functionality to undertake it in the correct manner. The problem is that "stable" isn't really receiving fixes... it's just a certain version that was chosen arbitrarily. The real problem is that many users don't know that, so they stay on stable, yet continue to report bugs against that have since been fixed.

5. Sometimes I see that a new version of MC is available via npm, and go to see what has changed in the commits, only to find that version hasn't been committed yet. I would have thought npm packages were deployed from github when changes were committed, which leads me to;

6. New features and functionality tends to break existing features / functionality at a decent rate, which could likely be avoided with unit/integration testing somewhere in the pipeline, making the project more stable overall. Of course, these are usually reported and fixed extremely quickly, but I'm sure we'd all prefer to avoid it if possible.

Love the project, the devs, its rapid development, and the community around it- and I know Ylian already has a ton on his plate, but I do think that these would be some great quality of life improvements for existing users and developers, as well as make it easier for others to join the project as well. While I'm all for learning, sadly I'm not familiar enough with these practices to set up the proper processes in the correct manner (at least not anytime soon).

[krayon007]

Part of the dilemma is that the Server and the Agent are in two different repos as they progress at different rates... (Plus the agent has other uses besides just connecting to Mesh Central, which is another reason why it's in it's own repo).

Some of the things in the server, such as the meshcore files and some of the module files, are dependent on what version the agent is, particularly if new features are being added.

The other issue is that of how simple/complex you want server installs to be. Because the agent is not written in node, so it's not something you can just pull down from npm and expect it to run/build. Compiling the agent actually requires a bit of setup/work. That would mean that you wouldn't simply be able to just pull down the server and have it just work if you had to compile the agents separately, especially if you need a different toolchain then the one you are using. This is especially true, if you need a different version of glibc for the target as well. Just because it compiles on your system, doesn't mean it will actually run on the intended platform.

[krayon007]

Plus throw in Windows and MacOS into the mix, and it's even harder if you need to support platforms that can't normally be compiled on the platform your server runs on.

[JustTNE]

but honestly I'm not familiar enough with all the changes happening and underlying functionality to undertake it in the correct manner.

Yea, backports have to be done by the main project devs, this is also something we do in prod

Part of the dilemma is that the Server and the Agent are in two different repos as they progress at different rates...

Yea, the way you would solve that is using git submodules. They are fixed to a specific commit hash and can easily be updated on demand. We do this quite often in our projects, here's an extreme example:

The other issue is that of how simple/complex you want server installs to be

Ideally, the NPM versions would already have agents and everything included, I'm only referring to source control here.

Compiling the agent actually requires a bit of setup/work.

A good solution for this would be docker containers in my opinion. Run the entire build process in a preconfigured docker container and then get the "artifacts" out. This is also something we do in prod: https://gitlab.com/nullworks/accgen/accgen-electron-app/-/blob/master/.gitlab-ci.yml

Plus throw in Windows and MacOS into the mix, and it's even harder if you need to support platforms that can't normally be compiled on the platform your server runs on.

Indeed, but the agents should ideally be built by CI service and the binaries should already be included in the NPM package.

[JustTNE]

It would be great if @krayon007 could document all requirements for the agents.

[JustTNE]

and if you're able to help implement them and improve the project, all the better!

@ryanblenis the problem is that Meshcentral is a huge monolithic application with some very complicated/questionable code. Webserver.js alone is 5k lines for example, while the equivalent for the projects I'm working on is max 100-300 lines, while also handling the initialization of the software. In my personal opinion, this makes Meshcentral very hard to maintain. I don't have any motivation at all to try and understand this many lines of code.

[ryanblenis]

@TotallyNotElite thanks for that info, I'm going to start a side project to get more familiar with CI and submodules to expand my knowledge.

There is certainly a lot going on in MC, especially with the server-agent architecture considered. I'll admit I mostly shy away from core MC updates because there's a lot of functionality that not everyone uses and without intimate knowledge of everything that's going on, even a small update is likely to break something. Then I feel bad because I put more on Ylian's plate instead of less... which unit testing probably would have caught and yelled at me for prior to submitting a PR

[Ylianst]

I like these "meta" threads from time to time. On the up side, it's nice that MeshCentral is at a point where someone would take the time to "rant" like this. There is certainly a lot of MeshCentral that comes from the early start where I was still learning JavaScript, NodeJS, GitHub and NPM. MeshCentral is certainly always going to be complex and we are doing with two full time people and community help what other companies are doing with 20x+ more people. I also have to maintain MeshCommander on top of all this, that alone should be a full time job. Also, I keep in mind that there is just no way to make everyone happy and the backlog of issues and requests is growing by the day. These are not bad problem to have, but does cause some stress.

To address more specifics, I do welcome suggestions on code reorganization but I would prefer someone to fork MeshCentral and show the way. One area that would be interesting to me is if someone would re-build the web application using ReactJS or some framework. I would gladly ship both your interfaces and/or transition and it would be interesting to see a different approach. Same for the server and agent.

Placing the agent binaries in GitHub does have security benefits since we have a traceable executable hash and the Windows executable are self-signed. When we debug the agent, we need to know what binary we are using, etc. This said, if there is a different approach, I do welcome looking at a fork or pull requests.

[Ylianst]

@ryanblenis Just a site note, don't feel shy of working any part of MeshCentral you like, I do accept and fix if needed any pull request I get and will gladly accept unit tests, etc.

[ryanblenis]

Just a site note, don't feel shy of working any part of MeshCentral you like, I do accept and fix if needed any pull request I get and will gladly accept unit tests, etc.

I know and that is fantastic (and thank you!), but in the larger scope of this meta-thread, it ties up the finite resource that is @Ylianst with things I could have been alerted to and fixed prior to a PR; thereby minimizing your touch-time. Otherwise some PR's basically just equate to a feature-request with a [buggy] proof of concept.

But always glad to know they're still welcome! Thank you as always for all your hard work!

[Ylianst]

@ryanblenis I get the point, but short term, I will gladly take a buggy proof of concept anytime! Any community work is much appreciated, including work to improve code organization or testing.