A binary executable system for providing module/dependencies as a universal medium for cross languages/framework

[iNerdStack]

Project description

Backend applications are built majorly on their language dependencies/modules. These dependencies/modules are tied to the language environment and not flexible enough to be used in another language environment. For example, a file converter module available on npm can't be used directly in a PHP project, you would have to use an equivalent module/dependency in composer or serve its feature through an API system.

A binary dependency system can help bridge these gaps and create a new way to work around dependencies across languages. Binaries (executable console applications) can be created as dependencies and then any backend application or software can execute these binaries and get results directly from the executed binary into their application. The response and output can be formatted using the language core libraries and also these binaries can be sorted in a common folder. This common folder would contain the binaries and wrappers that help execute and return responses from binaries into a backend application or software.

With a central binary module system, this will help:

• Create a global repository that can be used across different languages and frameworks.
• Open a new world to backend development: Software developers (regardless of language source) can build console applications as binaries modules and shared to a global repository.
• Provide flexibility for projects: Projects built on binary dependencies can port their project to a new language or framework without worrying about recreating or searching for equivalent modules in the new language. They can directly integrate the binary modules they used in the old project.

Relevant Technology

1. Package Publisher Website ( Web ) Github can be used as a code repository, software developers who create binary dependencies can provide the configurations of their binaries, create their usage and then publish on a web platform created for binary module publishers.

2. Package Management Software (Desktop Application) This will work like other package management systems e.g npm, composer, etc. It will be used to pull binary modules and their dependencies and also link them to your project. A module.json file (this will include all the module names and configurations) will be required at the project root.

3. Documentation Proper documentation is needed for the project and all sections of it

Complexity

• [ ] Beginner
• [x] Intermediate
• [ ] Advanced

Required time (ETA)

• [ ] Little work - A couple of days
• [ ] Medium work - A week or two
• [x] Much work - The project will take more than a couple of weeks and serious planning is required

Categories

• [ ] Mobile app
• [ ] IoT
• [ ] Web app
• [ ] Frontend/UI
• [ ] AI/ML
• [x] APIs/Backend
• [ ] Voice Assistant
• [ ] Extension/Plugin/Add-On
• [x] Developer Tooling
• [ ] Design/UX
• [ ] AR/VR
• [ ] Bots
• [ ] Security
• [ ] Blockchain
• [x] Futuristic Tech/Something Unique

Project Test Case

A simple binary module created in Go lang for generating hashes (md4, md5, sha512, etc) was published here and used as a test case for the project.

The module was created with a custom wrapper to use the modules directly in the projects below:

1. Laravel PHP
2. Node Express

Summary

The project test case was a success and was able to function without errors on live hosting like Heroku. We hope to bring more developers to join this project to help share their experience and skills to make the project beneficial to everyone. Thank you

[sherlock1453]

This is an interesting idea, but I see security as a major concern, what if I inject some malware in the module that I publish. Need to figure out a way to either make the binary follow principle of least privilege OR sandbox the binaries

[KOLANICH]

I guess this all is not needed. Lot of languages allow shared libraries. These shared libraries can be installed with a system package manager like apt. Lot of languages allow to interface binary shared libraries. These interface modules can also be installed with a system package manager like apt. Problem solved.

[iNerdStack]

This is an interesting idea, but I see security as a major concern, what if I inject some malware in the module that I publish. Need to figure out a way to either make the binary follow the principle of least privilege OR sandbox the binaries.

Thank you for your concern. I was expecting this issue with the security concern. There are a few solutions we have to this;

1. Package modules published will be verified before being made available to all. With sandbox and many other test cases to verify the module integrity.
2. Module source code has to be published on a public repository and the modules can be built directly from the source code for each version rather than depending on the publisher to upload the module themselves.

[iNerdStack]

I guess this all is not needed. Lot of languages allow shared libraries. These shared libraries can be installed with a system package manager like apt. Lot of languages allow interfacing binary shared libraries. These interface modules can also be installed with a system package manager like apt. Problem solved.

It's great to know there are other means to making this possible but I think the key thing is that the modules are made to be cross-platform meaning can be deployed to (windows & Linux servers) seamlessly, also independent of installations to the server os itself. Shared libraries are sometimes designed to work with specific OS.

One of the core concepts of the project is to provide a seamless experience with using these modules in different projects, language or server. With a console application, you can pull the desired module you need into your project and the right module binaries that matches your OS and also with a custom wrapper made for your language being integrated seamlessly.

[KOLANICH]

It's great to know there are other means to making this possible but I think the key thing is that the modules are made to be cross-platform meaning can be deployed to (windows & Linux servers) seamlessly, also independent of installations to the server os itself. Shared libraries are sometimes designed to work with specific OS.

If one needs cross-platformity there are 3 choices:

1. compile for each platform separately and use the binary for that platform.
2. use a userspace virtualization solution like qemu
3. use a bytecode virtual machine for that platform. Such as ones jitting wasm or GraalVM.

None of the variants completely avoid porting the code to the target platfkrm.

AFAIK 1. is the solution the most compatible to shared library approach. 2. is only compatible to standalone executables approach. 3. I guess is compatible mostly to executables. But to libs too. GraalVM transforms programs of multiple langs into the same bytecode and runs it over jit. JS, ruby, python, llvm bitcode and so on.

[iNerdStack]

It's great to know there are other means to making this possible but I think the key thing is that the modules are made to be cross-platform meaning can be deployed to (windows & Linux servers) seamlessly, also independent of installations to the server os itself. Shared libraries are sometimes designed to work with specific OS.

If one needs cross-platformity there are 3 choices:

1. compile for each platform separately and use the binary for that platform.
2. use a userspace virtualization solution like qemu
3. use a bytecode virtual machine for that platform. Such as ones jitting wasm or GraalVM.

None of the variants completely avoid porting the code to the target platfkrm.

AFAIK 1. is the solution the most compatible to shared library approach. 2. is only compatible to standalone executables approach. 3. I guess is compatible mostly to executables. But to libs too. GraalVM transforms programs of multiple langs into the same bytecode and runs it over jit. JS, ruby, python, llvm bitcode and so on.

Thank you so much for the insight. Interesting!

I would love to know if there’s a global package system in place using these technologies and if they seamlessly integrate these usage into a project/language/framework regardless of their language and server choice.

Is there a software or console application that provides a terminal interface to pull these binaries into their projects and also systematically create the bridge?

Also is there a way people can register/submit their binaries project without worrying about how it will be integrated globally with any project?

[iNerdStack]

@KOLANICH i would love to mentioned that the project aims to create a universal package/module system for all.

Binaries have seems to be the pick to building that bridge. The project can evolve beyond the binaries system.

Several functionalities/solutions are created in different language communities and are usually not easy to use directly in another language without using microservice system etc. Sometimes one have to depend on the available packages in your language ecosystem.

For example, Different file converter modules are made available on npm(node), same with composer(php) etc. Some have better functionality, contributors strength than the other but will only be limited to their language source.

With a universal package system. It’s possible a great converter package in npm can be compiled to a binary and with logic of a universal package system can be pull and integrated seamlessly into a php project, dotNet project etc.

With the evolution of this, we can have wide access to great number of packages

Projects will also be able to grow beyond depending heavily on its language package ecosystem

[KOLANICH]

universal package/module system for all.

You can be interested in looking at #50

[iNerdStack]

universal package/module system for all.

You can be interested in looking at #50

It is interesting to know the concept is the same but the architectural approach is different

From what I could understand

Similarities

1. Use of a universal package manager
2. Use of a frontend, backend, and also middleware system
3. Environment independence

Differences	Current Project	#50
Packages are built or converted to executable binaries and used with an auto integration system	A frontend interface/plugin is used in reading packages formats and invoking operations of all other package managers
Custom packages (unpublished) can also be integrated with a project with the help of the console/terminal application	Packages might have to be published on other package managers to work with the package management system.
More workload on package binaries verification and security checks	Lesser concern on security and integration of packages used through the system.

There is more to the differences in the approach. The goal is basically the same and would need a whole lot of expertise and shared experiences from a wide range of developers and software engineers to make the project possible with the best possible functionalities and seamlessness.

[KOLANICH]

Lesser concern on security and integration of packages used through the system.

Since the One package manager should interoperate with the repos of existing package managers, it has to use the security solution already used in the specific package managers. Fortunately they tend to converge to The Update Framework.

[md-shah]

Can we do something like a "Generic WebAssembly Bridge"? So that we can execute/compile the packages to wasm and run the wasm inside a sandbox which can be a cross-platform or something?

[KOLANICH]

https://wapm.io/

[ugurkorkmaz]

For a project like this, we can use GRPC and containers. All the tools and modules will be built to follow the GNU philosophy and work on their own. Then, we can create client packages for different programming languages using GRPC to access these tools or modules. However, it still seems like a huge project to me, and it needs a good infrastructure design.