search

DonyaOS / Donya

Donya is an Operating System. Yet Another Linux distro built using a package management system.
https://donyaos.github.io/Donya/
MIT License
59 stars 15 forks source link

🦸‍♀️ [OFFICIAL] Talk about Development 🦸‍♀️ #5

Open BaseMax opened 4 years ago

BaseMax commented 4 years ago

Hi,

The DonyaOS is not yet ready and complete and we need the help of others.

Here's a chat room.

We can talk about the tasks we want to work on here. And set a development path for ourselves.

Study and research can be one of the things that will always be needed in this project.

Please state if something goes wrong. I appreciate you.

BaseMax commented 4 years ago

At this point, we have a script that can generate a new Linux distribution and produce .ISO image file. https://github.com/DonyaOS/Donya/blob/815216b91e7df81c9d3aebc1217cd9a2b3d698b3/donya.sh

But this is not enough.

We need to separate the scripts and have such a separate file. (Not a single .sh file)

For example, Scripts:

Also Makefile can help to compile quickly.

ghost commented 4 years ago

With Donya's package manager, I'd suggest a packaging system like KISS Linux (here's an overview: https://k1ss.org/package-system). What did you have in mind?

ghost commented 4 years ago

OK, with package management, this is what I had in mind—just a suggestion, I'm not entirely sure what your plans are:

We make a repository in the DonyaOS GitHub organization, and we call it packages or something to that effect. This will be the default source of packages in the package manager. Then, the package manager (which I'll call dpak until we come up with a proper name) will look for packages within that GitHub repository, at the root, by filename; so if we were to do something like: dpak install ExamplePackage, then dpak would read the data in the ExamplePackage.yml file at the root of the packages repository. This is similar to how the Homebrew package manager on macOS works.

Each package file in the repository will have several standard data fields, which I suggest we represent in YAML. These fields will be description, license, homepage, dependencies, binary (optional), and install. The package manager will show the package metadata (including the list of dependencies) before prompting the user if they want to install (like with APT). The install variable is a series of commands which will be ran by the system to get the package installed, useful for if the package is compiled (like how Gentoo handles packages), not a binary. In the event a package is binary-only, then the binary variable will point to the URL of the binary release, and then dpak will simply install it into the system. Here's an example of a package called dlib; stored in DonyaOS/packages/dlib.yml:

description: "C++ library for machine learning"
homepage: "http://dlib.net/"
binary: "http://dlib.net/files/dlib-19.21.tar.bz2"
license: "BSL-1.0"
dependencies:
- cmake
- jpeg
- libpng
- openblas
install: |
  git clone https://github.com/davisking/dlib.git
  cd dlib-master
  mkdir build
  cd build
  cmake ..
  cmake --build . --config Release

This package manager model has several key differences over currently existing package managers:

  1. It's simple—package repositories are easy to create, because they're just GitHub repositories
    • APT and Snapcraft struggle greatly in this department. Packaging software, and then publishing it into a repository, is a difficult process for developers who want to get their work installed via those package managers. Moving everything to a simple, GitHub-centric system will make publishing easier.
  2. It allows for compiling packages rather than just getting binaries
    • This is only really seen in Gentoo nowadays. Making a simpler alternative to Portage is a good idea.
  3. Greater GitHub integration; by making each software source a GitHub repository, it will force software publishers to become familiar with the open-source ecosystem, particularly if they wish to publish in the primary DonyaOS package repository. This promotes open-source development.
BaseMax commented 4 years ago

Thanks for your suggestion. I did create packages repository in this organization.

https://github.com/DonyaOS/Packages

BaseMax commented 4 years ago

We need to design the package manager software. Something similar to apt in ubuntu, pacman in Arch, pkg in FreeBSD.

I think GoLang is a good choice for this purpose. C is also good.

Easy command line is very important.

We can use an alias:

For example:

d i php ; install php version 7.1
d i php7.4 ; install php version 7.2
d i gcc ; install gcc

d s php ; search all package with php perfix

d r php ; remove php package
d r php* ; remove all php prefix package

d i php* ; install all php prefix package

Display loading while downloading and package information before installing is a good feature. Centos has this feature as well.

Ubuntu apt has a bad feature, this forces you to remove lock files if you stop this using ctrl+x, ctrl+c etc.

BaseMax commented 4 years ago

Hi,

Does anyone want to work on the package manager or package service? @sudo-corvus @limeschool Or others We welcome this help. I think GoLang is a good choice for this purpose.

Please tell me your comment. Thanks,

BaseMax commented 4 years ago

PackageManager: It's a client software, this should be installed at DonyaOS in default. https://github.com/DonyaOS/PackageManager

Packages: This is an API. This is a web service, PackageManager will get his data from this repository.

https://github.com/DonyaOS/Packages

For example: If PackageManager wants to search in packages list, should get a list of packages from the Packages repository. and save this as a cache.

I think we can host packages files in GitHub. Otherwise, we can create a server and host on that. No problem with this.

amir-shiati commented 4 years ago

Hey @BaseMax i'm familiar with GoLang and i can help you guys with this! Just let me know where to begin...

BaseMax commented 4 years ago

Hey Amir, @amir-shiati Thanks for letting me know you agree to work on it. I gave you full access. Therefore, you can submit and send changes in the desired repository without any permission. Thanks,

ghost commented 4 years ago

I'm not familiar with Go or C. I'll try to make my own package manager in D, and then we can compare and pick which is best.

EDIT: D was not a good choice. Writing it in C# right now. It should perform rather well, with the time to complete an operation being almost entirely package-dependent and not noticeably impacted by any overhead.

prince-dsd commented 4 years ago

I will take this up in C I have been going through RPM to understand the process

BaseMax commented 4 years ago

Hi, @amir-shiati is working to create Package Manager. Can you please start working on the Packages repository? @sudo-corvus, and @limeschool

prince-dsd commented 4 years ago

all right on it @BaseMax

ghost commented 4 years ago

Just as a side project:

I've made a package manager in C# that works rather fast and does not require installing any dependencies to use it.

It's called gany and does not use the KISS structure currently in the DonyaOS/Packages repository, but rather a simpler one where each package is a single file. For an example of this, see my own package repository which has a package definition for gany itself, allowing for self-updating. gany can build packages, and it can download binary versions as well. Pretty neat, though a little incomplete right now.

Since you seem to be going with the KISS package structure, however, what packages should @sudo-corvus and I put in DonyaOS/Packages? Should we just copy over the default KISS software repository?

BaseMax commented 4 years ago

Hi Emil,

I was excited to see your activity. Where is package definition of the gany? (Your link is broken)

Anyway, you has full-access to this organization and can push your code/commits directly.

I've made a package manager in C#...

I not sure about this, Usually, C# is not a good choice for Linux commands And this is not a common thing.

This is a good language though. But C# is compiled into IL, not binary.

To install gany, you need 'libkrb5-dev' & '.NET Core 3.1' to build...

However, if this is necessary, We can think of rewriting it in another language. @amir-shiati, Have you done anything? This can give him a good view of how to develop a package manager.

KISS package structure...

I just use this as the first version for a starter package.

We can think about and develop a new structure. Anyway, I like your idea about the KISS structure. And this is great for most purposes.

For example ffmpeg: https://github.com/kisslinux/repo/tree/master/extra/ffmpeg

build
checksums
depends
sources
version

Please show me your current packages repository.

Update: https://github.com/limeschool/packages

You can add some sample packages just for testing: https://github.com/kisslinux/repo/tree/master/core/curl

Thanks,

BaseMax commented 4 years ago

Please push your repositories to the Donya organization. @limeschool

Please explain where the name gany comes from? donya and d was not a short and better name?

Some feature needed for package manager:

In your structure file size is not calculable. (This is missing side by side with bash commands)

BaseMax commented 4 years ago

A sample: https://github.com/DonyaOS/Packages/blob/master/core/curl/package.donya

Direct HTTP/S request: https://donyaos.github.io/Packages/core/curl/package.donya

In this case, But we need to know the name of the group. (core at this sample) Otherwise, we have to not group the packages.

BaseMax commented 4 years ago

If a package need multi-file to download. How we can control run make/configure in different downloaded files?

I no mean dependency. I mean multi required file to download.

https://github.com/DonyaOS/Packages/commit/9947725b08c32e8a63161259806b0ae2bd5bf785

We can do unrar x, unzip, and tar xvf automatically in the package manager.

But I think we have to write cd manually... What do you think?

BaseMax commented 4 years ago

About the package manager:

Consider that before the installation step (install in the package file), All required packages must be installed.

This must be done in a recursive manner. Because the required packages may also depend on other packages.

amir-shiati commented 4 years ago

Hey everyone! this is a simple diagram showing how the package manager is going to work, please take a look at it and let me know if any other blocks needs to be added to the diagram... Untitled Diagram

amir-shiati commented 4 years ago

And also this is an example of how a package would be installed: lets say the user wants to install curl: $ donya install curl or $ donya i curl

the package manager would send a request to the package manager api, the package manager api would search for curl and then return (if exists) the url of curls "package.donya" file which is for example this, the package manager would download the file and open it, then after that: 1- it would install all the required packages in a recursive manner. 2- it would download the package file. 3- it would check the checksum to make sure that the downloaded file is not corrupted. 4- it would run the given instructions to install package.

Let me know what you guys think...

BaseMax commented 4 years ago

Thanks, Amir, That's fine. @amir-shiati

BaseMax commented 4 years ago

Who is currently responsible for designing the API? @sudo-corvus, and @limeschool We have already assigned two people.

One of our problems is to host files at GitHub. Do we will can to develop API and web server based on GitHub?

We can set up files on own server, and connect the GitHub repository to server using an SSH Action.

But I think we can follow a better way.

Update:

Also, Using the GitHub API might be another way to do this. https://developer.github.com/v3/ https://developer.github.com/v3/search/#search-code

amir-shiati commented 4 years ago

Hey everyone, For hosting the API oh github i came across this, However the api is static.

BaseMax commented 4 years ago

Thanks for sharing this. It's awesome. @MatMoore

https://github.com/MatMoore/jargon

By using github pages you are accepting their usage limits which means that your API could still go down if it gets too much traffic, and they don’t recommend using it for commercial purposes:

GitHub Pages is not intended for or allowed to be used as a free web hosting service to run your online business, e-commerce site, or any other website that is primarily directed at either facilitating commercial transactions or providing commercial software as a service (SaaS).

https://docs.github.com/en/github/working-with-github-pages/about-github-pages

BaseMax commented 4 years ago

We cannot implement a powerful search API using GitHub pages. GitHub pages execute a script file(such as ruby) and create folder/files.

You can watch the gh-pages branch of the mentioned example: https://github.com/MatMoore/jargon/tree/gh-pages

We can cache a list of packages and store this in local system, finally search between packages using package manager client software.

BaseMax commented 4 years ago

Hi @limeschool, Thanks for your commits and message.

I do not think we can find an algorithm that finds the exact file-size of compiled files from the source code. However, we can specify the approximate size of the data in the package information.

Can you combine the test branch with the main branch in Packages repository? We need to go to the next steps.

BaseMax commented 4 years ago

The figure of pkg in FreeBSD:

root@base:~ # pkg install php74
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 4 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
    libargon2: 20190702
    libxml2: 2.9.10
    pcre2: 10.35
    php74: 7.4.8

Number of packages to be installed: 4

The process will require 40 MiB more space.
6 MiB to be downloaded.

Proceed with this action? [y/N]: wrong_key
Please type 'Y[es]' or 'N[o]' to make a selection

Proceed with this action? [y/N]: y
[1/4] Fetching php74-7.4.8.txz: 100%    4 MiB   4.2MB/s    00:01    
[2/4] Fetching libxml2-2.9.10.txz: 100%  828 KiB 848.0kB/s    00:01    
[3/4] Fetching libargon2-20190702.txz: 100%   64 KiB  65.5kB/s    00:01    
[4/4] Fetching pcre2-10.35.txz: 100%    1 MiB   1.3MB/s    00:01    
Checking integrity... done (0 conflicting)
[1/4] Installing libxml2-2.9.10...
[1/4] Extracting libxml2-2.9.10: 100%
[2/4] Installing libargon2-20190702...
[2/4] Extracting libargon2-20190702: 100%
[3/4] Installing pcre2-10.35...
[3/4] Extracting pcre2-10.35: 100%
[4/4] Installing php74-7.4.8...
[4/4] Extracting php74-7.4.8: 100%
BaseMax commented 4 years ago

The figure of pkg search in FreeBSD:

root@base:~ # pkg search mariadb | grep 10.3
mariadb103-client-10.3.23      Multithreaded SQL database (client)
mariadb103-server-10.3.23      Multithreaded SQL database (server)
root@base:~ # pkg search mariadb
mariadb-connector-c-3.1.9      MariaDB database connector for C
mariadb-connector-odbc-3.1.7_1 MariaDB database connector for odbc
mariadb103-client-10.3.23      Multithreaded SQL database (client)
mariadb103-server-10.3.23      Multithreaded SQL database (server)
mariadb104-client-10.4.13_4    Multithreaded SQL database (client)
mariadb104-server-10.4.13_4    Multithreaded SQL database (server)
rubygem-azure_mgmt_mariadb-0.17.3 Microsoft Azure Microsoft Azure MariaDB Library for Ruby Client Library for Ruby
BaseMax commented 4 years ago

Hi. Where did the package manager get to and how did it happen? @amir-shiati https://github.com/DonyaOS/PackageManager

BaseMax commented 4 years ago

Welcome @jbampton,

Because you are good at Go Language. Can you help with the package manager? Currently, one person works on it, And development is very slow and needs to help.

This is the last thing I saw or heard about the Package Manager and I have no news. @amir-shiati demo-donya

As Amir said: I know that this package is going to help: https://github.com/spf13/cobra (CLI interactions)

Regards, Max

BaseMax commented 4 years ago

Hi, Good news: Since esmaeel works, we will add Grub to the project this week.

It seems that Amir does not have the opportunity and there is no news about the package manager yet.

I have not received an answer yet, You are expected to at least respond even if you do not do anything.

He did the initial part, but we do not have access to the code. Sorry, Maybe I call to Amir today. @amir-shiati

We invite another member to continue, we need a package manager sooner.

Can you do this? @jbampton -- Regards, Max

BaseMax commented 4 years ago

Hey, Can you adding at least most useful packages to repository? The more, the better... @sudo-corvus

No problem if there is something wrong with the commands. We can improve step by step. -- Regards, Max

BaseMax commented 4 years ago

I would appreciate if anyone can continue the package manager. I created the file and load the cobra library.

If cobra is not suitable to needs, we can use another library. There is no limitations. Or even pure code :) cc @amir-shiati

https://github.com/DonyaOS/PackageManager/blob/master/src/donya.go

Are you ready to continue? @jbampton

jbampton commented 4 years ago

Hi @BaseMax !! Yes I would like to help on this go code. I am really tired here now and it's only lunch time.

I need to read this whole issue fully once again.

I have not used cobra before. I did use -> https://github.com/urfave/cli

BaseMax commented 4 years ago

Hey, Thanks,

No limitation for CLI library. Enjoy by way :)

P.S: Just a small draft for Package Manager in PHP 7.4, to show how we can control arguments. (Pure code in PHP) https://github.com/BaseMax/php-cli-apps

amir-shiati commented 4 years ago

Hey everyone. @BaseMax i do apologize for late response. I have been really busy this whole week. I will get back to you guys as soon as i'm done with this client. wish you all the best.

BaseMax commented 4 years ago

Hey Amir,

Thank you for sending files to me. John, Look at Amir source code, it's great. modular programming... @jbampton Separate file for different functions of Donya package manager. This was a good idea, and can help us to make source code more clear.

With the permission of Amir. file: donya.zip

We can follow this structure while developing package manager.

What do you think?

Update:

Thanks to Amir: https://github.com/DonyaOS/PackageManager/pull/5

BaseMax commented 4 years ago

Hey friends,

How you will installing/downloading a package in the package manager? @jbampton You need to clone Package repositories as cache? Or will need a API?

If need a API: We will need a API or somethings like this for Searching or finding packages. @sudo-corvus, and @limeschool

Otherwise: We will not need a API. Only download zip file: https://github.com/DonyaOS/Packages/archive/master.zip, then unzip and save as cache. e.g: $ donya update, will update local repository.

Related messages:

ivandavidov commented 4 years ago

Hi guys,

Thank you for inviting me to this project!

Here is what I think:

I'll keep an eye on the project development.

Regards, Ivan Davidov

BaseMax commented 4 years ago

Hello everyone,

Thanks for the effort of @esmaeelE We managed to get to the next stage of work.

The previous script is now obsolete. And new scripts from step1.sh, and step2.sh, step3.sh to step5.sh are. We need help to clean and improve the scripts.

This will require several files whose links are as follows:

Unfortunately I could not upload the files directly to the repository. (Limitation is 100MB)

For this reason, I will soon write a script that he can download with the help of wget if he does not have the desired file.

Regards, Max

BaseMax commented 4 years ago

Regarding kernel dependencies, I wrote a script to automatically download all the required files. If possible, please check and test: dependencies.sh

If you have a Linux (VM), So you can test and run scripts. Ask a question, if you face a problem or error while running scripts.

We need feedback and want to improve scripts. Please send PR.

esmaeelE commented 4 years ago

donyaOS todo list

First things first

About 5 step scripts

Then

Other things

  1. kernel

  2. base layer_0: bootloader, busybox, more essential tools for compliting busybox layer_1: drivers, nettools, package manager layer_2: window manager, tmux, base packages for other apps (compiler collection)

  3. application: all application that user desired

All of them is my opinion, feel free to express anything wrong and must be change beside that as English is not my mother tongue please correct me.

I thing this can be used as a todo for donyaOS

BaseMax commented 4 years ago

Hi, Some new changes were made to the repository, sorry for the separate commits.

Also a guide references for users who want to learn more:

Some related links

Thanks to @esmaeelE

esmaeelE commented 4 years ago

The Next proposed version of donyaOS release totally based on the Petros Koutoupis article on Linux journal.

  1. DIY: Build a Custom Minimal Linux Distribution from Source
  2. Build a Custom Minimal Linux Distribution from Source, Part II We want to extent it as a base for future developments

In the next step I will

  1. Solve donyaOS script issues, some of them depicted in the picture.

  2. Make an iso image that targets X86/64 architecture to install donyaOS