scottwilliambeasley / debian-from-scratch

An instruction manual for teaching Linux From Scratch users how to make a fully-fledged Debian system based on LFS.
243 stars 49 forks source link

Outdated #5

Open tcreek opened 4 years ago

tcreek commented 4 years ago

According to this manual, it states "These guides are outdated and no longer contain up-to-date information, which I intend to fix here in this manual."

It's been four years for the intended fix with an update which reflects version 9 of Linux from Scratch, and now Debian 10 which was recently released.

asp3n-hayley commented 4 years ago

According to this manual, it states "These guides are outdated and no longer contain up-to-date information, which I intend to fix here in this manual."

It's been four years for the intended fix with an update which reflects version 9 of Linux from Scratch, and now Debian 10 which was recently released.

There's not enough context in your issue. Here is the full quote:

"I chose to make this manual because I have seen woefully old guides on the internet teaching others how to get dpkg and apt running on their own custom Linux, and people asking on various forums on how to install dpkg and apt, but not getting the help that they need. These guides are outdated and no longer contain up-to-date information, which I intend to fix here in this manual."

The "oudated guides" are other guides the dev has seen on the internet, not this one. While this guide is made for a (slightly) older version of LFS, not enough has changed in the book for this to become obsolete, especially since those with the skills to build an LFS system should be able to account for any difficulties that may arise in the conflict between versions.

tcreek commented 4 years ago

Here is an example.

Version 9 of Linux from Scratch is using GCC 9.2.0. Meanwhile, Debian 10 (Buster) is using gcc 8.2.

In Linux from Scratch, they go on to use gcc 9.2.0 for the base system build Does this make any difference?

Also the package gnupg has been depreciated to gnupg2. How does this affect the built?

This is why an update is needed

tcreek commented 4 years ago

I am currently building this, so I guess I can add another issue to the pile on why it needs to be updated.

Trying to get dpkg compiled and already running into a problem with the configure while in the chroot environment listed here

bash: ./configure: /bin/sh: bad interpreter: No such file or directory

asp3n-hayley commented 4 years ago

Here is an example.

Version 9 of Linux from Scratch is using GCC 9.2.0. Meanwhile, Debian 10 (Buster) is using gcc 8.2.

In Linux from Scratch, they go on to use gcc 9.2.0 for the base system build Does this make any difference?

Also the package gnupg has been depreciated to gnupg2. How does this affect the built?

This is why an update is needed

You make good points. And I concur, that this needs to be updated. However, dpkg is a package manager, not an OS, so version conflicts with the OS and LFS do not necessarily reflect those concerning the package manager. And it's easier to update gcc on the host system than downgrade it because LFS uses an older version, if that makes any sense.

as far as the error you mentioned, are you certain that /bin/sh is a symbolic link to /bin/bash? I don't think that error has anything to do with this repo(albeit, my knowledge is limited, so dont take my word for it).

Good luck with your build, though! When you complete it, if you go so far as to make it an OS with an installer/package manager, etc, feel free to reach out to me as I'd love to test it. I'm always on the lookout for the "perfect distro" which is why i got started with LFS

tcreek commented 4 years ago

I went ahead and started over on the build just in case I made a mistake. Does not take that long now since I have everything I am doing put into a text file so I can copy and paste quickly.

With that, I am working on bringing back a popular Distro in which Microsoft bought out for $22 million to kill it. It was so popular, you could purchase a refurbished HP desktop computer with it pre-installed as the OS at Fry's Electronics

Microsoft allowed the domains to lapse after 10 years, and I managed to get the .org one.

If I have any more of the same issues

tcreek commented 4 years ago

I redid the chapter 5 build. and having the exact same issue as before when in chroot for running configure for dpkg.

bash: ./configure: /bin/sh: bad interpreter: No such file or directory

tcreek commented 4 years ago

Here is building BASH in version 9 of LFS

./configure --prefix=/tools --without-bash-malloc make -j4 make install ln -sv bash /tools/bin/sh

tcreek commented 4 years ago

I think I have an idea what could be the problem.

Since I have already created a LFS build on the same system as before, there could be some conflicts with the setup.

I think I will have to create a virtual machine environment and give it a test. Give me a day or so, and I will respond back.

asp3n-hayley commented 4 years ago

I went ahead and started over on the build just in case I made a mistake. Does not take that long now since I have everything I am doing put into a text file so I can copy and paste quickly.

With that, I am working on bringing back a popular Distro in which Microsoft bought out for $22 million to kill it. It was so popular, you could purchase a refurbished HP desktop computer with it pre-installed as the OS at Fry's Electronics

Microsoft allowed the domains to lapse after 10 years, and I managed to get the .org one.

If I have any more of the same issues

I've started my build over numerous times, albeit never finished one, so no worries there. I actually have a repo on GitHub with the wget-list and version-check.sh scripts, and also a brief script that I wrote that automatically mounts all virtual filesystems and whatnot so if you resume work after a reboot you're back where you left off in chroot after running the script.

I redid the chapter 5 build. and having the exact same issue as before when in chroot for running configure for dpkg.

bash: ./configure: /bin/sh: bad interpreter: No such file or directory

That's interesting....It still seems like a symlink issue, but it could be related to the DFS project, calling to evidence my issue, where a crazy systemwide issue appeared after compiling this(again, probably my fault).

Here is building BASH in version 9 of LFS

./configure --prefix=/tools --without-bash-malloc make -j4 make install ln -sv bash /tools/bin/sh

Everything looks could....a couple of suggestion off the top of my head: that error you're getting is related to the system thinking bash is not installed. First thing: on your rebuild, when building bash for the temporary toolchain, build it in single-core mode instead of using -j4(it can sometimes cause issues when building). [After skimming the section in LFS(v9, section 5.16) concerning bash, i see that the --without-bash-malloc flag prevents segmentation faults, so it's possible that i made a typo there when compiling mine, which resulted in the error i got(or maybe something i did when compiling glibc).] Second thing, try to run the configure script using the bash command instead of ./, like bash configure --prefix=/tools instead of ./configure --prefix=/tools. It's a shot in the dark, but my philosophy is leave no stone unturned.

I think I have an idea what could be the problem.

Since I have already created a LFS build on the same system as before, there could be some conflicts with the setup.

I think I will have to create a virtual machine environment and give it a test. Give me a day or so, and I will respond back.

That's an excellent idea, it's tricky to clean up the host after partially completing a build. I have around 15 2.5" hdds all with linux distros and a sata-to-usb cable so it's easy to start from scratch when i need too, although you have to keep an eye on /etc/fstab when using stuff like that. I doubt I've been much help, but I await your response, maybe if you get the same error again we can take it upon ourselves to fork this repo and update it, or even make a new one with pacman or yum(or something like that.

I found you first comment intriguing, about the distro you're building, and I feel like I know which distro you're talking about, but I can't for the life of me figure out what it is. What is the .org domain you aqquired? I commend your dedication to this project, however. I plan to try LFS again soon, but it's hard to find time haha.

asp3n-hayley commented 4 years ago

I was just looking at the README, are you using the commands in this to chroot or the ones in the LFS book? because that could have an effect on where /bin/sh is linked to

tcreek commented 4 years ago

I have tried both of the chroot command and it is the same result as before. The DLF has a more inclusive path.

Looking deeper at LFS Chapter 6, I see they create more directories, and symbolic links including a /bin directory., I think the old symbolic links to from/to that /bin is what could be affecting the build for DLF

For the distro I am trying to revive, I am not ready to plaster the name on in places like this yet because a lot more domains have been released since the original three public ones were available back then. Anyway it starts with L and rhymes with Windows. And that is why the reason MS bought them out. You can still get old books about it on eBay.

asp3n-hayley commented 4 years ago

I have tried both of the chroot command and it is the same result as before. The DLF has a more inclusive path.

Looking deeper at LFS Chapter 6, I see they create more directories, and symbolic links including a /bin directory., I think the old symbolic links to from/to that /bin is what could be affecting the build for DLF

For the distro I am trying to revive, I am not ready to plaster the name on in places like this yet because a lot more domains have been released since the original three public ones were available back then. Anyway it starts with L and rhymes with Windows. And that is why the reason MS bought them out. You can still get old books about it on eBay.

Oh ok, I see, just checking. And i think you might be right about the old symlinks.

and as far as the distro, say no more. I realize exactly what you are talking about, and I commend the project. I also saw the .org domain you spoke of, which leads me to my next point: I am a web developer, in case you want some help building the site. Good luck with the project

tcreek commented 4 years ago

I could use the help on a web site. If you visit the mail TLD you will see a basic Wordpress site setup with nothing on it. Thanks for the offer

asp3n-hayley commented 4 years ago

I could use the help on a web site. If you visit the mail TLD you will see a basic Wordpress site setup with nothing on it. Thanks for the offer

no prob. I'll work on it when i have time around school(i'm online so I have more time than on-campus students) so I don't have to charge you for it. If it's cool with you, my idea would be to just code it with HTML, CSS, and maybe a little JS and host it with github pages. Completely free, and at the end of it you have an awesome site. When you get time, send me some info on the design you'd like and content guidelines, but I'll get started right away. I'll make my own github repo for now, and when I'm finished you can just clone it and upload the code to your own, or to a hosting provider if you have one. It's really easy to connect your TLD to the github pages repo once you're ready to. What registrar do you own it through?

Is there any licensing issues we need to worry about btw? I know microsoft sued them way back when which was where the $20M came from, because microsoft lost. Just wanted to make sure we're all legal here haha

tcreek commented 4 years ago

That is fine for now. I guess I could use it for a year, or two before I need a more advanced site. I have no guidelines. Maybe take ideas from other distro sites, and put in what you like about each one.

I am using NameCheap

No licensing issues to worry about. Thanks for helping out!

asp3n-hayley commented 4 years ago

That is fine for now. I guess I could use it for a year, or two before I need a more advanced site. I have no guidelines. Maybe take ideas from other distro sites, and put in what you like about each one.

I am using NameCheap

No licensing issues to worry about. Thanks for helping out!

Ok perfect, I'll take care of the design. Namecheap is great, so when the time comes, it will be no prob to connect the domain. And good to know, no problem!

The only thing I need to know is branding. Are we just copying the original, or making the same project under a different name? If you don't want to discuss it here you can reach me by email at ironmatt613@gmail.com.

tcreek commented 4 years ago

I have some extra mother boards and SSDs laying around so I build a clean Debian 10 system

I am still getting the same error as before.

Looking at the choot command, I see it is nearly same as LFS:

/tools/bin/bash --login +h

This means it is using bash in the /tools/bin directory from the chapter 5.16 Bash section of LFS.

the dpkg configure file has #! /bin/sh at the first line.

In the non chroot environment we a symbolic link from sh pointing to bash

lrwxrwxrwx 1 root root 4 Jan 7 00:34 sh -> bash

So I see there is none in the chroot environment.

I tried making some from bash to sh without success.

ln -sf bash /tools/bin/sh ln -sf /bin/sh /tools/bin/sh

Seems project owner is not responding, so maybe time to try the email he has listed

asp3n-hayley commented 4 years ago

I have some extra mother boards and SSDs laying around so I build a clean Debian 10 system

I am still getting the same error as before.

Looking at the choot command, I see it is nearly same as LFS:

/tools/bin/bash --login +h

This means it is using bash in the /tools/bin directory from the chapter 5.16 Bash section of LFS.

the dpkg configure file has #! /bin/sh at the first line.

In the non chroot environment we a symbolic link from sh pointing to bash

lrwxrwxrwx 1 root root 4 Jan 7 00:34 sh -> bash

So I see there is none in the chroot environment.

I tried making some from bash to sh without success.

ln -sf bash /tools/bin/sh ln -sf /bin/sh /tools/bin/sh

Seems project owner is not responding, so maybe time to try the email he has listed

Interesting....So the problem is not some leftover files or symlinks from a previous build. try the chroot login command as /tools/bin/sh --login +h instead of /tools/bin/bash, because that error you're getting is a very straightforward issue, it's telling you that it's trying to use a bash interpreter in /bin/sh(relative to chroot, so should be /tools/bin/sh) but there isn't one there. Since you made the symbolic link to /bin/sh during compilation of the package, i think logging in with /tools/bin/bash might be causing a problem. Let me try sometime this week and we'll see if I get the same error and if it's something we're doing or something from DFS. My name is Matt by the way

tcreek commented 4 years ago

There is no /tools/bin/sh thus it fails.

I did try to make a /bin then once in chroot make a symbolic link with "ln -sf bash /bin/sh" The link is created

$> ls -l /bin total 0 lrwxrwxrwx 1 0 0 4 Jan 7 17:33 sh -> bash

but still I get the same error.

I sent the project owner an email 12 hours ago.

I will go very close over in what LFS is doing in Chapter 6 to not get that issue.

asp3n-hayley commented 4 years ago

There is no /tools/bin/sh thus it fails.

I did try to make a /bin then once in chroot make a symbolic link with "ln -sf bash /bin/sh" The link is created

$> ls -l /bin total 0 lrwxrwxrwx 1 0 0 4 Jan 7 17:33 sh -> bash

but still I get the same error.

I sent the project owner an email 12 hours ago.

I will go very close over in what LFS is doing in Chapter 6 to not get that issue.

That's odd...../tools/bin/sh should be created during the compilation of bash-5.0 as outlined in the LFS book:

Prepare Bash for compilation:

./configure --prefix=/tools --without-bash-malloc The meaning of the configure options: --without-bash-malloc This option turns off the use of Bash's memory allocation (malloc) function which is known to cause >segmentation faults. By turning this option off, Bash will use the malloc functions from Glibc which >are more stable. Compile the package: make Compilation is now complete. As discussed earlier, running the test suite is not mandatory for the >temporary tools here in this chapter. To run the Bash test suite anyway, issue the following >command: make tests Install the package: make install Make a link for the programs that use sh for a shell: ln -sv bash /tools/bin/sh #This is what creates the shell in /tools/bin/sh

You seem knowledgeable enough that I doubt you're making a mistake here, so there is something that is screwing up that symlink that we're not seeing...I'll try to do this on my system if I get time and let me know if you hear back from the project owner....maybe try contacting the folks at LFS as well, I'm sure they could provide some insight

asp3n-hayley commented 4 years ago

I think if you're making a symlink at this point it's not going to be in chroot, because the temporary toolchain is built from the host system, so any symlink you make should point to /tools/bin/sh not /bin/sh

tcreek commented 4 years ago

Actually, it is in chroot.

I see what LFS is doing. I think this project is wrong.

in LFS Chatper 6 after entering chroot

mkdir -pv /{bin,boot,etc/{opt,sysconfig},home,lib/firmware,mnt,opt} ln -sv /tools/bin/{bash,cat,chmod,dd,echo,ln,mkdir,pwd,rm,stty,touch} /bin ln -sv bash /bin/sh

...and it's working now, but still failed for other reasons,

config.status: error: in /sources/dpkg-1.19.7': config.status: error: Something went wrong bootstrapping makefile fragments for automatic dependency tracking. Try re-running configure with the '--disable-dependency-tracking' option to at least be able to build the package (albeit without support for automatic dependency tracking). Seeconfig.log' for more details

This is why this badly needs an update

asp3n-hayley commented 4 years ago

Actually, it is in chroot.

I see what LFS is doing. I think this project is wrong.

in LFS Chatper 6 after entering chroot

mkdir -pv /{bin,boot,etc/{opt,sysconfig},home,lib/firmware,mnt,opt} ln -sv /tools/bin/{bash,cat,chmod,dd,echo,ln,mkdir,pwd,rm,stty,touch} /bin ln -sv bash /bin/sh

...and it's working now, but still failed for other reasons,

config.status: error: in /sources/dpkg-1.19.7': config.status: error: Something went wrong bootstrapping makefile fragments for automatic dependency tracking. Try re-running configure with the '--disable-dependency-tracking' option to at least be able to build the package (albeit without support for automatic dependency tracking). Seeconfig.log' for more details

This is why this badly needs an update

Wait the temporary toolchain is built in chroot? Sheesh it's been too long since I built an LFS system, sorry about that.

That's strange....and yeah you're definitely right that this needs an update...I'm going to look into other guides for setting up automatic package management for LFS, maybe with pacman or something. Hopefully the owner will update this sometime, although it looks like the project might have been abandoned

asp3n-hayley commented 4 years ago

what does the config.log say for that compilation attempt, just out of curiosity?

tcreek commented 4 years ago

Looking further at the guide, I see he does have some errors:

When we get to the point of installing debian's "priority:essential" packages, which include both of these shells, these symlinks will be overwritten with native copies of these binaries.

mkdir /bin ln /tools/bin/bash /bin/bash ln /tools/bin/bash /bin/sh

This is showing being done AFTER compiling dpkg, when it needs to be done BEFORE.

Anyway, I followed the advice of the error, and dpkg is completed.

asp3n-hayley commented 4 years ago

that's funny, I was just reading the README and was about to point that out.

dpkg is completed without dependency tracking or with?

asp3n-hayley commented 4 years ago

Because that defeats the purpose of it without, although now that you have the tool you might be able to find a workaround.

You should fork this repo and update it with the correct instructions and updated packages for LFS 9.0

tcreek commented 4 years ago

I was forced to do it this way.

./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --build=x86_64-unknown-linux-gnu --disable-dependency-tracking

I really have no experience with compiling or using dpkg so I should research this tracking.

asp3n-hayley commented 4 years ago

I was forced to do it this way.

./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --build=x86_64-unknown-linux-gnu --disable-dependency-tracking

right, makes sense. Like i said though, now that you have the package installed, even without full functionality, you just might be able to find a workaround with a bit of manual dependency tracking until you get the full functionality installed, POSSIBLY.

I really have no experience with compiling or using dpkg so I should research this tracking.

Nor I, honestly, but I know that dependency tracking is what makes installing dpkg on an LFS system a good think haha. It basically throws in a bit of automation to installing from source, so if you install package x using make from a tarball, and it needs packages, w, y, and z, you'll have to manually compile those as well and hope you do it right, in addition to any dependencies they need, like if package y needs package b, and package b needs package f, and so on until you want to pull your hair out just trying to install a word processor or something. Dependency tracking automates this(as long as you have an internet connection) so if you say, sudo apt install x, it will automatically say, "oh x needs w, y, and z so let me download those and install them as well as anything else this will need so you get the entire package installed with just one command. Once it's working right, It MAJORLY reduces system maintenance headaches

tcreek commented 4 years ago

I accidentally ran some of those ln and mkdir commands outside of chroot so I am going to have to rebuild the host system because it really messed it up.

Once I get back to the dkpg error, I will post the config.log on pastbin. Then I will also join the dkpg mailing list and ask on there what is going on.

PS: I did try this on a virtual machine, but I had issue trying to mount a drive when I am using an 2.5 SATA to USB adapter.

asp3n-hayley commented 4 years ago

I accidentally ran some of those ln and mkdir commands outside of chroot so I am going to have to rebuild the host system because it really messed it up.

Once I get back to the dkpg error, I will post the config.log on pastbin. Then I will also join the dkpg mailing list and ask on there what is going on.

PS: I did try this on a virtual machine, but I had issue trying to mount a drive when I am using an 2.5 SATA to USB adapter.

dang ok I've done that my fair share of times as well haha. Sound like a plan, I'll do a little digging later as well and see what I can find, and keep an eye out for your log.

Yeah, it's a pain to mount external drives on a VM sometimes, although(virtualbox is actually my preference) I've had more success on vmware, as far as that goes. What type of host system are you working on?

All the LFS systems I built in the past were made on a custom PC(budget), with a ryzen 3 2200G, 16G of RAM, a western digital m.2 SN750, and an nvidia 550ti(ancient). However, I recently was gifted an MSi Alpha 15, with a ryzen 7 3750H, 16G RAM, a 500G samsung m.2 ssd, a 1tb seagate sshd i threw in there, and the first ever laptop to carry and AMD rx5500M gpu. So I'm anxious to try a build on this as well, it may eliminate some headaches I had with the PC concerning wifi, and nvidia drivers, in addition to compilation being faster because of the higher core count and clock speed of the R7 3750H vs R3 2200G

tcreek commented 4 years ago

I am using a very low end / low power usage Celeron board. I think it only uses 7 watts power even while at max Ghz speed. I am already on the ding list for the power company for using too much power so my power costs are very high right now.

By the summer I hope to be able to purchase one of those cheap used servers on eBay with 20+ cores with 48GB of RAM, and use some sort of solar energy to run it, or put it in a co-location for doing the compiling. Right now, right now, just trying to get a system up and working, I don't need it.

asp3n-hayley commented 4 years ago

I am using a very low end / low power usage Celeron board. I think it only uses 7 watts power even while at max Ghz speed. I am already on the ding list for the power company for using too much power so my power costs are very high right now.

By the summer I hope to be able to purchase one of those cheap used servers on eBay with 20+ cores with 48GB of RAM, and use some sort of solar energy to run it, or put it in a co-location for doing the compiling. Right now, right now, just trying to get a system up and working, I don't need it.

Oh wow that's tough. Makes sense though, I suppose. I mean, I have some older laptops that I've been trying to sell, without much luck because they're not great. A dell inspiron 15 3542 with a pentium and a lenovo n22 with a celeron(windows version, not the chromebook). If you're interested, I can sell them to you on mercari for cheap.

But, like you said, right now you're just trying to get a functional system, and you may be better off with the used server thing. Just throwing it out there.

Good luck!

scottwilliambeasley commented 4 years ago

Hi guys,

It's looks like you've been busy while I was away on vacation.

It's been four years for the intended fix with an update which reflects version 9 of Linux from Scratch, and now Debian 10 which was recently released.

Also the package gnupg has been depreciated to gnupg2. How does this affect the built? This is why an update is needed

I don't disagree with this.

Like any project, without maintenance, it's bound to become outdated. It's been years since I wrote this guide and honestly I myself would need to spend time bringing myself back up to speed on it again.

You should fork this repo and update it with the correct instructions and updated packages for LFS 9.0

My idea of how this can be handled is a bit different.

I personally would prefer to keep all DFS-related efforts in a single repo, with active contributors submitting tweaks to source code so that it becomes an active 'living' project, just like any other larger open source project, instead of a ton of disparate forked repos that no one ever looks at or contributes to.

With each successive release of LFS, a respective branch can be created in this repo to reflect the proper DFS build process for said release.

Since the master branch of this repo reflects an older LFS version, the idea that I have is the following:

  1. to move the current master's code over to branch lfs-7.9 to reflect how outdated it's become
  2. to change README.MD of master to reflect the new structure of the repo (each branch targeted to a different LFS release, e.g. lfs-9.0)
  3. give willing participants the ability to push code directly to the respective branches they are working on

If you two are interested, I would be willing to mark both of you contributors to the project. The condition of course being that you finish the manual for the respective branch you're contributing to, so that what you find becomes useful to other people and not just yourself (in the same vein as the intent of this project).

The whole intent of the project originally was just to help teach other people the things I learned about this process, so that they then could take that knowledge and apply it.

Basically, in the same vein as LFS, but slightly better as the guide goes into a little bit more detail essentially telling you why you're doing something (instead of the vagueness that the original LFS is known for). I had to look hard and long at some LFS entries to understand exactly what and why something was being done, but then again that was probably the intent of the LFS maintainers.

If you don't intend to build upon that or pass what you learn along, then I suggest to proceed with forking.

I accidentally ran some of those ln and mkdir commands outside of chroot so I am going to have to rebuild the host system because it really messed it up.

Welcome to LFS :) I've done that many times as well.

Instead of doing compilation by hand, a more modern way of doing things might involve creating a CI/CD pipeline to essentially build a system and test it for you. That way, if a mistake is made then you'd know it was a manual fat-finger/typo as opposed to the exact command one might need to run.

asp3n-hayley commented 4 years ago

Hi guys,

It's looks like you've been busy while I was away on vacation.

It's been four years for the intended fix with an update which reflects version 9 of Linux from Scratch, and now Debian 10 which was recently released.

Also the package gnupg has been depreciated to gnupg2. How does this affect the built? This is why an update is needed

I don't disagree with this.

Like any project, without maintenance, it's bound to become outdated. It's been years since I wrote this guide and honestly I myself would need to spend time bringing myself back up to speed on it again.

You should fork this repo and update it with the correct instructions and updated packages for LFS 9.0

My idea of how this can be handled is a bit different.

I personally would prefer to keep all DFS-related efforts in a single repo, with active contributors submitting tweaks to source code so that it becomes an active 'living' project, just like any other larger open source project, instead of a ton of disparate forked repos that no one ever looks at or contributes to.

With each successive release of LFS, a respective branch can be created in this repo to reflect the proper DFS build process for said release.

Since the master branch of this repo reflects an older LFS version, the idea that I have is the following:

1. to move the current master's code over to branch lfs-7.9 to reflect how outdated it's become

2. to change README.MD of master to reflect the new structure of the repo (each branch targeted to a different LFS release, e.g. lfs-9.0)

3. give willing participants the ability to push code directly to the respective branches they are working on

If you two are interested, I would be willing to mark both of you contributors to the project. The condition of course being that you finish the manual for the respective branch you're contributing to, so that what you find becomes useful to other people and not just yourself (in the same vein as the intent of this project).

The whole intent of the project originally was just to help teach other people the things I learned about this process, so that they then could take that knowledge and apply it.

Basically, in the same vein as LFS, but slightly better as the guide goes into a little bit more detail essentially telling you why you're doing something (instead of the vagueness that the original LFS is known for). I had to look hard and long at some LFS entries to understand exactly what and why something was being done, but then again that was probably the intent of the LFS maintainers.

If you don't intend to build upon that or pass what you learn along, then I suggest to proceed with forking.

I accidentally ran some of those ln and mkdir commands outside of chroot so I am going to have to rebuild the host system because it really messed it up.

Welcome to LFS :) I've done that many times as well.

Instead of doing compilation by hand, a more modern way of doing things might involve creating a CI/CD pipeline to essentially build a system and test it for you. That way, if a mistake is made then you'd know it was a manual fat-finger/typo as opposed to the exact command one might need to run.

Firstly, It's great to engage with the original creator of this repo, everything else aside, I am a huge fan of this project and admire the work that you've put into it. I completely understand the difficulty of maintaining a project that relies on someone else's project, as LFS is constantly updated. As far as contributing to the project, I would certainly be interested, although my skills are very limited and I don't know how much help I'd actually be. Albeit, I would certainly be able to complete manuals for a certain branch and willingly contribute anything I could within my scope of ability.

Regrettably, finding time for projects like this is also difficult, as I am only 18, I have a part time job and am pursuing an ExcelTrack Degree in Cyber Security at Purdue University online. However, I still do have some time, especially on the weekends.

Lastly, as far as your actual idea for version control, It seems like an excellent plan, especially since the original instructions are still available for those who might be trying to maintain a system built using an older version of LFS. I also have one small idea: I have a repo on here that I use to store files that I manually made for LFS but will be used repeatedly, such as the wget-list, version-check.sh, and a script that automatically chroots the user into the temporary system. These are dull and time-consuming to create but don't contribute a whole lot to the learning factor of LFS, which is why we could include the files, edited for each version, in their respective branches. Just a thought.

tcreek commented 4 years ago

Instead of doing compilation by hand, a more modern way of doing things might involve creating a CI/CD pipeline to essentially build a system and test it for you. That way, if a mistake is made then you'd know it was a manual fat-finger/typo as opposed to the exact command one might need to run.

I am not so familiar with that. Maybe if you can explain, or point to a resource. It would be nice to auto-build all this stuff.

I can help with a new version

tcreek commented 4 years ago

Okay, I rebuilt the host, and LFS chapter 5 again.

This time I ran the following commands before trying the compiling of dpkg

touch /var/lib/dpkg/status mkdir /bin ln /tools/bin/bash /bin/bash ln /tools/bin/bash /bin/sh

This time, the configuration ran without problem, so yes there is a problem with the documentation.

However, I still ran into the same configure error as before.

config.status: error: in /sources/dpkg-1.19.7': config.status: error: Something went wrong bootstrapping makefile fragments for automatic dependency tracking. Try re-running configure with the '--disable-dependency-tracking' option to at least be able to build the package (albeit without support for automatic dependency tracking). Seeconfig.log' for more details

Here is the link for the config.log

https://pastebin.com/HsWZYXmR

tcreek commented 4 years ago

After doing some more research, I don't think Mr. @scottwilliambeasley tested this guide he built.

According to the README on dpkg, one of the requirements is pkg-config. Not many updates have been done to pkg-config, so I can probably safely assume this was a requirement for dpkg back when this guide was published.

Looking at LFS, this is not built in section 5, but in section 6.23 in version 9. Looking at LFS version 7.9, I see it was also included, but using version 0.29. The current version is 0.29.2 from 2017.

I will look at how they build pkg-config, and try to incorporate it into the /tools folder, then see what happens.

scottwilliambeasley commented 4 years ago

Instead of doing compilation by hand, a more modern way of doing things might involve creating a CI/CD pipeline to essentially build a system and test it for you. That way, if a mistake is made then you'd know it was a manual fat-finger/typo as opposed to the exact command one might need to run.

I am not so familiar with that. Maybe if you can explain, or point to a resource. It would be nice to auto-build all this stuff.

I can help with a new version

CI/CD (continuous integration/continuous delivery) is a relatively modern concept. Nowadays in large development projects it's pretty rare to find something that doesn't use it to some extent.

The idea behind it is that your source control system (in this case, Github) notifies an external system (typically Jenkins) whenever new code is pushed into a branch.

This external system pulls the relevant code upon notification, and then scrutinizes the quality of the code in an automated fashion according to certain steps and constraints you set up.

It creates a build (in our case building the toolchains, chrooting, executing various commands etc), and then runs tests against said build to see if the result is acceptable.

Essentially, this is automation doing the work of developers and maintainers.

https://www.mabl.com/blog/what-is-cicd

scottwilliambeasley commented 4 years ago

After doing some more research, I don't think Mr. @scottwilliambeasley tested this guide he built.

According to the README on dpkg, one of the requirements is pkg-config. Not many updates have been done to pkg-config, so I can probably safely assume this was a requirement for dpkg back when this guide was published.

Looking at LFS, this is not built in section 5, but in section 6.23 in version 9. Looking at LFS version 7.9, I see it was also included, but using version 0.29. The current version is 0.29.2 from 2017.

I will look at how they build pkg-config, and try to incorporate it into the /tools folder, then see what happens.

I must confess that I may have had one too many coffees when originally writing this guide some 4 years ago.

I could have sworn that I had gotten everything right, but then again I am only human.

With this, I'd like to respectfully iterate over a few points.

  1. Bugs exist in any kind of project. Even ones with the best intentions.
  2. Most developers do not receive financial compensation for continual maintenance and development of projects like this. Motivation for passion projects such as these waxes and wanes. There is typically no existing contract stating that continual maintenance should be kept as time passes by.
  3. Given points 1 & 2, and in the spirit of the greater open source community, the best method towards quick resolution of discovered bugs is a.) do some research on it and then to b.) submit a pull request proposing a fix for said bug for inclusion into the project.

It seems that you believe you've already identified a flaw, done research etc.

Why not submit a pull request?

asp3n-hayley commented 4 years ago

Okay, I rebuilt the host, and LFS chapter 5 again.

This time I ran the following commands before trying the compiling of dpkg

touch /var/lib/dpkg/status mkdir /bin ln /tools/bin/bash /bin/bash ln /tools/bin/bash /bin/sh

This time, the configuration ran without problem, so yes there is a problem with the documentation.

However, I still ran into the same configure error as before.

config.status: error: in /sources/dpkg-1.19.7': config.status: error: Something went wrong bootstrapping makefile fragments for automatic dependency tracking. Try re-running configure with the '--disable-dependency-tracking' option to at least be able to build the package (albeit without support for automatic dependency tracking). Seeconfig.log' for more details

Here is the link for the config.log

https://pastebin.com/HsWZYXmR

ok, it seems like there is multiple errors in that log and I don't have enough knowledge to understand them all, however: on a hunch, can you run this script on your host system and verify that every tool returns a version: https://github.com/ironmatt613/LFS/blob/master/version-check.sh ? It's just the version-check.sh from LFS that verifies everything you need is installed on the host system. Oh wait...DFS is compiled in chroot right? maybe run the script in chroot and install the missing tools from source in $LFS/tools....or not. Sorry I'm typing as I'm thinking and again, I'm not super knowledgeable in this subject.

After doing some more research, I don't think Mr. @scottwilliambeasley tested this guide he built. According to the README on dpkg, one of the requirements is pkg-config. Not many updates have been done to pkg-config, so I can probably safely assume this was a requirement for dpkg back when this guide was published. Looking at LFS, this is not built in section 5, but in section 6.23 in version 9. Looking at LFS version 7.9, I see it was also included, but using version 0.29. The current version is 0.29.2 from 2017. I will look at how they build pkg-config, and try to incorporate it into the /tools folder, then see what happens.

I must confess that I may have had one too many coffees when originally writing this guide some 4 years ago.

I could have sworn that I had gotten everything right, but then again I am only human.

With this, I'd like to respectfully iterate over a few points.

1. Bugs exist in any kind of project. Even ones with the best intentions.

2. Most developers do not receive financial compensation for continual maintenance and development of projects like this. Motivation for passion projects such as these waxes and wanes. There is typically no existing contract stating that continual maintenance should be kept as time passes by.

3. Given points 1 & 2, and in the spirit of the greater open source community, the best method towards quick resolution of discovered bugs is a.) do some research on it and then to b.) submit a pull request proposing a fix for said bug for inclusion into the project.

It seems that you believe you've already identified a flaw, done research etc.

Why not submit a pull request?

I'm not really a part of this thread specifically but I agree with Mr. Beasley. I've tried to maintain several projects in the past and inevitably they all get abandoned, because there is not much motivation to keep them going aside from contributing to the community and school/work take precedence. If I had the knowledge to go through and fix discrepancies in a project like this, I would certainly submit a pull request, but regrettably I am somewhat lacking. However I'm scheduled to take a few advanced linux classes next term in addition to whatever I teach myself so I may re-evaluate in a few months.

Instead of doing compilation by hand, a more modern way of doing things might involve creating a CI/CD pipeline to essentially build a system and test it for you. That way, if a mistake is made then you'd know it was a manual fat-finger/typo as opposed to the exact command one might need to run.

I am not so familiar with that. Maybe if you can explain, or point to a resource. It would be nice to auto-build all this stuff. I can help with a new version

CI/CD (continuous integration/continuous delivery) is a relatively modern concept. Nowadays in large development projects it's pretty rare to find something that doesn't use it to some extent.

The idea behind it is that your source control system (in this case, Github) notifies an external system (typically Jenkins) whenever new code is pushed into a branch.

This external system pulls the relevant code upon notification, and then scrutinizes the quality of the code in an automated fashion according to certain steps and constraints you set up.

It creates a build (in our case building the toolchains, chrooting, executing various commands etc), and then runs tests against said build to see if the result is acceptable.

Essentially, this is automation doing the work of developers and maintainers.

https://www.mabl.com/blog/what-is-cicd

I've heard of this but never looked into it too much. I suppose it might be worth learning because it could significantly reduce errors, build time, and frankly, it sounds awesome!

tcreek commented 4 years ago

With this, I'd like to respectfully iterate over a few points.

Bugs exist in any kind of project. Even ones with the best intentions.
Most developers do not receive financial compensation for continual maintenance and development of projects like this. Motivation for passion projects such as these waxes and wanes. There is typically no existing contract stating that continual maintenance should be kept as time passes by.
Given points 1 & 2, and in the spirit of the greater open source community, the best method towards quick resolution of discovered bugs is a.) do some research on it and then to b.) submit a pull request proposing a fix for said bug for inclusion into the project.

I can understand bugs, but after digging a bit deeper, this goes beyond bugs. The guide is mostly incorrect. I would think that if one is going to to take the time to put something together, and put their name on it, they would at least make sure it is correct, and mostly functioning as best as possible. This is good information, if it was correct. I am surprised nobody has brought the problem with this up before now.

I cannot even compile pkg-config under the chroot environment you have built for this project.

I do appreciate the effort, but my thinking is if you are going to do something, do it right.

I will go through and try to correct the incorrect parts as best as I can. I think issues are going to have to be put in another thread as the more I dig, the more issues I find.

ok, it seems like there is multiple errors in that log and I don't have enough knowledge to understand them all, however: on a hunch, can you run this script on your host system and verify that every tool returns a version: https://github.com/ironmatt613/LFS/blob/master/version-check.sh ? It's just the version-check.sh from LFS that verifies everything you need is installed on the host system. Oh wait...DFS is compiled in chroot right? maybe run the script in chroot and install the missing tools from source in $LFS/tools....or not. Sorry I'm typing as I'm thinking and again, I'm not super knowledgeable in this subject.

I already mentioned the issue of the problem. It is missing a requirement of pkg-config, which is part of Chapter 6.23 in LFS, but that will not build either due to environment issues not configured in this guide (missing). It is not because this guide is outdated, but incorrect.

CI/CD (continuous integration/continuous delivery) is a relatively modern concept. Nowadays in large development projects it's pretty rare to find something that doesn't use it to some extent.

The idea behind it is that your source control system (in this case, Github) notifies an external system (typically Jenkins) whenever new code is pushed into a branch.

This external system pulls the relevant code upon notification, and then scrutinizes the quality of the code in an automated fashion according to certain steps and constraints you set up.

It creates a build (in our case building the toolchains, chrooting, executing various commands etc), and then runs tests against said build to see if the result is acceptable.

Essentially, this is automation doing the work of developers and maintainers.

https://www.mabl.com/blog/what-is-cicd

Thanks for this great information. I was not aware it existed. And People have been fumbling around with the Automatic LFS script. I will give it good read upon when I have a chance.

Sorry if I sound harsh, but people coming to any project would expect it to function as stated. You presented it as "this is how it is done," it is incorrect.

tcreek commented 4 years ago

Just a little update. After some tinkering, the first .deb package was successfully installed

cryptoatropine commented 3 years ago

I am going to be doing this, and i am wounding: " should I takes notes on anything that could be useful ? "

I Will however "hopefully" be using Debian Testing [SID]. Side question maybe some one knows [ i did not see one my self ] I want to do this again for mobian dose such a guild exist ? should i write one while i do it ?

tcreek commented 3 years ago

Well, don't waste your time. It is now broken beyond Debian 8. I tried and tried, but could never get it working based on this guide with Debian 10. I just going in circles with dependencies.

cryptoatropine commented 3 years ago

Well, I'm not one to give up that easily, But thanks for the heads up, i got time to muss about .. if i do figure something out I'll post about it.

tcreek commented 3 years ago

It's not a matter of "giving up." but a matter of reality. Unless you have deep knowledge of building Debian, it is going to fail. Look at all the posts i made on here about many issues, and the not even the owner of this project responds anymore.

cryptoatropine commented 3 years ago

well in any-case i'm in the middle moving now ... so this is on hold for a month or two while i settle in ... just as a note incase it's helpful to you to get this done. have you tried doing it with the Debian patches from the very start i mean at LFS section 1?

tcreek commented 3 years ago

I have completed LFS a few times over. The patches are in other sections