Picking up Build System

[ari-kendall]

Hi

I saw recently about BE packages being dropped and wanted to pick that up. I am putting together some machines to do so, but I was wondering how you set up your box to just build endlessly.

I want to have 32 and 64 both build, and have it be able to scan for new packages. If this is the wrong place to ask, I will just stumble through it till I figure it out.

[q66]

i don't really have anything special around the build system, mostly just a collection of shell and python scripts; you'll want to set up your own

what kind of machines are you putting together?

[github-actions[bot]]

Issues become stale 90 days after last activity and are closed 14 days after that. If this issue is still relevant bump it or assign it.

[crynux]

@q66 I'd be interested in helping out with / maintaining the big-endian packages. I have access to a few G5 macs ... and can probably virtualize a few things if/where needed. How/where do I start?

Thanks in advance!

[q66]

current builds are done on an 18-core power9 and it's still just barely enough to be workable, so don't expect G5s with their fraction of that performance to be useful for builds at all, no matter how many of them you have

[crynux]

Yeah I get that entirely; I was thinking more of using them for testing, and virtualizing the builds themselves.

[ari-kendall]

Well its what I have shrug

Sent from ProtonMail mobile

-------- Original Message -------- On Jun 14, 2022, 3:44 AM, q66 wrote:

current builds are done on an 18-core power9 and it's still just barely enough to be workable, so don't expect G5s with their fraction of that performance to be useful for builds at all, no matter how many of them you have

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[q66]

Yeah I get that entirely; I was thinking more of using them for testing, and virtualizing the builds themselves.

i don't see how you expect to virtualize that; an emulated machine is going to be even much slower

[crynux]

Well ... then I can just cross compile it then; and test virtually. Whatever works.

[q66]

the project will not support any cross-compiled efforts (a lot of the packages are not available then + results are frequently janky and broken + unit tests cannot be run), one of the points of void-ppc has been that it's all native

you can obviously publish your own repos, but you'll be on your own with that

[crynux]

Yeah seems like it might be easier to just to it separately anyways.

[ari-kendall]

i don't really have anything special around the build system, mostly just a collection of shell and python scripts; you'll want to set up your own

what kind of machines are you putting together?

Sorry for getting back late. As of right now I only have PowerPC Macs and some intel machines that I can use for cross compile (though I really don't want that). I will be getting a raptor system soon.

I understand that POWER4 is old, however, I don't care. I am very good at optimizing old systems and I am willing to give it a shot if it means I will have packages for my laptop. As well, its a super famous distro on facebook and youtube, it would really be a shame for it to die like that.

Could you show me your shell scripts?

My end goal is to make a file that anyone can flash over firewire, turn the machine on, plug it into the net with an SSD running the system, and then just walk away. THough first I have to make it work. Then, its not just me who is building packages, but literally anyone who wants to help. It would be a distributed compute platform for VPPC based mostly on G5's and PA6

[HadetTheUndying]

While I don't have the time or capacity to offer support. I will say that it is entirely possible to maintain and build packages and updates you need locally on your x86 machines. If you were looking to take over and host a build system and binary repos you can occasionally find POWER/8 systems for reasonable prices but they are not very efficient. FreeBSD is also a viable option though Big Endian is considered tier 2 support.

[ari-kendall]

Um, no, this is the system I want. And no unfortunately because of the nature of the OS, I will not be crosscompiling if I can help it. I might do like make packages and stuff just so I can have stuff set up faster, but I need the right processor for the job.

Also P8 is still heavily used in my arera, I will never find one. Not even P7. Efficiency is like the last thing I care about.

[HadetTheUndying]

Um, no, this is the system I want. And no unfortunately because of the nature of the OS, I will not be crosscompiling if I can help it.

One of xbps/xbps-src's best features for me is the ease of crossbuild. If you're dead set on a native build environment your solutions are pretty pricey. I'm not going to spam up the issue though. Just making a few suggestions. Good luck

[ari-kendall]

Um, no, this is the system I want. And no unfortunately because of the nature of the OS, I will not be crosscompiling if I can help it.

One of xbps/xbps-src's best features for me is the ease of crossbuild. If you're dead set on a native build environment your solutions are pretty pricey. I'm not going to spam up the issue though. Just making a few suggestions. Good luck

Well no actually you just gave me a reason to expand on my idea more.

As I said I want to make a version of Void that is completely dumpable through firewire. This will start out as a thing for the more technically minded, but if I could make an OSX app to copy UUID's and set up partitioning and maaaaybe OFW tweaks (if thats even possible through firewire? THough I guess it could be done through ethernet...?), and just save the end user the hassle of needing to think about it, the build server, as it were, will not be a server. It will be a network of machines actively connecting, checking what is built, looking for an unclaimed package, and starting to build it. I want it to be able to be run on Desktops or Laptops, and I want it to just be turn on, boot, get a TTY to select a mirror you want to contribute to or add your own, and another TTY for boot logs and system logs, though IDK what apps are out there as a general log viewer for TTY, I could need something completely new and not know it yet. But I don't want the log viewer to run all the time, the entire point of the system is to boot, and walk away from it. You should have to select a specific TTY and have to enter a password to start the viewer, and then when you change back to the other TTY or swap out it needs to ask you if you want to quit out of the viewer entirely. I DO NOT want the pipeline filled up with random crap or for the icache to get tagged by way too much stuff and be made unuseable.

If done right, anyone can set up a G4, G5, amigaone, Pa6 board, and just plug in a USB and leave. The machine will do the rest. Defaults are for the main repo, and there will need to be a file somewhere for laying claim to specific builds.

The complete purpose of this is to give this platform a framework to autonomously support itself, if one would so desire. I don't know if you guys have been paying attention or not but until a few years ago PPC was completely dead in the water with no development happening basically at all publicly. And ATP theres actually been some really cool stuff put out within even the last year for Linux and OSX that make the platform its own unique system.

WHile yes, newer machines are fucking ridiculously expensive and I really DON'T want to buy one, being able to have someone claim a package, and to some extent, just leave it building, time won't matter, As long as its designed correctly, this could theoretically work, and enull the fact that Raptors are stupidly priced.

As well, have you actually seen the port selection on a powerbook G4? Its surprisingly modern and useful, at least to me. With that, being able to completely rescue a system over TD mode and reconfigure stuff in OFW to be more efficient is stupidly handy. One of my projects is an alternate OFW with a menu instead of a terminal. But I have to still figure out the correct development environment to be able to work on that as I am not the most handy when it comes to that. No training or anything here, I have literally no clue what I am doing, just going at it and seeing what I could learn along the way. Hey it worked for me so far right? Trust me lol

What do you think? Then after setting this up QBoy wqill never need to build BE packages again, AND, the packlages will ACTUALLY BE BUILT ON THE RIGHT PROCESSOR (I have issues with cross compiling between LE and BE, but I'm autistic ignore me lol). I tpersonally really want this to happen, and I am going to start researching the whole topic more and more in thge coming months.

[ari-kendall]

This is not only making a software system for an old computer platform that refuses to be dead, this is getting rid of more and more ewaste. Think about it. This is also enabling a lot of right to repair stuff, and thats the most exciting thing to me. I am actually going on a road trip USA statewide to research what all is available and what can and could be done, who to talk to, etc. I have an RV to live in while doing this, this is my go juice man. I will also be making youtube channels around this whole experience and documenting everything.

To an effect I have been planning this for a number of years as more and more things have happened in the world, in my life, and in my hobbies. This is my Joe Grand level of lifestyle, basically. THough I am just starting, I hope to get something cool out of it. PPC machines is just a tiny part of that though, so don't think I am just riding on that alone lol.

[q66]

even if you managed to implement this (i think you might be biting off more than you can chew), since void packages are not reproducible, you have no way to ensure the third party builders can be trusted and aren't feeding you malicious stuff

and also if you want to implement something like that you have much bigger things to worry about than me sharing a bunch of trivial scripts that won't probably even be super useful to you

lastly, you talk about e-waste but don't address that your distributed cluster of builders will still be slower overall and will use significantly more power than a single raptor machine, which does not need all this complexity

[HadetTheUndying]

lastly, you talk about e-waste but don't address that your distributed cluster of builders will still be slower overall and will use significantly more power than a single raptor machine, which does not need all this complexity

Yeah running a G5 Quad Tower 24/7 is a nightmare let alone like 50 of them to pick up what a Single POWER9 system would do.

If the worry is electronic waste you certainly don't wanna be running a ton of sorry inefficient old systems.

The kind of cost you're looking at here you'd be better off actually picking up the 2U Talos II that actually is in stock right now if you want to pick up build support for Big Endian it should work. I understand the sticker shock though. Which is why I recommended just doing crossbuild locally for yourself.

[ari-kendall]

even if you managed to implement this (i think you might be biting off more than you can chew), since void packages are not reproducible, you have no way to ensure the third party builders can be trusted and aren't feeding you malicious stuff

and also if you want to implement something like that you have much bigger things to worry about than me sharing a bunch of trivial scripts that won't probably even be super useful to you

lastly, you talk about e-waste but don't address that your distributed cluster of builders will still be slower overall and will use significantly more power than a single raptor machine, which does not need all this complexity

IDRC what is in the scripts, I just don't know how this works yet and I am still learning.

And yeah, this is not a solo project liek at all. However if BE is going to be a platform, and I think theres an available market, it needs something liek this implemented. IDK what could be done for security though. Thered need to be a checksumming system of some sort, or an internal testing center and bugged / shit packages get trtashed. That COULD actually be done in a VM on X86 and I wouldn't care about that.

Yes this would be a complex implementation, but honestly linux needs more systems like this in place to even attract normal devs.

Also yeah, but its on a per user basis. The power load is literally spread out over multiple states, its not in my shed. Did you read the post or was I not clear enough? Yes I understand the system does not need the complexity that I have lain out, however my design is intended to be autonomous. This is so the basic user at home who likes these machines can contribute to their software platform folding at home style. The more end users that contribute, the more packages are built and the more software becomes available. This also doesn't have to be just for BE, this could be how it just works.

Maybe I should just branch void out and do that in another project then, seems like everyone just laughs at the good ideas nowadays or boos them down because they didn't think of that. My target is lowest common denominator, however, and honestly thats the user that gets screweed over wanting to use the cool laptop they inherited from their grampa and his business but now he has learned its ewaste. That would bum me the hell out. I want these machines to be as much an option as anything else, and people like CRD and others have shown time and time again, humans are literally just throwing good hardware into the oceans.

I intend to start work on this either way, cooperation from fellow devs would be nice though. This is good for end users, either way.

Need I remind you as well, I was one of the users pushing for on-system builds when this project was in its infancy, and I did research for Stenstorp. This is a new design I want to contribute. I don't have any credit anywhere, so I intend to make a mark. This OS wouldn't be an option without that, so to make it a usable system via distributed computing with the same end goal, is that not better than nothing at all? Like seriously, not everyone wants to pay a billion dollars for a stupid talos board. I can pick up a G5 at the dump, thats more economic.

[ari-kendall]

lastly, you talk about e-waste but don't address that your distributed cluster of builders will still be slower overall and will use significantly more power than a single raptor machine, which does not need all this complexity

Yeah running a G5 Quad Tower 24/7 is a nightmare let alone like 50 of them to pick up what a Single POWER9 system would do.

If the worry is electronic waste you certainly don't wanna be running a ton of sorry inefficient old systems.

The kind of cost you're looking at here you'd be better off actually picking up the 2U Talos II that actually is in stock right now if you want to pick up build support for Big Endian it should work. I understand the sticker shock though. Which is why I recommended just doing crossbuild locally for yourself.

Yeah except I am poor and I am not running these machines myself. THis is so I can drop an image on the PPC facebook group and IRC and bam now people have packages. With the ability to contribute, making more packages available. Distributed computing is a thing.

[q66]

again, you have no way to do any "checksumming" or another way to ensure packages are not malicious because void packages are not reproducible (i.e. not guaranteed to come out the same between build runs)

there are two ways to ensure a package is good:

1) making packages reproducible so that they are the same between different build runs (can't have that at this point) 2) or ensuring the computers doing the building can't have their systems altered (you'd need some kind of secure boot mechanism for that, which old ppc machines do not have, in addition to signed kernel and locked down userland)

realistically you can't have either of them

additionally, even if you implemented this securely somehow (which you can't), you'd need a massive cluster to get even close to a single fast machine (which does not need any of this)

as far as environmental footprint goes, it would be pretty massive (it does not matter between how many states it's distributed, the emissions do not go anywhere)

additionally, such system would not be autonomous, because you still need somebody to work on patches and fixing breakages, and I don't think you appreciate how much of that actually happens and how much i fix up and how much i don't end up fixing because I don't have the time or energy or interest and so far I haven't seen anybody else actually attempt to contribute such fixes (e.g. i pointed out that most of qt6 is busted and does not even build months ago - the response has been nothing)

if you think you can somehow solve all that, and also bring in developers who are willing to actually maintain the platform in a way that is satisfactory, go right ahead

but remember, code talks, bullshit walks

[ari-kendall]

I don't know if you are aware of this but thertes ppl on like facebook and other places that want to help, but they literally don't know that help is needed. I brought up BE packages dying and ppl freaked out, hadn't even seen the site post. Also you kind of shit on ppl religiously, that doesn't inspire confidence in wanting to fix stuff like QT6. I will be looking at that, btw.

You have to actually give a shit about talking to people or you are the only one working on your project. Like cool that you have a talos and all, no one else gives a shit tho.

[HadetTheUndying]

I don't know if you are aware of this but thertes ppl on like facebook and other places that want to help, but they literally don't know that help is needed. I brought up BE packages dying and ppl freaked out, hadn't even seen the site post. Also you kind of shit on ppl religiously, that doesn't inspire confidence in wanting to fix stuff like QT6. I will be looking at that, btw.

You have to actually give a shit about talking to people or you are the only one working on your project. Like cool that you have a talos and all, no one else gives a shit tho.

Nothing is stopping you from forking the repo, and working on your own tooling with those people since they don't seem interested to be interested in the current outstanding Big Endian issues in this repo. There is no reason all of a sudden get hostile because people are explaining to you the issues that they do not have the time to actively correct, and security implications + issues with what you're currently proposing. The project is here for people to come and communicate and contribute to so long as the contributions meet the quality requirements. One person has been maintaining the majority of this project with very little help other than upstream Void contributors and maintainers for years. A person I personally know to be very approachable when I've needed support, or had some silly idea, so there is no need to be disrespectful because they are being realistic about the situation and their own free time.

[q66]

I don't know if you are aware of this but thertes ppl on like facebook and other places that want to help, but they literally don't know that help is needed. I brought up BE packages dying and ppl freaked out, hadn't even seen the site post. Also you kind of shit on ppl religiously, that doesn't inspire confidence in wanting to fix stuff like QT6. I will be looking at that, btw.

you contradict yourself

if they wanted to contribute, they would've reached out over the years, the channels are right there (irc, matrix, github); if they are not aware of them, then they don't care enough to get out of their bubble, every FOSS project, especially one as large as a distro on a niche platform, needs help and effort by default

I'm not doing this for myself, so it's not like anybody does me a favor by helping out, or that I should actively seek people out in random places - if they want the project to continue existing, they need to do something, or it'll go away, as simple as that

you also talk about me "not inspiring confidence" - i thought you said nobody actually knew that help was needed?

You have to actually give a shit about talking to people or you are the only one working on your project. Like cool that you have a talos and all, no one else gives a shit tho.

look, i don't even have facebook, and again having to actually seek out randos to help on a project whose existence is primarily in their interest and not mine is dumb

[ari-kendall]

Again, I am learning. And maybe its just distributed designs NEED to stay with distributed designs, and monolith designs stay with monolith designs. I literally don't know. And yeah a check summing and testing system would need to be implemented. Yes I know I won't be able to just build it and not ever touch it, I want that for the END USER not for us.

Yeah I get your point, but like any time I have tried to talk to you you have always been really abbrasive. IRC, hell I was the reason you ended up on L1, me and sten were trying to get everything wrangled together.

However I'm gunna tell you, I was trying to learn how XBPS even worked at that time, and it was kind of a pain in the butt. I know how to use it now obviously, but back then I could see the system itself is light and easy to balance, and when I found interest I pushed harder. Thats why this project means anything in the first place.

People aren't nerds if they own PPC. THey have memories or they have reasons otherwise for wanting the machine. Not everyone is gunna sit and stare at documentation until 2 in the morning just because the devs who made the image in the first place stopped making packages for their system and the only other option that works halfassed is a G5 OS or old ubuntu.

I mean really dude come on

If this is how the FOSS community is gunna act I'll just figure it out myself.

[q66]

lol? you seem to be mistaken about some things here

1) this project was never started with old ppc stuff in mind; initially (around ~2018), the only supported architecture was ppc64le, and it was so that i could have the distro working on my own computer, and other owners of raptor/IBM machines (which continue to be the primary target, and old stuff is merely piggybacking) 2) big endian profiles and packages were only added later, because it wasn't a big deal to have them; from the start i have made it clear that this is entirely reliant on community involvement and that it would not continue to exist in the long term otherwise; in the end, there was no community involvement, and it has started becoming too much work, so i deprecated it 3) stenstorp did do some enablement work in the upstream void-packages repo for a while, but it was relatively minor and incomplete (and largely based on earlier, older work by foxlet); all the packages have been built by me from the start though (unless you count foxlet's old repository, which only contained a relatively small amount of packages and was never updated afterwards); he disappeared a couple weeks after this work was merged, leaving me alone to do any further maintenance and development 4) the image creation infra was ported by me from the start, stenstorp generated and published some images with it before i was done with it, and published them on L1 (that is what brought me in there, as i randomly came across them)