search

7Ji-PKGBUILDs / .meta

1 stars 0 forks source link

Naming, repo merge, PKGBUILD storage, and more #19

Closed 7Ji closed 4 months ago

7Ji commented 5 months ago

I'm opening this issue on the public list to accept public ideas, opinions from non-members are welcomed.

Naming

As this org and 7Ji/archrepo is picking popularity, I'm feeling it inappropriate to name them with such a personal name. I'm considering to rename both the repo ( 7Ji/archrepo ) and the PKGBUILDs org ( 7Ji-PKGBUILDs ) to drop the 7Ji factor. The new name could be a stereotype name reflecting our theme: kernel + media, especially for the aarch64 platform, but note I also maintain packages for x86 and build them for x86 (e.g. kodi-git, ffmpeg-full); it could also be some catchy name without actual meaning.

My proposal would be "mediArch" (media + Arch), what's your opinion on this and what would your proposal be?

Logo

If we would change the name I also would like the logo to be swapped. I don't want to use Arch logo directly because I want it clear that we're nonofficial. What's your opinion on this?

Repo merge

After the rename, I'd like to migrate the package release (7JI/archrepo) to this org, as it's also not really my personal project any more. But what would its name be? .repo would be simple and that sticks to our current special repo naming scheme (.report, .room, etc), but if we're going to change our PKGBUILD storage scheme them (see below) then we could also use a name without that dot to mark a hidden namespace.

But merging the repo AFTER the naming change would mean both the repo name and URL would need to be updated in users' config, so that would be hard to achieve and cause a lot of trouble to existing users.

PKGBUILD storage

Quote hbiyik from https://github.com/7Ji-PKGBUILDs/.report/issues/17#issuecomment-1914750671 :

From my POV i think only 1 git repo for my packages is enough i dont think i need a seperate repo for each also, it is easier to maintain for me as well. So yeah, lets burn this repo and let the users cook their own food.

At first I started this org to host my PKBGUILDs in an alternative place alongside AUR, so I want them to be inter-changeable. After some things we know some of the PKGBUILDs are migrated to here as their sole upstream and are not synced to AUR any more. But I still want to keep some hope. Staying this way also means we would have Arch's guideline to follow (not enforced, but at least we have something to look up to when creating new packages) with only a few points adapted to our use case.

The current scheme also means it would be very easy to clone and build a single package as long as you know the name. If we would migrate them to a single repo, then picking a single PKGBUILD would not be friendly for third-party builder (I always want our PKGBUILDs to be friendly to any one, meaning any one could pick one and build by simple makepkg, not relying on my builder, unlike some of the nonofficial repos out there). And most importantly that would mean we would have more merge conflicts to go resolve which is what I don't like.

My personal opinion would be to stick to the current storage scheme. What's your opinion?

FYI, if you have github-cli, you can create a repo easily on your CLI:

set pkg xray-geoip-placeholder
gh repo create 7Ji-PKGBUILDs/$pkg --public
git init $pkg
cd $pkg
git remote add origin git@github.com:7Ji-PKGBUILDs/$pkg.git

-rockchip, we're all in?

Although nyanmisaka and hbiyik's effort of hardware decoding are focusing on ffmpeg-rockchip now, I don't want to pull the plug of life support from these old -mpp packages. Going full in and dropping support of the old packages would be very damaging to the user base and would most likely cause a split. If you're dropping the old packages, please at least don't delete them, their existing history would be educational, archiving them is enough.

What's your opinions? Should we blow up old -mpp packages and go all in with -rockchip packages?

New package, and their naming

Since we're going from a new start for the new direction of hardware decoding, we have an opportunity to re-do everything right, can we have the new packages created and old packages adapted in an easier to maintain way? E.g.

What would you think about this?

Infrastructure

Some of you might not know, I currently build the aarch64 packages for the repo on my Orangepi 5 Plus, with ccache and distcc enabled, with 7 other SBCs/Set-top boxes and the x86_64 builder as its distcc volunteers, and x86_64 packages on my old 1600x desktop. The time is acceptable but the network not friendly to the builders as I live in China and situations got worse in the past half months. I'm planning to rent an ARM VPS or getting Oracle's free ARM instance for free. But neither is affordable for me now: I don't have income currently (never had, for personal reasons, but am currently looking for a job) to afford such a VPS on addition to my existing x86 servers out there and don't have a VISA/Mastercard to pass Oracle's verification.

I do all of my projects without sponsorship for various reasons, and I would keep them this way so I won't accept sponsorship to pay up for the build server. So until my network is not jammed and my connection is not only working half of the time, building and uploading packages would be of problem (the network situation got like this since ~half a month ago).

Do we need dedicated infrastructure?

Pinging for notification @hbiyik @wyf9661

kyak commented 5 months ago

Maybe we should just create a new package kodi-mpp-git and focus the efforts there?

That would be great!

My pain currently is the following. I switched to binary packages from 7Ji's repo apart from kodi package. I build kodi-ext-git instead roughly once a week to follow upstream (I find it much more workable than Nexus branch). I also point kodi via LD_LIBRARY_PATH to a specially built ffmpeg-mpp from exp_refactor_all branch (which I have to rebuild from time to time). This is very painful. I have to do this in order to have working SD videos instead of a green screen.

I respect what packages/repo maintainers and developers do. Without you, it would've been so much harder for humble users. If everyone uses the same kodi and ffmpeg packages, that would benefit both users, packagers and developers.

My goal is to be able to do pacman -Syu and be done with upgraded system without needing any additional steps.

hbiyik commented 5 months ago

Logo change:

i dont have any preference, i didnt even notice that there was a logo :)

PKGBUILD storage

i agree, for such conventional repos it makes sense, but my comment was in the direction small projects, where things change rapidly and has lots of implications on other dependencies starting from kernel to user space apps like kodi. Now thats a burden. to maintainer.

After working 1 week for such a tool, i came up with agr, it works exactly like AUR but git in the backend, this way distributed projects can focus on their own changes, and users now have to compile those things. Therefore the effort is balanced in between devs, maintainers, and users. And this kind of provides a form of devops link in between users and developers, where users can get latest rapidly, and devs can get the feedback immediately. This is very important for small projects to launch.

KODI versions

Kodi has an API that changes on each release time to time. I am also developing kodi addons, and i need multiple versions of kodi at the same time to test, everytime when kodi switches a version soemthing breaks, it is just the norm.

But as the time goes active version gets matured, and the older version is no more used. When i first created those packages nexus was very new and i needed both nexus and matrix, but currently nexus is enough. But when omega is released i will need both again, so having 2 latest versions is needed from my end, nobody else i guess.

7Ji commented 5 months ago

After working 1 week for such a tool, i came up with agr, it works exactly like AUR but git in the backend, this way distributed projects can focus on their own changes, and users now have to compile those things.

That's a nice project!

But... isn't that a feature already supported by paru since v2.0.0 ?

With the following section in .config/paru/paru.conf:

[hbiyik]
Url = https://github.com/hbiyik/agrrepo.git

And with the pkgbuilds cached (you can also use a local path instead of a git url, so this step could be skipped)

paru -Sy --pkgbuilds

You and users can search and install them just like other AUR packages:

> paru -Ss mesa-panfork
hbiyik/mesa-panfork-git r164484.120202c6757-2
    Libgl & gbm with panfrost for Mali G610
aur/mesa-panfork-git r163343.3504c9b1b31-3 [+1 ~0.00]
    Mesa with Panfrost that supports Mali G610/G710 GPUs (Valhall v10 CSF)
aur/opencl-mesa-panfork-git r163343.3504c9b1b31-3 [+1 ~0.00]
    OpenCL support for AMD/ATI Radeon mesa drivers - Panfork
hbiyik commented 5 months ago

if paru had really such a feature i would just jump out of the window :). let me try this with paru.

hbiyik commented 5 months ago

Paru almost works, but it can not handle the dynamic pkgver handling, it is creating the SRCINFO without downloading the sources, and using $pkgver static value, without executing pkgver() even with the --needed flag. This might even be a bug in paru side now im puzzled if i should PR paru to fix that part or just use agr.

Other than that, if that flow works, would there still be a need for a binary repo?

EDIT: --needed works only when the specific package name with its develsuffix is explicitly given

to get a list of packages: works: paru -Ss --pkgbuilds it is not discovering nested directory structures, but thats fine.

to update single package with dynamic pkgver works: paru -S dri2to3-git --pkgbuilds --needed --develsuffixes git

does not work: paru -S dri2to3-git --pkgbuilds --needed

to update the system with dynamic pkgver does not work and this is the biggest issue from my pov: does not work: paru -Syu --pkgbuilds --needed --develsuffixes git,

so one has to to manual install of each package with --needed flag with the right dependency order to update the system, which is not nice.

one additonal benefir of paru, is that it can handle the aur package as a dependency since it is an aur manager, agr packages can only depend pacman repos. but i think thats a little nuance.

EDIT: I am still working on the packages, i think i have a good plan to satisfy everyone. I will update soon

hbiyik commented 5 months ago

@7Ji We can set the kodi mirrors with -DKODI_MIRROR flag in cmake, and i see that chinese mirrors are excluded from kodi mirror list. so it wont redirect auto when reqested from default mirrors.kodi.tv

https://mirrors.kodi.tv/build-deps/sources/taglib-1.13.tar.gz?mirrorlist

There is however below server which should work for you, i think it is in taiwan, or mainland china dont know. China https://mirrors.tuna.tsinghua.edu.cn/kodi/build-deps/sources/

will that help?

I have also reduced the external dependencies to below list, i think it is fairly less.

./spdlog-1.10.0.tar.gz
./libdvdnav-6.1.1-Next-Nexus-Alpha2-2.tar.gz
./libudfread-1.1.2.tar.gz
./libdvdread-6.1.3-Next-Nexus-Alpha2-2.tar.gz
./libdvdcss-1.4.3-Next-Nexus-Alpha2-2.tar.gz
./fstrcmp-0.7.D001.tar.gz
./crossguid-ca1bf4b810e2d188d04cb6286f957008ee1b7681.tar.gz
./src/kodi-build/build/download/taglib-1.12.tar.gz
./src/kodi-build/build/download/rapidjson-1.1.0.tar.gz
kwankiu commented 5 months ago

@7Ji will you build linux-rkbsp-joshua-git for archrepo? I noticed that it was removed yesterday with the introduction of linux-rkbsp, but now linux-rkbsp-joshua-git is updated to rk-6.1-rkr1 with new wifi support and patches

https://github.com/7Ji-PKGBUILDs/linux-rkbsp6.1-joshua-git branch name seems incorrect, I submitted a PR.

wyf9661 commented 5 months ago

@kwankiu

I noticed that it was removed yesterday with the introduction of linux-rkbsp

for some reason, I thought joshua won't be focus on kernel 6.1, so I delete this repo from nightly build and create a non-git repo which catches from rkbsp upsteam directly.

but now linux-rkbsp-joshua-git is updated to rk-6.1-rkr1 with new wifi support and patches

new patch about wifi and usb is also involved in the package linux-rkbsp in 7Ji, If you don't mind, you can install this package as well.

https://github.com/7Ji-PKGBUILDs/linux-rkbsp6.1-joshua-git branch name seems incorrect, I submitted a PR.

thanks a lot, it was a mistake. I'v merged your PR in master.

will linux-rkbsp-joshua-git be built for archrepo?

maybe it will not at least recently, it is suggested to download the repo and bulid it temporarily.

kwankiu commented 5 months ago

new patch about wifi and usb is also involved in the package linux-rkbsp in 7Ji

I think that Joshua has some new commits after you added linux-rkbsp? I tested it yesterday and it does not work. Then joshua came up with the rk-6.1-rkr1

wyf9661 commented 5 months ago

new patch about wifi and usb is also involved in the package linux-rkbsp in 7Ji

I think that Joshua has some new commits after you added linux-rkbsp? I tested it yesterday and it does not work. Then joshua came up with the rk-6.1-rkr1

The package yesterday is built with linux-rkbsp-1, but I update the repo and change the pkgrel to 2 after it is built, you need to build it temporarily as well...

kwankiu commented 5 months ago

@kwankiu

I noticed that it was removed yesterday with the introduction of linux-rkbsp

for some reason, I thought joshua won't be focus on kernel 6.1, so I delete this repo from nightly build and create a non-git repo which catches from rkbsp upsteam directly.

but now linux-rkbsp-joshua-git is updated to rk-6.1-rkr1 with new wifi support and patches

new patch about wifi and usb is also involved in the package linux-rkbsp in 7Ji, If you don't mind, you can install this package as well.

https://github.com/7Ji-PKGBUILDs/linux-rkbsp6.1-joshua-git branch name seems incorrect, I submitted a PR.

thanks a lot, it was a mistake. I'v merged your PR in master.

will linux-rkbsp-joshua-git be built for archrepo?

maybe it will not at least recently, it is suggested to download the repo and bulid it temporarily.

Since I am testing and is moving forward to use Joshua’s kernel for the installer I made https://github.com/kwankiu/archlinux-installer-rock5 the removal of the -git kernel packages broke my installer yesterday, then I noticed that you added the linux-rkbsp package.

wyf9661 commented 5 months ago

@kwankiu

I noticed that it was removed yesterday with the introduction of linux-rkbsp

for some reason, I thought joshua won't be focus on kernel 6.1, so I delete this repo from nightly build and create a non-git repo which catches from rkbsp upsteam directly.

but now linux-rkbsp-joshua-git is updated to rk-6.1-rkr1 with new wifi support and patches

new patch about wifi and usb is also involved in the package linux-rkbsp in 7Ji, If you don't mind, you can install this package as well.

https://github.com/7Ji-PKGBUILDs/linux-rkbsp6.1-joshua-git branch name seems incorrect, I submitted a PR.

thanks a lot, it was a mistake. I'v merged your PR in master.

will linux-rkbsp-joshua-git be built for archrepo?

maybe it will not at least recently, it is suggested to download the repo and bulid it temporarily.

Since I am testing and is moving forward to use Joshua’s kernel for the installer I made https://github.com/kwankiu/archlinux-installer-rock5 the removal of the -git kernel packages broke my installer yesterday, then I noticed that you added the linux-rkbsp package.

image So I delete it and set the non-git package, as the latest package in 7ji repo of linux-rkbsp-joshua-git can only use mesa, the panfork can not work.

kwankiu commented 5 months ago

Ok, I just want to make sure it is now safe to use linux-rockchip-joshua right? Since I plan to use it as a default option.

wyf9661 commented 5 months ago

Ok, I just want to make sure it is now safe to use linux-rockchip-joshua right? Since I plan to use it as a default option.

it is stable and I will keep update it as daily:)

7Ji commented 5 months ago

@kwankiu

Sorry if there're missing/outdated packages and they're not fixed in time. I've been invited to work in a different city 1200 km away from my home before the Chinese New Year and I set out 2 days ago after the holiday. It's very important to me as this is my first job. Since then, I've been busy moving home, renting apartment, and doing other stuffs to get ready to work in this city.

The building server (opi5 plus) was therefore put down and it's one of the few devices I brought with me, without my network infrastructure. I'll need a few days to settle down before I could bring it back online 7x24. After I received my wage I'll try to register a VISA credit card so I could open a free ARM server on Oracle Cloud and migrate the builder there so I don't need to keep a SBC up 7*24 any more. (By the way, the x86_64 builder was already migrated to one of my VPS)

Before that, please wait.

hbiyik commented 4 months ago

@7Ji

after a vacation and a short development on AGR and several things here are my proposal for future management of the repo:

1) Gitweb-dlagent: https://github.com/7Ji-PKGBUILDs/gitweb-dlagent

with this, no need to clone the whole git trees for big -git packages. This thing handles them by scraping the github and affectively reducing the download time and disk usage up to 5x.

2) AGR Submodules:

https://github.com/hbiyik/agrrepo now agr fully handles sub git repos, this means i can use the 7ji-pkgbuilds as a single source of repo for each package, and use them in agr. As it is current now, all the packages are now linked to 7ji-pkgbuilds, except kodi, which is problematic to build each and every time so AGR is a good fit for kodi only.

Questions:

To make this work, i need to use ffmpeg-rockchip-git and linux-radxa-rkbsp5-git packages with gitweb-dlagent, i have migrated them in their respective gitweb branches here and here. But i have not made them in master yet, since i think you would like to be informed because now you have a new gitweb-dlagent make dependency.

addionatlly: gl4es-git, mpp-git, librga-multi and chromium-mpp packages are fixed/improved.

7Ji commented 4 months ago

addionatlly: gl4es-git, mpp-git, librga-multi and chromium-mpp packages are fixed/improved.

Thank you as always

I appreciate the changes to the PKGBUILDs and the efforts you've put in agr and gitweb-dlagent.

However, I don't like the intrusive changes needed in PKGBUILDs if gitweb-dlagent would be used. For a long time I've always tried to keep these PKGBUILDs not locked-in with either my repo or my builder. I wrote a full PKGBUILD parser in the builder just so I could let my builder parse and cache the sources without modifying PKGBUILDs, while others could still just git clone and makepkg, without using any of my builders and scripts. If we introduce the gitweb-dlagent changes into the PKGBUILDs, then third-party builders would be forced to install gitweb-dlagent to build those PKGBUILDs, be them a single user, or another repo to use our PKGBUILDs, or any one in any case that just want to git clone the PKGBUILD and makepkg to check the changes before our updates, a forced-in dependency is not enjoyable.

The problem about building kodi, for my builder, is not about fetching git repos every time, that's not the case. My builder arb would parse the PKGBUILDs to get the lists of git sources and cache deduped git repos under sources/git alongside sources/file-sha256, sources/file-md5, etc, and they survive through rebuilds and reboots. For each new run of the repo builder, it just fetch a subset of refs to keep them up-to-date, and hashing those file caches to ensure integrity and download them lazily. For each build, git repos and files are linked to the build folder and makepkg routine would treat them as-if non-link sources. And each build would have its own chroot so deps would not conflict.

The problem about building kodi, is that it actively download files in its own routine, even if you've prepared all of the sources in the makepkg routine. Caching all sources listed in PKGBUILD does not help with this. As long as we do clean build these sources only known to kodi's own build system need to be re-downloaded. Re-build on existing build folder is a no-go because makepkg's routine is not clever enough on this and more than a few PKGBUILDs would break in this case. This would be solved if I run the whole builder behind a proxy (I could, as the international bandwidth is not jammed any more after I moved to a different city), or on an ARM VPS I rent outboard (only would be possible after I get a VISA credit card after I receive my wages for a few months as I just began to work for my first job).

Therefore the changes about gitweb-dlagent would not really improve the repo's building performance about -git packages,, as the builder already caches git sources. And even if it would nicely improve the performance of calculating some metadata we would use for pkgver, I don't like the idea the PKGBUILDs must be modified with intrusive changes and new builds without dlagent would become impossible.

hbiyik commented 4 months ago

I understand, then i can easily manage two branches in such repos.

Currently 4, rkbsp5 kernel, ffmpeg-rockchip, kodi-nexus, kodi-nexus-binary-addons. kodi stuff is not in the builder and only in agr so two of them i can maintain 2 branches.

Because it really helps for normal users where normally people do not use SRCDEST to cache the pkgbuild sources, and even if they use, the downloaded source is 3x~4x less. Ie: kodi in total is 8gb when cloned with git since commit 1, 2GB when gitweb-dlagent is used. (see huge git repo deps at kodi-binary-addons )Similar for kernel packages.

For kodi, it is possible to force the MIRROR for internal deps by using an env variable, but on the otherhand, i understand it is quite painful to build it each an everything about kodi so we can just remove it from the builder and from the 7ji-pkgbuilds as well. https://github.com/7Ji-PKGBUILDs/.report/issues/19#issuecomment-1928590363

So nothing to be done further i guess, i am moving to joshua kernel stuff, those packages and may be even the joshua's git tree might need some maintenance. I talked with @Joshua-Riek he said he is expecting PRs and ready for cooperation.

edit: @7ji tagged for notification

7Ji commented 4 months ago

Because it really helps for normal users where normally people do not use SRCDEST to cache the pkgbuild sources, and even if they use, the downloaded source is 3x~4x less. Ie: kodi in total is 8gb when cloned with git since commit 1, 2GB when gitweb-dlagent is used. (see huge git repo deps at kodi-binary-addons )Similar for kernel packages.

2GB is an abnormal number, because you don't need to fetch all refs, and don't need to check out your git sources. The source for kodi should take only ~825M:

nomad7ji@ali /s/repo (master)> du -sh sources/git/4c42255751faacb5
825M    sources/git/4c42255751faacb5
nomad7ji@ali /s/repo (master)> grep url sources/git/4c42255751faacb5/config 
        url = https://github.com/xbmc/xbmc.git

There is also another non-intrusive way to cache git repos: https://github.com/7Ji/archpkg_cachegit . This expects a running instance of https://github.com/7Ji/git-mirrorer to cache a list of git repos in a centralized way. I use that for kernel, systemd, etc any way so there's not extra effort needed.

For kodi, it is possible to force the MIRROR for internal deps by using an env variable, but on the otherhand, i understand it is quite painful to build it each an everything about kodi so we can just remove it from the builder and from the 7ji-pkgbuilds as well.

I don't quite want to remove it from the build list. The situation was temporary, and I'll re-enable that for the builder soon.

hbiyik commented 4 months ago

There is also another non-intrusive way to cache git repos: https://github.com/7Ji/archpkg_cachegit . This expects a running instance of https://github.com/7Ji/git-mirrorer to cache a list of git repos in a centralized way. I use that for kernel, systemd, etc any way so there's not extra effort needed.

i like the approach with it, it also eliminates having redundant repo clones since the cache is hash based. I think very similar concept of prefetching might be in another form available in git where you need to instantiate a daemon. But nevertheless this eliminates most of the source related build bottlenecks in "-git" packages.

2GB is an abnormal number, because you don't need to fetch all refs, and don't need to check out your git sources. The source for kodi should take only ~825M:

That was the whole package directory size when the package is built (debug smys disabled), a total user benefit in terms of size. If only the source size is compared, the gains are more marginal. Ie xbmc#branch=Nexus is 52.1MB where git is 825MB, so the gain is 15x.

actual source sizes with gitweb-dlagent: xbmc.git: 52.1MB ffmpeg.git: 13.8MB binaryaddons (85 different repos!) : 159.2MB

I don't quite want to remove it from the build list. The situation was temporary, and I'll re-enable that for the builder soon.

Then i need to sync those packages. I have completely seperated Kodi packages and binary addons package where there is 85 different addon. I think binary-addons dont need to be built right? if you build it, i am fine and appreciate, but it really takes a lot of time and size.

Other than than, i suggest to rename the package as previously discussed.

kodi-nexus-mpp-git -> kodi-mpp-git kodi-nexus-binary-addons-git -> kodi-binary-addons-git (agr only?)

i can handle the release cycle of kodi (ie: kodi-nexus-, kodi-omega-) in agr, but the 7ji repo can have only one kodi, which is the current latest version. I can have the older versions in agr as the lifecycle continues.

Ofc, i will update kodi stuff with normal git in master branch where you can build them as they are, with gitweb in gitweb branch for agr. This is not much effort for me,

I think i can not implement archpkg_cachegit to agr, since it requires mirror daemon to run, and thats a complicated task for normal users. The point of AGR is to simulate AUR experience since we are exiled from there.

hbiyik commented 4 months ago

in the meanwhile, does it makes sense to have some general rules on kernel package naming?

7Ji commented 4 months ago

in the meanwhile, does it makes sense to have some general rules on kernel package naming?

Yes, there shall be a rule. But the current situation makes it very painful to transform to whatever new rule we would enforce. We shall discuss that in a new thread, voting for a few proposals.

7Ji commented 4 months ago

I've renamed this repo to .meta as I decided to store the package guidelines here, it's more than report now.

This is the package guidelines I just wrote: https://github.com/7Ji-PKGBUILDs/.meta/blob/main/PackageGuidelines.md , the naming guideline about kernel package is also in there. Any one with write permission could modify that, and those without could open PR.