search

repology / repology-updater

Repology backend service to update repository and package data
https://repology.org
GNU General Public License v3.0
491 stars 172 forks source link

Additional repository support ideas #1

Closed AMDmi3 closed 3 years ago

AMDmi3 commented 7 years ago

Please share any ideas on what additional repositories we can support. A description on how to fetch all package data from specific repository is preferred. Approved repositories with determined fetching algorithm are split to separate bugs and eventually implemented.

Classic *nix package repositories

From Fedora release-monitoring

Unsorted

Other platforms

Since these will contain too many unrelevant unique packages, doable as shadow repos:

Doable as shadow repos as well

...more ideas?

AMDmi3 commented 7 years ago

Homebrew doesn't seem to specify version numbers for formulas. Parse it from url?

jbicha commented 7 years ago

Ubuntu

AMDmi3 commented 7 years ago

Ubuntu

It is already supported (repo is commented out though). It'll be more easily available from command line with the next update I'm going to push quite soon. I didn't plan to support it on the website though, as it seemed to have the same versions as Debian, but I see now that's not the case, so I'll add it.

jktrigg commented 7 years ago

SuSE

AMDmi3 commented 7 years ago

SuSE

Any ideas how to get information on its packages?

jktrigg commented 7 years ago

Tumbleweed (equivalent to CURRENT): http://download.opensuse.org/tumbleweed/repo/oss/suse/{noarch,x86_64} Leap (equivalent to STABLE): http://download.opensuse.org/distribution/leap/42.1/repo/oss/suse/{noarch,x86_64}

AMDmi3 commented 7 years ago

These contain too little info. Well, they are parsable so can try, but I can only parse package names vesions out of it, while the project already uses maintainer field and will use more fields in future. Isn't there a way to get all .spec files?

AMDmi3 commented 7 years ago

@jktrigg I've added OpenSUSE support they way you've described. It's not really suitable for comparison though, as these binary packages have different naming schema: for example, for each libfoo in other repos we'll have libfooN libfoo-devel and libfooN-32bit here.

AMDmi3 commented 7 years ago

@jbicha FYI, Ubuntu is fully supported @jktrigg FYI, OpenSUSE is fully supported

vadi2 commented 7 years ago

I'd like to see homebrew supported - I see http://braumeister.org is able to show the version information.

AMDmi3 commented 7 years ago

@vadi2 me too. Homebrew author is defiantly unsupportive, so yes, braumeister.org is our hope. I've already filed https://github.com/koraktor/braumeister.org/issues/45, you could :+1: it

AMDmi3 commented 7 years ago

Homebrew support is now tracked in separate issue #198

dg0yt commented 7 years ago

How about MSYS2? It is based on Arch Linux' pacman.

AMDmi3 commented 7 years ago

Are there any links?

dg0yt commented 7 years ago

Apart from http://www.msys2.org/, there is the repository at http://repo.msys2.org/, which has index files such as http://repo.msys2.org/msys/x86_64/msys.files.tar.gz. But I'm not an expert for Arch or pacman.

AMDmi3 commented 7 years ago

Nice! MSYS2 support added.

dg0yt commented 7 years ago

This was quick!

However, I start to understand that the MSYS2 "subsystems" need to be treated individually:

The mingw subsystems provide native Windows programs and are the main focus of the project.

The msys2 subsystem provides an emulated mostly-POSIX-compliant environment for building software, package management, and shell scripting.

Most userland software resides in the mingw subsystem. For example, repology doesn't indicate that there is a gdal package in the mingw subsystem at the moment. So probably there needs to be

(Note: For mingw, there is another minor difference, mingw32 vs. mingw64, but I think this can be neglected.)

AMDmi3 commented 7 years ago

On the second glance, I don't think we can support this yet. We need to parse PKGBUILDs from git repos, and that depends on #166. With processed databases from repo.msys2.org it's impossible to filter out subpackages properly.

AMDmi3 commented 7 years ago

Let's move MSYS2 discussion to #262

tkelman commented 7 years ago

Cygwin: http://mirrors.mit.edu/cygwin/x86_64/setup.ini, http://mirrors.mit.edu/cygwin/x86/setup.ini

AMDmi3 commented 7 years ago

Cygwin

Let's discuss it in #269

ikeydoherty commented 6 years ago

I notice you've got Solus listed there. Note our repos are over at https://dev.solus-project.com

If you need us to provide some parsers or an additional indexing method, let us know. Long story short for the published index in unstable the Source is shared among a package group, and History.Updates[0].Version is the published version.

AMDmi3 commented 6 years ago

@ikeydoherty I think I've already investigated Solus index. What stopped me from adding support for it right away is the fact that the index lists binary packages, while repology prefers source packages. While it's possible to work with binary packages, merging (libfoo, libfoo-devel, libfoo-dbginfo, libfoo-docs, libfoo-32bit-*) etc. requires extra work and is error prone. It could be also possible to get package names from Source.Name, but I haven't investigated yet if that would work. Parsing package.yml files from sources could work too, but I see no way to download them in one go yet. Downloading a repository for each package is not feasible.

ikeydoherty commented 6 years ago

Yep, I'd agree with you on that one. After I'm back from NYC I'll be finishing off the other parts of the new Solus build infra. One of those aspects will require us to have a server performing pulls/validation on all of our source repositories so that we can enable interdependency analysis in build queueing. Honestly I'd planned to defer it for a bit, but this might give the the excuse I need. With that mass-source-import running on our servers, we could simple emit a singular source index for the totality of our repos.

If you're interested in that lemme know, and what sort of formats you'd like to be using (I'm pretty sure we all want XML feeds to die at this point.)

AMDmi3 commented 6 years ago

I'm interested in supporting as many repos as possible :) If there's also collaboration from the repository side, that's absolutely great. JSON dump would be most convenient for me.

AMDmi3 commented 6 years ago

Let's continue discussion in #341

davidak commented 6 years ago

elementary OS 0.4 Loki

based on Ubuntu (deb based)

deb http://ppa.launchpad.net/elementary-os/stable/ubuntu xenial main
deb http://ppa.launchpad.net/elementary-os/os-patches/ubuntu xenial main

They have some original software for their distro, also packaged in AUR and Fedora. Also they will have a new release probably soon, but it will be great to see the progress.

jbicha commented 6 years ago

I don't know if elementary OS has diverged enough for it to be useful.

Here is a different view of the packages for their 0.4 release:

stable

os-patches

jbicha commented 6 years ago

Well on second thought, there are other similarly small derivative distros listed so never mind.

AMDmi3 commented 6 years ago

Elementary doesn't look usable as it uses custom mangled version scheme (appends rXXX).

AMDmi3 commented 6 years ago

Locking this. Please submit new ideas as separate issues.