Resolving indirect AUR dependencies (via `Provides` field)

[polygamma]

Hey!

Trying to install ros-indigo-desktop-full (https://aur.archlinux.org/packages/ros-indigo-desktop-full) yields that python2-catkin-pkg cannot be found in AUR and hence ros-indigo-desktop-full cannot be installed.

Two things:

• It would be nice if pikaur would show that the missing dep is a dep of ros-indigo-catkin
• It actually is possible to know for pikaur that python2-catkin-pkg is being provided by python2-catkin_pkg since it's being listed explicitly in other packages of the dependency chain, hence it should be possible to solve the dependency chain for ros-indigo-desktop-full all in all

[actionless]

the thing is what search in AUR's rpc is not indexing Provides field so the only thing which i can do here -- get the full package list from AUR (like in -Ss without args) and next search through local 'mirror' of AUR db for such packages

but it takes about half minute, so mb before going to that type of search to prompt user for confirmation (like Package not found in AUR. Try to sync AUR locally to analyze 'Provides' sections of the packages?)

[AladW]

Do any of you two know PHP? Sounds easier to file a patch to aurweb than come up with a good workaround here :smile:

https://bugs.archlinux.org/task/49090

Also what is your definition of "local mirror"? Do you retrieve packages.gz to query the AUR for all of them and merge the output somehow? IDK...

[actionless]

i don't know php but i think i can add to RPC one more filter criteria, as name and name-desc as it have now to have also either provides or name-desc-provides (probably someone else should come up with a better name )

[actionless]

i didn't noticed your edits on the original comment:

regarding local mirror, yes, see here: https://github.com/actionless/pikaur/blob/master/pikaur/aur.py#L306-L310

[polygamma]

@actionless Reference: https://github.com/AladW/aurutils/issues/10#issuecomment-370270445

The first part explains how aurman fetches packages and thus gets the needed information. May be interesting for you.

[actionless]

@AladW i had a moment to have a look on https://git.archlinux.org/aurweb.git/tree/web/lib/aurjson.class.php it doesn't look complicated, what is the procedure to submit a patch?

[actionless]

or alternative approach to fixing that on AUR RPC side could be when searching by name match not only Name field but also Provides field

(but that should make search by name slower)

[AladW]

@actionless the procedure is to send your patch to aur-dev where it can be merged after discussion. If you haven't used git send-email before, see https://www.freedesktop.org/wiki/Software/PulseAudio/HowToUseGitSendEmail/

[actionless]

so i can't start a discussion topic / send patch without subscribing to the whole mail list?

[AladW]

I don't recall if you have to be actually subscribed to send messages. If you do and you don't want to recieve other topics, you can disable recieving them in the mailman settings.

[actionless]

ok, i'll try doing that tonight or so

[actionless]

@AladW how do you think which of the approaches will be more acceptable:

1) adding name-desc-provides filter 2) adding provides filter 3) when searching by name match not only Name field but also Provides field 4) mb smth else?

[AladW]

Personally I would prefer the third option, since I'm not sure what the benefits of an additional filter over a transparent approach are. I don't speak for the AUR development team though, so I'd propose these options on aur-dev as well. (the list is equally suited for discussion and posting of patches)

[actionless]

@AladW i've submitted a message there but it didn't appeared here: https://lists.archlinux.org/pipermail/aur-dev/2018-March/date.html

could it be because my e-mail address is not subscribed to the mail list?

[AladW]

Possible, you should have had a rejection mail in that case. Try subscribing then posting it (as mentioned, you can opt-out to receive messages from the list while still being subscribed)

[actionless]

i haven't received rejection neither, but i'll try to subscribe and send the message again

[actionless]

oh, or i can just parse AUR html web page which lists all the package alternatives :dancing_women:

[AladW]

...please don't.

[actionless]

@AladW see last message from Florian in the mail list -- such kind of control freaks is the reason why i didn't wanted to touch the mail list even with a long stick

[AladW]

You shouldnt take it personally - the Arch mailing lists have a long history with email issues so people will remind you immediately. Anyway as you have may have noticed there's a positive response to the proposed changes so it's worthwhile to continue working on them. :smiley:

[actionless]

don't worry, i'm not giving up ;-)

just mentioned why i so wanted to avoid that initially

[XavierCLL]

There are provides problems too when pikaur try to delete the make dependencies after finished making a package: (here python-distribute is provides by python-setuptools)

[actionless]

@XavierCLL could you check this again with the latest git version of pikaur?

[XavierCLL]

not yet, same issue using Pikaur 0.8+19+g094ee87

[actionless]

@XavierCLL pushed more fixes

[XavierCLL]

perfect, the last commit fixed the problem, thanks @actionless

[polygamma]

@actionless, I've looked at the following patch: https://transfer.sh/L3H4Z/0001-feat-rpc-return-providers-packages-when-querying-by-.patch

I think it's actually a very bad idea to alter the existing name and name-desc fields. People are already working with that, and if you change it, many things might break, since you are changing what those fields do. Everybody, who is working with AurJson needs to change existing code, to adapt to the new usage of the fields. Such API changes should really never, ever happen.

[actionless]

i did proposed another alternatives but this one was chosen, feel free to file a patch implementing that in an other way

[AladW]

Simple solution: bump the RPC version to 6. People who want the old API can keep using 5.

[Morganamilo]

What about adding this to info? info allows multiple arguments, so say a package you're fetching the dependencies for has 10 AUR dependencies. Before you would just do 1 info request with those 10 packages. Now you would have to do an individual search request for every dependency then get info from then. Which brings the request count up to 11.

Not to mention search is for, well searching. It does not look for exact names. So when you have the dependency foo and you want to find packages that also provide foo after you preform the search on the rpc you would have to check each package to see if they actually provide foo, or just happen to have a similar name.

I think this does belong in name and name-desc for -Ss type searching but for actual dependency resolving this should also also be added to info.

[actionless]

time passed from submitting patch for AUR: 10 days time necessary to implement in python parsing of such a simple html page: 10 minutes or so

i'm putting this to 0.10 milestone (magic number here is not intentional ;-) ) and so if it won't be any progress till then will merge html-based solution

[afq984]

Late to the discussion here, and I don't actually know how to post on the mailing list, so I'm posting my thoughts here.

I think we'd better introduce a new type=provides or search_by=provides. My understanding is that there's no way in the RPC we can get a list of packages that are capable of providing the dependency. Putting it into the search means AUR helpers will have to verify whether the search results actually provides the required dependency afterwards.

@actionless How did you get the testing server working? I'm interested in creating a patch but have trouble getting the database set up.

What I've done in the config is this but I think it's wrong someway as the web page shows Error - Could not connect to AUR database:

[database]
backend = sqlite
name = /home/afg/aurweb/aurweb.sqlite3

Parsing the html just seems soooo easy compared to this though :(

[actionless]

@afg984 i was also complaining on the maillist about having readme and not working example config files (which have to be edited manually according to the readme to get them work :D) instead of just small clean Docker/Vagrant file which instantly works as expected and serves as a reference setup example. that's just so not right in 2018.

regarding your problem specifically -- mb you haven't enabled pdo module for sqlite

[Morganamilo]

To be more specific, uncomment extention=pdo_sqlite in /etc/php/php.ini.

[AladW]

time necessary to implement in python parsing of such a simple html page: 10 minutes or so

Yes, and that also means that everyone else who wants this would also have to implement expensive and ugly HTML parsing. A patch means everyone can take advantage of a new and improved API.

PS. I agree with @Morganamilo on adding this to info

[actionless]

there's still no reviews after i've submitted it in the readable format, nuff said

[AladW]

Eh. You submitted it as a link to some 10 mb javascript page that doesn't even display the actual file. I linked the standard procedure over 2 weeks ago in https://github.com/actionless/pikaur/issues/53#issuecomment-370431173, not sure why it's so hard to follow.

I certainly managed to do so in the past and I consider myself the most clumsy CLI user in the history of Arch.

[afq984]

Thanks for the hints, got local aurweb running.

I'm all for having the information in info. Still hoping we can search_by=provides for somewhat reason we use a PKGBUILD that's not obtained from the AUR but want resolve dependencies properly. It's not possible without crawling the whole AUR.

For the html stuff, I think fetching the information from the web and also comparing the results with the RPC is quite reliable. A xpath query of '//*[@id="pkgdepslist"]/li' will just do the trick.

[actionless]

@AladW i've already explained what that procedure is not designed for accessing not from the personal pc

btw transfer.sh is opening files with noscript enabled

but anyway i already know what if i will upload it differently those guys will find some new way to complain and write 9000 lines of text intead of writing 3 lines of code to just fix the AUR

[Morganamilo]

I'll submit it right now if you like. I would like this to get reviewed so that we can progress with this. When your patch is all fixed up and finalised I can look into making a version of it for info.

[actionless]

yup, count the patch as public domain

[Morganamilo]

Sent. Hope it goes somewhere so I can translate it to info.

[actionless]

@Morganamilo @AladW any move on this on AUR side? i am about to close the milestone soon

[AladW]

Upstream agrees it's a good thing to add. The only remaining point to argue about is what to do with info queries.

https://lists.archlinux.org/pipermail/aur-dev/2018-March/004495.html

[actionless]

i think if it will be one api endpoint for both search and info it wouldn't be such design decision problem as now

imho instead of having that branching in code to support every request argument explicitly it could be better if it will be just handling any of fields available for search and just abstract api endoint, which takes one or more fieldnames and their values and doing search. i think such implementation will be the same SLOC as it is now or so but support all the fields for querying instead of few fixed ones

[AladW]

Until someone implements this, I've thought of an alternate approach: just retrieve ALL metadata from the RPC and store it locally, perhaps in some modified form. As this is ~20mb of JSON, this should be done manually through --updatedb or some such flag. This data could then be used to find the correct providers (and much more, such as full regex support for searches) - in fact, a local db synced from the AUR was one of the original suggestions to handle issues such as this one.

[actionless]

why do you think it will be better than parsing a web page?

[actionless]

btw i've just realized what retrieving of all aur metadata is already implemented in pikaur -Ss #without args (right now progressbar is broken after i switched from asyncio to threading, was going to address it before 0.10 release)

what is missing is only saving/loading that metadata to disk

--updatedb

i was more into doing that on --refresh because it seems to be almost the same thing from the user point of view, and so it could be controlled with additional --aur or --repo options (as it's done now in, for example, -Ss)

[AladW]

Because parsing the webpage for 50 000 packages takes an absurd amount of time, while doing so over the RPC doesn't? Anyway it should be a separate flag since it puts severe tax on the AUR, especially if users do an -Sy every 10 seconds.

[Morganamilo]

I think the point is to just parse the web pages to be installed. For example pikaur -S pikaur would fetch the deps from:

No need to go through 50k pages.