search

thoth-station / solver

Dependency solver for the Thoth project
https://thoth-station.ninja/
GNU General Public License v3.0
20 stars 12 forks source link

Some package indexes have forbidden access #5221

Closed mayaCostantini closed 1 year ago

mayaCostantini commented 2 years ago

Bug description

When running solvers, the following error occurs when trying to reach some dependency indexes:

requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://download.pytorch.org/whl/cpu/six

The indexes concerned are https://download.pytorch.org/whl/cu111/ and https://download.pytorch.org/whl/cpu/.

Steps to Reproduce

Run a solver for a package in the management API and observe the solver logs. In the case of the error above, the solver scheduled was fedora-35-py310 and the package copr with version specifier "*".

Expected behavior

All indexes used by solvers are accessible.

mayaCostantini commented 2 years ago

/sig stack-guidance /priority critical-urgent

fridex commented 2 years ago

These package indexes host packages published by the PyTorch team. They do not provide all the packages - the log shows that accessing the index gives 403 (which is the response code they configured to return). Thoth should log these and say that these packages are not available from the mentioned indexes. Does this behavior somehow affect Thoth's recommendations?

mayaCostantini commented 2 years ago

Does this behavior somehow affect Thoth's recommendations?

I think it might if a dependency could not be found on any other index we have?

fridex commented 2 years ago

Does this behavior somehow affect Thoth's recommendations?

I think it might if a dependency could not be found on any other index we have?

What would be the expected behavior in such cases?

mayaCostantini commented 2 years ago

What would be the expected behavior in such cases?

If I understand correctly, the current behavior is simply to warn that the given package could not be resolved and to add the corresponding error report to the list of unresolved package reports.

In the case where a user is requesting a package to be solved and the package itself or one of its dependencies cannot be resolved because it is not accessible in any index, maybe we should make sure that this information is correctly propagated to allow the user to try specifying a different index or to remove/modify the requirements. Currently, this information is only present in the solver logs when the error is raised, but the error code (in this case, 403) is not propagated to the error report.

sesheta commented 1 year ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

goern commented 1 year ago

/remove-lifecycle stale /assign @mayaCostantini

goern commented 1 year ago

/assign @VannTen

codificat commented 1 year ago

from today's sig-sg call: /triage accepted /lifecycle active

VannTen commented 1 year ago

Weird, I posted a reply in this last week by email but apparently Github did not catch it... :/.

The relation to storages is not obvious to me, what API should we provide / fix to help this issue ?

VannTen commented 1 year ago

@mayaCostantini any pointers on the above question ?

mayaCostantini commented 1 year ago

@VannTen as not all the packages provided by the https://download.pytorch.org/ index have a forbidden access, I propose we move this issue to the solver repo.

Given the previous discussion on this issue, what about verifying that solver propagates the correct error (forbidden index) when trying to solve a package which has forbidden indexes only? I am not sure this is visible on solver logs, so it might be worth testing the ingestion of a package specifying forbidden indexes only to observe the solver behavior. Ideally, if a package is in a project requirements and couldn't be solved because of this error, users should be informed so that they can change the indexes from which they get the package. Wdyt?

VannTen commented 1 year ago

On Wed, Sep 28, 2022 at 04:11:35AM -0700, Maya Costantini wrote:

@VannTen as not all the packages provided by the https://download.pytorch.org/ index have a forbidden access, I propose we move this issue to the solver repo. +1

Given the previous discussion on this issue, what about verifying that solver propagates the correct error (forbidden index) when trying to solve a package which has forbidden indexes only? I am not sure this is visible on solver logs, so it might be worth testing the ingestion of a package specifying forbidden indexes only to observe the solver behavior. Ideally, if a package is in a project requirements and couldn't be solved because of this error, users should be informed so that they can change the indexes from which they get the package. Wdyt?

Do I understand correctly "forbidden index" as: during package solving, when trying to retrieve the package from that index, we got a 403 error on the request ?

If so, yes, forwaring the error to the user looks like the right thing to do. Presumably in that case the user would have also some kind of error from other software trying to retrieve the package (unless our infra is encountering network/dns/proxy problems somehow)

harshad16 commented 1 year ago

we are catching these errors and implemented exception handling via https://github.com/thoth-station/python/pull/481 in solvers. This would be caught and reflected in logging. closing this issue. please feel free to open it again if needed. thanks.