[3pt] Some packages fail to be solved using solver-rhel-8-py38

[fridex]

Describe the bug

See stage environment - the metric for unsolved packages keeps on ~130 package releases for some time. Checking the namespace, some packages fail to be solved using the solver-rhel-8-py38 in the given time and this solving never succeeds.

Packages that might be affected:

[('uwsgi', '2.0.17.1', 'https://pypi.org/simple'),
 ('spacy-nightly', '2.1.0a9', 'https://pypi.org/simple'),
 ('spacy', '2.1.3', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.8.4', 'https://pypi.org/simple'),
 ('spacy-nightly', '2.1.0a7', 'https://pypi.org/simple'),
 ('uwsgi', '2.0.19', 'https://pypi.org/simple'),
 ('spacy', '2.1.0', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.24.0rc1', 'https://pypi.org/simple'),
 ('spacy-nightly', '2.1.0a9.dev0', 'https://pypi.org/simple'),
 ('spacy', '2.1.4', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.8.3', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.16.0', 'https://pypi.org/simple'),
 ('uwsgi', '2.0.18', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.6.0rc1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.11.1rc1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.2.1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.22.1', 'https://pypi.org/simple'),
 ('multiprocess', '0.70.5', 'https://pypi.org/simple'),
 ('uwsgi', '2.0.19.1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.14.0rc1', 'https://pypi.org/simple'),
 ('numpy', '1.15.1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.1.0', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.2.0', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.14.1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.23.0rc1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.2.0rc1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.20.1', 'https://pypi.org/simple'),
 ('apache-airflow', '2.1.3rc1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.10.0', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.9.0rc1', 'https://pypi.org/simple'),
 ('apache-airflow', '2.0.1rc2', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.14.0', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.22.0', 'https://pypi.org/simple'),
 ('numpy', '1.15.0rc2', 'https://pypi.org/simple'),
 ('blis', '0.3.1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.8.0rc2', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.4.0rc1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.17.0rc1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.16.0rc1', 'https://pypi.org/simple'),
 ('apache-airflow', '2.0.0rc3', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.3.0', 'https://pypi.org/simple'),
 ('spacy', '2.1.1.dev0', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.11.0rc1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.2.0rc2', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.11.0rc2', 'https://pypi.org/simple'),
 ('apache-airflow', '2.0.1', 'https://pypi.org/simple'),
 ('spacy-nightly', '2.1.0a12', 'https://pypi.org/simple'),
 ('apache-airflow', '2.0.0', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.9.1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.15.0', 'https://pypi.org/simple'),
 ('uwsgi', '2.0.15', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.1.0rc1', 'https://pypi.org/simple'),
 ('numpy', '1.15.2', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.17.0', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.14.0rc2', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.17.1rc1', 'https://pypi.org/simple'),
 ('uwsgi', '2.0.16', 'https://pypi.org/simple'),
 ('spacy', '2.1.9', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.3.5', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.10.1rc2', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.18.0', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.21.1rc1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.24.1', 'https://pypi.org/simple'),
 ('blis', '0.4.0.dev0', 'https://pypi.org/simple'),
 ('multiprocess', '0.70.6', 'https://pypi.org/simple'),
 ('apache-airflow', '1.10.15', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.7.3', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.23.1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.15.0rc1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.22.0rc1', 'https://pypi.org/simple'),
 ('scikit-image', '0.18.2rc1', 'https://pypi.org/simple'),
 ('uwsgi', '2.0.17', 'https://pypi.org/simple'),
 ('spacy-nightly', '2.1.0a13', 'https://pypi.org/simple'),
 ('spacy', '2.1.8', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.6.3', 'https://pypi.org/simple'),
 ('spacy', '2.1.1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.16.1', 'https://pypi.org/simple'),
 ('spacy', '2.1.5', 'https://pypi.org/simple'),
 ('spacy', '2.2.0.dev10', 'https://pypi.org/simple'),
 ('spacy', '2.1.6', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.18.0rc1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.12.0rc1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.10.0rc1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.10.1rc1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.4.0', 'https://pypi.org/simple'),
 ('spacy', '2.1.7', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.19.0', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.8.2', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.10.0rc2', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.8.1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.1.3', 'https://pypi.org/simple'),
 ('spacy', '2.1.2', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.16.1rc1', 'https://pypi.org/simple'),
 ('apache-airflow', '2.0.0rc2', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.24.0', 'https://pypi.org/simple'),
 ('apache-airflow', '1.10.15rc1', 'https://pypi.org/simple'),
 ('pyppmd', '0.16.1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.14.2rc1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.8.0rc1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.21.1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.14.2', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.12.1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.10.1', 'https://pypi.org/simple'),
 ('spacy', '2.1.7.dev0', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.9.0rc3', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.20.0', 'https://pypi.org/simple'),
 ('spacy', '2.2.0.dev13', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.6.0', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.20.0rc2', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.11.1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.9.0rc2', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.8.0rc3', 'https://pypi.org/simple'),
 ('spacy-nightly', '2.1.0a8', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.13.0rc3', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.21.0rc1', 'https://pypi.org/simple'),
 ('spacy', '2.2.0.dev11', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.20.0rc1', 'https://pypi.org/simple'),
 ('spacy', '2.2.0.dev15', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.12.0', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.11.0', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.23.0', 'https://pypi.org/simple'),
 ('multiprocess', '0.70.6.1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.9.0', 'https://pypi.org/simple'),
 ('moto', '2.2.4.dev7', 'https://pypi.org/simple'),
 ('numpy', '1.15.0', 'https://pypi.org/simple'),
 ('multiprocess', '0.70.7', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.8.6', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.13.0', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.19.0rc1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.17.1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.7.0rc1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.13.0rc2', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.13.0rc1', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.20.0rc3', 'https://pypi.org/simple'),
 ('grpcio-tools', '1.7.0', 'https://pypi.org/simple'),
 ('spacy-nightly', '2.1.0a10', 'https://pypi.org/simple')]

Note not all the packages might be causing troubles as I saw package-releases job run in parallel that inserted some new releases.

To Reproduce Steps to reproduce the behavior:

1. Go to the stage environment
2. See failed solver jobs that keep rescheduled
3. See the error

Expected behavior

Package releases should be successfully solved.

Additional context

I tried to locally resolve grpcio-tools===1.1.3 using thoth-solver, it looks like it works. Solver jobs reported exceeded time to solve the job.

Acceptence criteria

• [ ] Execute the solver on the issue and list out the error
• [ ] implement the fix for the solver and adjust the error out packages
• [ ] verify the list of packages from the above table and go through them.

[sesheta]

@fridex: This issue is currently awaiting triage. One of the @thoth-station/devs will take care of the issue, and will accept the issue by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[fridex]

Briefly checking the releases (checked grpcio-tools and spacy)- the issue behind timeout might be a need to compile these packages as they do not provide pre-built wheel binaries for Python 3.8. Given the hardware resources and CPU time allocated, the build in solver workflow never succeeds and thus the workflow task timeouts.

[erikerlandson]

xref https://github.com/AICoE/aicoe-ci/issues/130 for tunable build resource allocations

[fridex]

xref AICoE/aicoe-ci#130 for tunable build resource allocations

This one is related to solver jobs we run to aggregate dependency information for resolver. Resources are part of the corresponding solver Argo workflow task in this case:

https://github.com/thoth-station/thoth-application/blob/3bfb721f99324788173a697decfce0794c030945/solver/base/argo-workflows/solver-template.yaml#L115-L121

[goern]

Briefly checking the releases (checked grpcio-tools and spacy)- the issue behind timeout might be a need to compile these packages as they do not provide pre-built wheel binaries for Python 3.8. Given the hardware resources and CPU time allocated, the build in solver workflow never succeeds and thus the workflow task timeouts.

Shall we create a re-build pipeline that will do the compilation and publish to our own PI?

[goern]

/priority important-soon /area solver

[fridex]

Briefly checking the releases (checked grpcio-tools and spacy)- the issue behind timeout might be a need to compile these packages as they do not provide pre-built wheel binaries for Python 3.8. Given the hardware resources and CPU time allocated, the build in solver workflow never succeeds and thus the workflow task timeouts.

Shall we create a re-build pipeline that will do the compilation and publish to our own PI?

Good idea for the future - we could do it for the selected set of packages that Thoth users use.

[sesheta]

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

/close

[sesheta]

@sesheta: Closing this issue.

In response to [this](https://github.com/thoth-station/support/issues/11#issuecomment-922240363): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[goern]

/remove-lifecycle rotten /lifecycle frozen /reopen

[sesheta]

@goern: Reopened this issue.

In response to [this](https://github.com/thoth-station/support/issues/11#issuecomment-922815698): >/remove-lifecycle rotten >/lifecycle frozen >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[harshad16]

cc: @harshad16

[codificat]

/triage accepted

[mayaCostantini]

Briefly checking the releases (checked grpcio-tools and spacy)- the issue behind timeout might be a need to compile these packages as they do not provide pre-built wheel binaries for Python 3.8. Given the hardware resources and CPU time allocated, the build in solver workflow never succeeds and thus the workflow task timeouts.

Shall we create a re-build pipeline that will do the compilation and publish to our own PI?

Good idea for the future - we could do it for the selected set of packages that Thoth users use.

@goern @harshad16 is that something we still want to implement?

[goern]

+1 @mayaCostantini could you extract this out into an issue on its own and add it to https://github.com/orgs/open-services-group/projects/21