search

thoth-station / management-api

An API service used for administration of deployed Thoth
GNU General Public License v3.0
3 stars 11 forks source link

solver test failed #659

Closed goern closed 3 years ago

goern commented 3 years ago

Describe the bug

Feature: Check for minimum solvers available in Thoth # features/solvers.feature:1

  Scenario: Check there are minimum solvers available in Thoth   # features/solvers.feature:2
    Given deployment is accessible using HTTPS                   # features/steps/basic.py:31 1.106s
    When we ask for the available solvers                        # features/steps/solvers.py:30 0.591s
    Then they should include at least the minimum set of solvers # features/steps/solvers.py:39 0.001s

  Scenario: Schedule solver jobs for all available solvers in Thoth      # features/solvers.feature:7
    Given deployment is accessible using HTTPS                           # features/steps/basic.py:31 1.095s
    When we ask for the available solvers                                # features/steps/solvers.py:30 0.556s
    Then schedule solver analyses for package selinon with version 1.0.0 # features/steps/solvers.py:47 0.848s
    Then wait for analyses to finish successfully                        # features/steps/solvers.py:64 135.990s
      Assertion Failed: Solver solver-rhel-8-py36-d20ef8e118cae6b9d76e4db031864ec9 run on management.stage.thoth-station.ninja was not successful

To Reproduce Steps to reproduce the behavior:

  1. run integration-tests

/kind bug /label sig/devops

pacospace commented 3 years ago

Describe the bug

Feature: Check for minimum solvers available in Thoth # features/solvers.feature:1

  Scenario: Check there are minimum solvers available in Thoth   # features/solvers.feature:2
    Given deployment is accessible using HTTPS                   # features/steps/basic.py:31 1.106s
    When we ask for the available solvers                        # features/steps/solvers.py:30 0.591s
    Then they should include at least the minimum set of solvers # features/steps/solvers.py:39 0.001s

  Scenario: Schedule solver jobs for all available solvers in Thoth      # features/solvers.feature:7
    Given deployment is accessible using HTTPS                           # features/steps/basic.py:31 1.095s
    When we ask for the available solvers                                # features/steps/solvers.py:30 0.556s
    Then schedule solver analyses for package selinon with version 1.0.0 # features/steps/solvers.py:47 0.848s
    Then wait for analyses to finish successfully                        # features/steps/solvers.py:64 135.990s
      Assertion Failed: Solver solver-rhel-8-py36-d20ef8e118cae6b9d76e4db031864ec9 run on management.stage.thoth-station.ninja was not successful

To Reproduce Steps to reproduce the behavior:

  1. run integration-tests

/kind bug /label sig/devops

I rescheduled same package

Name:                solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce
Namespace:           thoth-middletier-stage
ServiceAccount:      argo
Status:              Failed
Conditions:          
 Completed           True
Created:             Thu Jan 14 12:56:14 +0100 (3 minutes ago)
Started:             Thu Jan 14 12:56:14 +0100 (3 minutes ago)
Finished:            Thu Jan 14 12:59:39 +0100 (now)
Duration:            3 minutes 25 seconds
ResourcesDuration:   3m23s*(1 cpu),12m49s*(100Mi memory)
Parameters:          
  ceph_bucket_prefix: data
  ceph_bucket_name:  thoth
  ceph_host:         s3.upshift.redhat.com
  deployment_name:   ocp4-stage
  THOTH_SOLVER_NAME: solver-rhel-8-py36
  THOTH_SOLVER_WORKFLOW_ID: solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce
  THOTH_LOG_SOLVER:  INFO
  THOTH_SOLVER_NO_TRANSITIVE: 1
  THOTH_SOLVER_PACKAGES: selinon==1.0.0
  THOTH_SOLVER_INDEXES: https://pypi.org/simple
  THOTH_FORCE_SYNC:  0

STEP                                                    TEMPLATE               PODNAME                                                         DURATION  MESSAGE
 ✖ solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce  solve-and-sync                                                                                                            
 ├-✔ solverany(0)                                       solver-any/solve       solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-112752507   30s                                
 ├-✖ graph-sync-solverany                               graph-sync/graph-sync                                                                            No more retries left     
 | ├-✖ graph-sync-solverany(0)                          graph-sync/graph-sync  solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-4184177582  54s       failed with exit code 1  
 | ├-✖ graph-sync-solverany(1)                          graph-sync/graph-sync  solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-3580036203  33s       failed with exit code 1  
 | ├-✖ graph-sync-solverany(2)                          graph-sync/graph-sync  solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1969929232  33s       failed with exit code 1  
 | └-✖ graph-sync-solverany(3)                          graph-sync/graph-sync  solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757  43s       failed with exit code 1

The issue is related to graph-sync-job, solver task succeeded, but the graph-sync-job cannot be performed by design for package that we already solved to prevent from duplicates.

solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.780162303Z Traceback (most recent call last):
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.783238253Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.78593985Z 
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.78593985Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.78593985Z     cursor.execute(statement, parameters)
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.78593985Z psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "python_package_version_package_name_package_version_python__key"
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.78593985Z DETAIL:  Key (package_name, package_version, python_package_index_id, os_name, os_version, python_version)=(selinon, 1.0.0, 1, rhel, 8, 3.6) already exists.
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.78593985Z 
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.78593985Z 
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.78593985Z The above exception was the direct cause of the following exception:
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.78593985Z   File "/opt/app-root/lib64/python3.8/site-packages/thoth/storages/graph/models_base.py", line 52, in get_or_create
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.78593985Z Traceback (most recent call last):
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.78593985Z     self.dialect.do_execute(
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.78593985Z     session.commit()
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.78593985Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/orm/session.py", line 1042, in commit
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.78593985Z     self.transaction.commit()
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.78593985Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/orm/session.py", line 504, in commit
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.78593985Z     self._prepare_impl()
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.78593985Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.866682108Z     self.session.flush()
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.866682108Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/orm/session.py", line 2536, in flush
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.867475608Z     self._flush(objects)
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.867475608Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/orm/session.py", line 2678, in _flush
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.868434293Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.868434293Z     transaction.rollback(_capture_exception=True)
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.868565506Z     compat.raise_(
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.868565506Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.868689383Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/orm/session.py", line 2638, in _flush
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.868689383Z     raise exception
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.869301912Z     flush_context.execute()
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.869301912Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.869594987Z     rec.execute(self)
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.869594987Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.869837203Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 239, in save_obj
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.869837203Z     persistence.save_obj(
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.869962047Z     _emit_insert_statements(
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.869962047Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1135, in _emit_insert_statements
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.870270065Z     result = cached_connections[connection].execute(
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.870270065Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.870564513Z     return meth(self, multiparams, params)
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.870564513Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.870722939Z     return connection._execute_clauseelement(self, multiparams, params)
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.870722939Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.871019209Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.871019209Z     ret = self._execute_context(
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.882580835Z     self._handle_dbapi_exception(
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.882580835Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.882580835Z     util.raise_(
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.882580835Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.882580835Z     raise exception
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.882580835Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.882580835Z     return session.query(cls).filter_by(**kwargs).one(), True
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.882580835Z   File "/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.882580835Z     cursor.execute(statement, parameters)
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.882580835Z sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "python_package_version_package_name_package_version_python__key"
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.882580835Z DETAIL:  Key (package_name, package_version, python_package_index_id, os_name, os_version, python_version)=(selinon, 1.0.0, 1, rhel, 8, 3.6) already exists.
solver-rhel-8-py36-c219b7045b32333b9efc8a8fe2e4b9ce-1634229757: 2021-01-14T11:59:29.882580835Z
pacospace commented 3 years ago

We need to modify the integration test accordingly or the endpoint we use. @goern @fridex

if it is run on newly created database from init-job, this test would not fail.

fridex commented 3 years ago

I wonder if this behavior is expected though. If we would want to rerun the solver with the same configuration for some reason (e.g. fix in solver), the most recent solver results should be taken into account in adviser, thus this syncing should proceed even if the given package was already seen in the previous solver runs.

pacospace commented 3 years ago

I wonder if this behavior is expected though. If we would want to rerun the solver with the same configuration for some reason (e.g. fix in solver), the most recent solver results should be taken into account in adviser, thus this syncing should proceed even if the given package was already seen in the previous solver runs.

In that case, shouldn't we just force the sync?

fridex commented 3 years ago

In that case, shouldn't we just force the sync?

The duplicate key violation on unique constraint set up in the database will raise this exception regardless of force sync used in our code.

pacospace commented 3 years ago

In that case, shouldn't we just force the sync?

The duplicate key violation on unique constraint set up in the database will raise this exception regardless of force sync used in our code.

yeah right sorry :)

goern commented 3 years ago

/assign @pacospace

fridex commented 3 years ago

The scenario succeeds, all the issues stated are fixed now. Closing, feel free to reopen if you spot any other issue.