Thoth error when installing dependencies

[Gkrumbach07]

Describe the bug I get an error when I try to install dependencies from the UI under manage dependencies

To Reproduce

1. Create a new notebook
2. select manage dependencies button
3. type pandas and save
4. keep default options
5. install
6. error

Screenshots

Environment information

• jupyterlab version: 3.2.8
• jupyterlab-requirements version: v0.14.0
• Installation source: PyPi
• Operating system: OSX 11.6.1
• Environment type:local

Stack information pandas = "*"

[pacospace]

Thanks @Gkrumbach07 for opening this issue. Never tested in on Mac to be honest :)

Can you please provide also your runtime information? they should be automatically discovered and added to the options before using the install button:

can you also check and paste here the error logs in the terminal where you started jupyter lab please? (I run the same example and it worked)

Or if you can open the development panel in the browser (I think is F12 or your keyboard), can you show if there are any related errors? and the state shown when you just open the UI when clicking the dependency button in the jupyter notebook?

[Gkrumbach07]

Thoth resolution engine error: Error locking dependencies, check pod logs for more details about the error. (400)
Reason: BAD REQUEST
HTTP response headers: HTTPHeaderDict({'server': 'gunicorn', 'date': 'Fri, 21 Jan 2022 16:41:36 GMT', 'content-type': 'application/problem+json', 'content-length': '161', 'access-control-allow-origin': '*', 'x-thoth-version': '0.33.1', 'x-user-api-service-version': '0.33.1+messaging.0.16.0.storages.0.66.0.common.0.34.0.python.0.16.6', 'set-cookie': '99770cb82864be05282857f803e02327=a9c08162c4b3564e2339ab05883c220d; path=/; HttpOnly; Secure; SameSite=None'})
HTTP response body: b'{\n  "detail": "\'None\' is not of type \'integer\' - \'runtime_environment.hardware.cpu_model\'",\n  "status": 400,\n  "title": "Bad Request",\n  "type": "about:blank"\n}\n'

[pacospace]

Thoth resolution engine error: Error locking dependencies, check pod logs for more details about the error. (400)
Reason: BAD REQUEST
HTTP response headers: HTTPHeaderDict({'server': 'gunicorn', 'date': 'Fri, 21 Jan 2022 16:41:36 GMT', 'content-type': 'application/problem+json', 'content-length': '161', 'access-control-allow-origin': '*', 'x-thoth-version': '0.33.1', 'x-user-api-service-version': '0.33.1+messaging.0.16.0.storages.0.66.0.common.0.34.0.python.0.16.6', 'set-cookie': '99770cb82864be05282857f803e02327=a9c08162c4b3564e2339ab05883c220d; path=/; HttpOnly; Secure; SameSite=None'})
HTTP response body: b'{\n  "detail": "\'None\' is not of type \'integer\' - \'runtime_environment.hardware.cpu_model\'",\n  "status": 400,\n  "title": "Bad Request",\n  "type": "about:blank"\n}\n'

Thoth does not have Mac runtime environment for sure. Nevertheless, this is not the error expected when a runtime environment is not known.

I check pod logs from User-API:

{"name": "connexion.decorators.validation", "levelname": "ERROR", "module": "validation", "lineno": 213, "funcname": "validate_schema", "created": 1642783296.6427915, "asctime": "2022-01-21 16:41:36,642", "msecs": 642.7915096282959, "relative_created": 685315745.511055, "process": 22, "message": "http://khemenu.thoth-station.ninja/api/v1/advise/python?recommendation_type=latest&count=1&origin=git%40github.com%3Athoth-station%2Fjupyterlab-requirements.git&source_type=jupyter_notebook&dev=False&debug=True&force=False validation error: 'None' is not of type 'integer' - 'runtime_environment.hardware.cpu_model'"}
{"name": "sentry_sdk.errors", "levelname": "ERROR", "module": "hub", "lineno": 388, "funcname": "_capture_internal_exception", "created": 1642783296.6437557, "asctime": "2022-01-21 16:41:36,643", "msecs": 643.7556743621826, "relative_created": 685315746.4752197, "process": 22, "message": "Internal error in sentry_sdk\nTraceback (most recent call last):\n  File \"/opt/app-root/lib64/python3.8/site-packages/connexion/decorators/validation.py\", line 210, in validate_schema\n    self.validator.validate(data)\n  File \"/opt/app-root/lib64/python3.8/site-packages/jsonschema/validators.py\", line 353, in validate\n    raise error\njsonschema.exceptions.ValidationError: 'None' is not of type 'integer'\n\nFailed validating 'type' in schema['properties']['runtime_environment']['properties']['hardware']['properties']['cpu_model']:\n    {'example': 6, 'nullable': True, 'type': 'integer'}\n\nOn instance['runtime_environment']['hardware']['cpu_model']:\n    'None'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/opt/app-root/lib64/python3.8/site-packages/sentry_sdk/client.py\", line 210, in _prepare_event\n    new_event = before_send(event, hint or {})\n  File \"/opt/app-root/lib64/python3.8/site-packages/thoth/common/logging.py\", line 201, in before_send_handler\n    if exp_signature == log_record[\"name\"]:\nTypeError: string indices must be integers"}
{"name": "sentry_sdk.errors", "levelname": "INFO", "module": "client", "lineno": 212, "funcname": "_prepare_event", "created": 1642783296.6441314, "asctime": "2022-01-21 16:41:36,644", "msecs": 644.1314220428467, "relative_created": 685315746.8509674, "process": 22, "message": "before send dropped event ({'level': 'error', 'logger': 'connexion.decorators.validation', 'logentry': {'message': \"http://khemenu.thoth-station.ninja/api/v1/advise/python?recommendation_type=latest&count=1&origin=git%40github.com%3Athoth-station%2Fjupyterlab-requirements.git&source_type=jupyter_notebook&dev=False&debug=True&force=False validation error: 'None' is not of type 'integer' - 'runtime_environment.hardware.cpu_model'\", 'params': []}, 'extra': {'validator': 'body', 'asctime': '2022-01-21 16:41:36,642', 'sys.argv': ['/opt/app-root/bin/gunicorn', 'thoth.user_api.openapi_server:app', '--bind=0.0.0.0:8080', '--access-logfile=-', '--config', 'gunicorn.conf.py']}, 'event_id': '9a5d1c55abc4421e94c6b3e100ce8fa4', 'timestamp': '2022-01-21T16:41:36.642975Z', 'breadcrumbs': {'values': []}, 'transaction': '/api/v1.thoth_user_api_api_v1_post_advise_python', 'contexts': {'trace': {'trace_id': '7e22c812ff0f4f51b2c6c40ddc4f0b3f', 'span_id': '89a5bd845519e839', 'parent_span_id': None, 'op': 'http.server', 'description': None, 'tracestate': 'sentry=eyJ0cmFjZV9pZCI6ICI3ZTIyYzgxMmZmMGY0ZjUxYjJjNmM0MGRkYzRmMGIzZiIsICJlbnZpcm9ubWVudCI6ICJhd3MtcHJvZCIsICJyZWxlYXNlIjogIjg3YjQ4Njc0YTY2ODRjYmJjYmIwNWMyN2ZjYzIzZTE1MjdjNDdkMWEiLCAicHVibGljX2tleSI6ICJkMjYxMjYzNjRiYTI0YzkxYWNhNDYwOGM4YTliOTNlNSIsICJ0cmFuc2FjdGlvbiI6ICIvYXBpL3YxLnRob3RoX3VzZXJfYXBpX2FwaV92MV9wb3N0X2FkdmlzZV9weXRob24ifQ'}, 'runtime': {'name': 'CPython', 'version': '3.8.6', 'build': '3.8.6 (default, Jan 22 2021, 11:41:28) \\n[GCC 8.4.1 20200928 (Red Hat 8.4.1-1)]'}}, 'modules': {'zipp': '3.7.0', 'yarl': '1.7.2', 'werkzeug': '1.0.1', 'websocket-client': '1.2.3', 'voluptuous': '0.12.2', 'urllib3': '1.26.8', 'tzlocal': '4.1', 'tzdata': '2021.5', 'typing-extensions': '4.0.1', 'toml': '0.10.2', 'thoth-storages': '0.66.0', 'thoth-ssdeep': '3.4', 'thoth-python': '0.16.6', 'thoth-messaging': '0.16.0', 'thoth-common': '0.34.0', 'thoth-analyzer': '0.1.8', 'swagger-ui-bundle': '0.0.9', 'sqlalchemy': '1.3.24', 'sqlalchemy-utils': '0.38.2', 'six': '1.16.0', 'setuptools': '41.6.0', 'sentry-sdk': '1.5.2', 'semantic-version': '2.8.5', 's3transfer': '0.5.0', 'ruamel.yaml': '0.17.20', 'ruamel.yaml.clib': '0.2.6', 'rsa': '4.8', 'rfc5424-logging-handler': '1.4.3', 'requests': '2.27.1', 'requests-oauthlib': '1.3.0', 'pyyaml': '5.4.1', 'pytz': '2021.3', 'pytz-deprecation-shim': '0.1.0.post0', 'python-string-utils': '1.0.0', 'python-json-logger': '2.0.2', 'python-dateutil': '2.8.2', 'pyrsistent': '0.18.0', 'pyparsing': '3.0.6', 'pyelftools': '0.27', 'pydantic': '1.9.0', 'pycparser': '2.21', 'pyasn1': '0.4.8', 'pyasn1-modules': '0.2.8', 'ptyprocess': '0.7.0', 'psycopg2-binary': '2.9.3', 'protobuf': '3.19.3', 'prometheus-flask-exporter': '0.18.7', 'prometheus-client': '0.12.0', 'pip': '20.3.3', 'pexpect': '4.8.0', 'packaging': '21.3', 'openshift': '0.11.0', 'openapi-spec-validator': '0.3.1', 'openapi-schema-validator': '0.1.6', 'oauthlib': '3.1.1', 'multidict': '5.2.0', 'mock': '4.0.3', 'markupsafe': '2.0.1', 'mako': '1.1.6', 'lxml': '4.7.1', 'kubernetes': '11.0.0', 'jsonschema': '3.2.0', 'jsonformatter': '0.3.1', 'jmespath': '0.10.0', 'jinja2': '2.11.3', 'itsdangerous': '1.1.0', 'isodate': '0.6.1', 'inflection': '0.5.1', 'importlib-resources': '5.4.0', 'importlib-metadata': '4.10.0', 'idna': '3.3', 'gunicorn': '20.1.0', 'grpcio': '1.27.2', 'grpcio-tools': '1.27.2', 'google-auth': '2.3.3', 'frozenlist': '1.2.0', 'flask': '1.1.4', 'flask-script': '2.0.6', 'flask-cors': '3.0.10', 'distro': '1.6.0', 'dictdiffer': '0.9.0', 'delegator.py': '0.1.1', 'daiquiri': '3.0.1', 'connexion': '2.9.0', 'confluent-kafka': '1.8.2', 'clickclick': '20.10.2', 'click': '7.1.2', 'charset-normalizer': '2.0.10', 'cffi': '1.15.0', 'certifi': '2021.10.8', 'cachetools': '4.2.4', 'botocore': '1.23.34', 'boto3': '1.20.34', 'blinker': '1.4', 'beautifulsoup4': '4.6.3', 'backports.zoneinfo': '0.2.1', 'attrs': '21.4.0', 'attrdict': '2.0.1', 'async-timeout': '4.0.2', 'argo-workflows': '3.6.1', 'amun': '0.5.0', 'alembic': '1.7.5', 'aiosignal': '1.2.0', 'aiohttp': '3.8.1', 'aiocontextvars': '0.2.2'}, 'request': {'url': 'http://khemenu.thoth-station.ninja/api/v1/advise/python', 'query_string': 'recommendation_type=latest&count=1&origin=git%40github.com%3Athoth-station%2Fjupyterlab-requirements.git&source_type=jupyter_notebook&dev=False&debug=True&force=False', 'method': 'POST', 'env': {'SERVER_NAME': '0.0.0.0', 'SERVER_PORT': '8080'}, 'headers': {'Accept-Encoding': 'identity', 'Content-Length': '701', 'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'Thamos/1.24.0 (Python 3.8.10; Darwin 20.6.0)', 'Host': 'khemenu.thoth-station.ninja', 'X-Forwarded-Host': 'khemenu.thoth-station.ninja', 'X-Forwarded-Port': '443', 'X-Forwarded-Proto': 'https', 'Forwarded': 'for=75.6.179.172;host=khemenu.thoth-station.ninja;proto=https', 'X-Forwarded-For': ''}, 'data': {'application_stack': {'requirements': '[[source]]\\nurl = \"https://pypi.org/simple\"\\nverify_ssl = true\\nname = \"pypi\"\\n\\n[packages]\\npandas = \"*\"\\n\\n[dev-packages]\\n\\n[requires]\\npython_version = \"3.8\"\\n', 'requirements_lock': ''}, 'runtime_environment': {'name': 'jupyterlab-requirements', 'operating_system': {'name': 'darwin', 'version': '20.6.0'}, 'labels': None, 'hardware': {'cpu_family': 'None', 'cpu_model': 'None', 'gpu_model': None}, 'python_version': '3.8', 'cuda_version': None, 'platform': 'macosx-10.9-x86_64', 'openblas_version': None, 'openmpi_version': None, 'cudnn_version': None}, 'library_usage': {'report': {}, 'version': '0.2.1'}}}, 'release': '87b48674a6684cbbcbb05c27fcc23e1527c47d1a', 'environment': 'aws-prod', 'server_name': 'user-api-8-bvq94', 'sdk': {'name': 'sentry.python', 'version': '1.5.2', 'packages': [{'name': 'pypi:sentry-sdk', 'version': '1.5.2'}], 'integrations': ['aiohttp', 'argv', 'atexit', 'boto3', 'dedupe', 'excepthook', 'flask', 'logging', 'modules', 'sqlalchemy', 'stdlib', 'threading']}, 'platform': 'python', '_meta': {'request': {'headers': {'X-Forwarded-For': {'': {'rem': [['!config', 'x', 0, 12]]}}}, 'data': {'runtime_environment': {'': {'len': 12}}}}}})"}

[pacospace]

@Gkrumbach07 can you please show what you have at cat .local/share/thoth/kernels/jupyterlab-requirements/.thoth.yaml (don't know the equivalent of cat on Mac :D )?

[Gkrumbach07]

host: khemenu.thoth-station.ninja
tls_verify: false
requirements_format: pipenv

runtime_environments:
  - name: ubi8
    operating_system:
      name: ubi
      version: "8"
    python_version: "3.8"
    recommendation_type: latest

managers:
  - name: thoth-advise
    configuration:
      labels: [bot]
  - name: info
  - name: version
    configuration:
      maintainers:
        - fridex
        - goern
        - harshad16
        - pacospace
      assignees:
        - sesheta
      labels: [bot]
      changelog_file: true
      changelog_smart: true

[pacospace]

host: khemenu.thoth-station.ninja
tls_verify: false
requirements_format: pipenv

runtime_environments:
  - name: ubi8
    operating_system:
      name: ubi
      version: "8"
    python_version: "3.8"
    recommendation_type: latest

managers:
  - name: thoth-advise
    configuration:
      labels: [bot]
  - name: info
  - name: version
    configuration:
      maintainers:
        - fridex
        - goern
        - harshad16
        - pacospace
      assignees:
        - sesheta
      labels: [bot]
      changelog_file: true
      changelog_smart: true

mmm don't think this is the one discovered as there is ubi8, this seems the local .thoth.yaml from this repository, can you try thamos discover in a folder without .thoth.yaml please?

[pacospace]

.thoth.yaml created on my machine:

 ✘ fmurdaca@pc-7  ~  cat .thoth.yaml 
# This is  Thoth's configuration file placed in a root of a repo
# (named as .thoth.yaml) used by Thamos CLI as well as by Thoth bots. Please
# adjust values listed below as desired.

# A remote Thoth service to talk to:
host: khemenu.thoth-station.ninja

# Configure TLS verification for communication with remote Thoth instance:
tls_verify: true

# Format of requirements file, supported are "pip" and "pipenv":
requirements_format: pipenv
# A path to overlays directory relative to this configuration file. If null provided, no overlays are used.
overlays_dir: overlays
# Allow or disable managing virtual environment for each overlay.
virtualenv: false

runtime_environments:
  - name: 'fedora-34'
    # Operating system for which the recommendations should be created:
    operating_system:
      name: fedora
      version: '34'
    # Hardware information for the recommendation engine:
    hardware:
      # Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz
      cpu_family: 6
      cpu_model: 142
      gpu_model: null
    # Software configuration of runtime environment:
    python_version: '3.9'
    cuda_version: null
    # Recommendation type - one of:
    #  * testing
    #  * stable
    #  * latest
    #  * performance
    #  * security
    # See https://thoth-station.ninja/recommendation-types/
    recommendation_type: latest
    # Platform used for running the application - corresponds to sysconfig.get_platform() call (e.g. 'linux-x86_64')
    platform: linux-x86_64
    # Additional options:
    openblas_version: null
    openmpi_version: null
    cudnn_version: null
    mkl_version: null
    # Base container image used to run the application.
    base_image: null

#
# Configuration of bots:
#
managers:
  - name: pipfile-requirements
  - name: info
  - name: version
    configuration:
      # A list of maintainers (GitHub or GitLab accounts) of this repository:
      maintainers: []
      # A list of assignees to which the opened pull requests and issues should
      # be assigned to:
      assignees: []
      # Labels for issues and pull requests:
      labels:
        - bot
      # Automatically maintain a changelog file stating features of new
      # releases:
      changelog_file: true
      # Use AI/ML to group messages in a smart way.
      changelog_smart: true
  - name: update
    configuration:
      labels: [bot]

[pacospace]

Actually, I think I found the problem from your screenshot above, the CPU family and CPU model are assigned as "None", not as nulll, because they are not recognized on a Mac.

[Gkrumbach07]

So will this not work on Mac?

[pacospace]

So will this not work on Mac?

can you show me in particular from the browser logs the thoth config section:

we need to identify where that conversion happen and why "None" is not converted in null when moving from dict to json.

[Gkrumbach07]

[pacospace]

I'm afraid the problem could be in thamos then when doing thamos discovery but needs to check more, cc @fridex, https://github.com/thoth-station/thamos/blob/f83cbd7f2e613da327f9b5b2ba0439d3f8a82922/thamos/data/defaultThoth.yaml#L29 as gpu_model is correctly assigned as null value, while the others as 'None'

[fridex]

Should be fixed with thoth-station/thamos#1021.

[pacospace]

Should be fixed with thoth-station/thamos#1021.

Thanks @fridex!