search

chaoss / augur

Python library and web service for Open Source Software Health and Sustainability metrics & data collection. You can find our documentation and new contributor information easily here: https://oss-augur.readthedocs.io/en/main/ and learn more about Augur at our website https://augurlabs.io
https://oss-augur.readthedocs.io/en/main/
MIT License
592 stars 845 forks source link

Libyear Worker Patch Needs #2926

Closed sgoggins closed 1 week ago

sgoggins commented 1 month ago

Libyear Worker

NPM

Traceback (most recent call last):
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/tasks.py", line 12, in process_libyear_dependency_metrics
    deps_libyear_model(logger, repo_git)
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/core.py", line 21, in deps_libyear_model
    generate_deps_libyear_data(logger, repo.repo_id, absolute_repo_path)
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/core.py", line 33, in generate_deps_libyear_data
    deps = get_deps_libyear_data(path,logger)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/libyear_util/util.py", line 142, in get_deps_libyear_data
    current_version = get_npm_current_version(data, dependency['requirement'])
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/libyear_util/npm_libyear_utils.py", line 85, in get_npm_current_version
    return get_lastest_minor(clean_version(requirement), data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/libyear_util/npm_libyear_utils.py", line 50, in get_lastest_minor
    raise KeyError("'versions' key not found")
KeyError: "'versions' key not found"

Python Errors

First One

Traceback (most recent call last):
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/tasks.py", line 12, in process_libyear_dependency_metrics
    deps_libyear_model(logger, repo_git)
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/core.py", line 21, in deps_libyear_model
    generate_deps_libyear_data(logger, repo.repo_id, absolute_repo_path)
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/core.py", line 33, in generate_deps_libyear_data
    deps = get_deps_libyear_data(path,logger)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/libyear_util/util.py", line 109, in get_deps_libyear_data
    dependencies = get_parsed_deps(path,logger)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/libyear_util/util.py", line 52, in get_parsed_deps
    dependency_list = parse_pipfile(file_handle)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/libyear_util/pypi_parser.py", line 80, in parse_pipfile
    return map_dependencies_pipfile(manifest['packages'],'runtime') + map_dependencies_pipfile(manifest['dev-packages'], 'develop')
                                                                                               ~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'dev-packages'

Second One

Traceback (most recent call last):
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/tasks.py", line 12, in process_libyear_dependency_metrics
    deps_libyear_model(logger, repo_git)
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/core.py", line 21, in deps_libyear_model
    generate_deps_libyear_data(logger, repo.repo_id, absolute_repo_path)
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/core.py", line 33, in generate_deps_libyear_data
    deps = get_deps_libyear_data(path,logger)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/libyear_util/util.py", line 109, in get_deps_libyear_data
    dependencies = get_parsed_deps(path,logger)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/libyear_util/util.py", line 64, in get_parsed_deps
    dependency_list = parse_conda(file_handle)
                      ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/libyear_util/pypi_parser.py", line 172, in parse_conda
    if (type(dep) is dict) and dep['pip']:
                               ~~~^^^^^^^
KeyError: 'pip'

Third Error: Dynamic Inline Dict?

Traceback (most recent call last):
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
    self.dialect.do_execute(
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute
    cursor.execute(statement, parameters)
psycopg2.ProgrammingError: can't adapt type 'DynamicInlineTableDict'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/github/augur/augur/application/db/lib.py", line 338, in bulk_insert_dicts
    connection.execute(stmnt)
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1416, in execute
    return meth(
           ^^^^^
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1639, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1848, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1988, in _exec_single_context
    self._handle_dbapi_exception(
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2343, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
    self.dialect.do_execute(
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) can't adapt type 'DynamicInlineTableDict'
[SQL: INSERT INTO augur_data.repo_deps_libyear (repo_id, name, requirement, type, package_manager, current_verion, latest_version, current_release_date, latest_release_date, libyear, tool_source, tool_version, data_source, data_collection_date) VALUES (%(repo_id_m0)s, %(name_m0)s, %(requirement_m0)s, %(type_m0)s, %(package_manager_m0)s, %(current_verion_m0)s, %(latest_version_m0)s, %(current_release_date_m0)s, %(latest_release_date_m0)s, %(libyear_m0)s, %(tool_source_m0)s, %(tool_version_m0)s, %(data_source_m0)s, %(data_collection_date_m0)s), (%(repo_id_m1)s, %(name_m1)s, %(requirement_m1)s, %(type_m1)s, %(package_manager_m1)s, %(current_verion_m1)s, %(latest_version_m1)s, %(current_release_date_m1)s, %(latest_release_date_m1)s, %(libyear_m1)s, %(tool_source_m1)s, %(tool_version_m1)s, %(data_source_m1)s, %(data_collection_date_m1)s), (%(repo_id_m2)s, %(name_m2)s, %(requirement_m2)s, %(type_m2)s, %(package_manager_m2)s, %(current_verion_m2)s, %(latest_version_m2)s, %(current_release_date_m2)s, %(latest_release_date_m2)s, %(libyear_m2)s, %(tool_source_m2)s, %(tool_version_m2)s, %(data_source_m2)s, %(data_collection_date_m2)s) ON CONFLICT (repo_id, name, data_collection_date) DO UPDATE SET repo_id = excluded.repo_id, name = excluded.name, requirement = excluded.requirement, type = excluded.type, package_manager = excluded.package_manager, current_verion = excluded.current_verion, latest_version = excluded.latest_version, current_release_date = excluded.current_release_date, latest_release_date = excluded.latest_release_date, libyear = excluded.libyear, tool_source = excluded.tool_source, tool_version = excluded.tool_version, data_source = excluded.data_source, data_collection_date = excluded.data_collection_date]
[parameters: {'repo_id_m0': 127812, 'name_m0': 'molecule', 'requirement_m0': {'extras': ['docker'], 'version': '~=2.22.0'}, 'type_m0': 'develop', 'package_manager_m0': 'PYPI', 'current_verion_m0': '24.9.0', 'latest_version_m0': '24.9.0', 'current_release_date_m0': '2024-09-12T03:11:32.154311Z', 'latest_release_date_m0': '2024-09-12T03:11:32.154311Z', 'libyear_m0': 0, 'tool_source_m0': 'deps_libyear', 'tool_version_m0': '0.44.3', 'data_source_m0': 'git', 'data_collection_date_m0': '2024-10-14T00:20:42Z', 'repo_id_m1': 127812, 'name_m1': 'ansible', 'requirement_m1': '~=2.8.15', 'type_m1': 'develop', 'package_manager_m1': 'PYPI', 'current_verion_m1': '2.8.15', 'latest_version_m1': '10.5.0', 'current_release_date_m1': '2020-09-01T02:59:58.023918Z', 'latest_release_date_m1': '2024-10-08T18:49:37.999447Z', 'libyear_m1': 4.104109589041096, 'tool_source_m1': 'deps_libyear', 'tool_version_m1': '0.44.3', 'data_source_m1': 'git', 'data_collection_date_m1': '2024-10-14T00:20:42Z', 'repo_id_m2': 127812, 'name_m2': 'ansible-lint', 'requirement_m2': '<4.3.0', 'type_m2': 'develop', 'package_manager_m2': 'PYPI', 'current_verion_m2': '4.2.0rc2', 'latest_version_m2': '24.9.2', 'current_release_date_m2': '2019-12-03T14:22:39.066895Z', 'latest_release_date_m2': '2024-09-19T17:21:22.414380Z', 'libyear_m2': 4.8, 'tool_source_m2': 'deps_libyear', 'tool_version_m2': '0.44.3', 'data_source_m2': 'git', 'data_collection_date_m2': '2024-10-14T00:20:42Z'}]
(Background on this error at: https://sqlalche.me/e/20/f405)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
    self.dialect.do_execute(
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute
    cursor.execute(statement, parameters)
psycopg2.ProgrammingError: can't adapt type 'DynamicInlineTableDict'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/tasks.py", line 12, in process_libyear_dependency_metrics
    deps_libyear_model(logger, repo_git)
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/core.py", line 21, in deps_libyear_model
    generate_deps_libyear_data(logger, repo.repo_id, absolute_repo_path)
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/core.py", line 67, in generate_deps_libyear_data
    bulk_insert_dicts(logger, to_insert, RepoDepsLibyear, ["repo_id","name","data_collection_date"])
  File "/home/ubuntu/github/augur/augur/application/db/lib.py", line 362, in bulk_insert_dicts
    bulk_insert_dicts(logger, first_half, table, natural_keys, return_columns, string_fields, on_conflict_update)
  File "/home/ubuntu/github/augur/augur/application/db/lib.py", line 356, in bulk_insert_dicts
    raise e
  File "/home/ubuntu/github/augur/augur/application/db/lib.py", line 338, in bulk_insert_dicts
    connection.execute(stmnt)
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1416, in execute
    return meth(
           ^^^^^
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1639, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1848, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1988, in _exec_single_context
    self._handle_dbapi_exception(
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2343, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
    self.dialect.do_execute(
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) can't adapt type 'DynamicInlineTableDict'
[SQL: INSERT INTO augur_data.repo_deps_libyear (repo_id, name, requirement, type, package_manager, current_verion, latest_version, current_release_date, latest_release_date, libyear, tool_source, tool_version, data_source, data_collection_date) VALUES (%(repo_id_m0)s, %(name_m0)s, %(requirement_m0)s, %(type_m0)s, %(package_manager_m0)s, %(current_verion_m0)s, %(latest_version_m0)s, %(current_release_date_m0)s, %(latest_release_date_m0)s, %(libyear_m0)s, %(tool_source_m0)s, %(tool_version_m0)s, %(data_source_m0)s, %(data_collection_date_m0)s) ON CONFLICT (repo_id, name, data_collection_date) DO UPDATE SET repo_id = excluded.repo_id, name = excluded.name, requirement = excluded.requirement, type = excluded.type, package_manager = excluded.package_manager, current_verion = excluded.current_verion, latest_version = excluded.latest_version, current_release_date = excluded.current_release_date, latest_release_date = excluded.latest_release_date, libyear = excluded.libyear, tool_source = excluded.tool_source, tool_version = excluded.tool_version, data_source = excluded.data_source, data_collection_date = excluded.data_collection_date]
[parameters: {'repo_id_m0': 127812, 'name_m0': 'molecule', 'requirement_m0': {'extras': ['docker'], 'version': '~=2.22.0'}, 'type_m0': 'develop', 'package_manager_m0': 'PYPI', 'current_verion_m0': '24.9.0', 'latest_version_m0': '24.9.0', 'current_release_date_m0': '2024-09-12T03:11:32.154311Z', 'latest_release_date_m0': '2024-09-12T03:11:32.154311Z', 'libyear_m0': 0, 'tool_source_m0': 'deps_libyear', 'tool_version_m0': '0.44.3', 'data_source_m0': 'git', 'data_collection_date_m0': '2024-10-14T00:20:42Z'}]
(Background on this error at: https://sqlalche.me/e/20/f405)

Fourth One: JSON ISSUE?

Traceback (most recent call last):
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/tasks.py", line 12, in process_libyear_dependency_metrics
    deps_libyear_model(logger, repo_git)
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/core.py", line 21, in deps_libyear_model
    generate_deps_libyear_data(logger, repo.repo_id, absolute_repo_path)
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/core.py", line 33, in generate_deps_libyear_data
    deps = get_deps_libyear_data(path,logger)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/libyear_util/util.py", line 109, in get_deps_libyear_data
    dependencies = get_parsed_deps(path,logger)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/libyear_util/util.py", line 77, in get_parsed_deps
    dependency_list = parse_package_json(file_handle)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/github/augur/augur/tasks/git/dependency_libyear_tasks/libyear_util/npm_parser.py", line 14, in parse_package_json
    manifest = json.load(file_handle)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/__init__.py", line 293, in load
    return loads(fp.read(),
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/__init__.py", line 335, in loads
    raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)",
json.decoder.JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)
sgoggins commented 1 month ago

Full log : 

 Skipping file...
2024-10-15 23:15:35,423 [PID: 1342873] process_libyear_dependency_metrics [get_deps_libyear_data() in util.py:L125] [ERROR]: Could not get current version of dependency for path /mnt/repos/repos/hostedaugur/hosted/273566-github.com-microsoft-sarathi-serve/sarathi-serve.
  Dependency: {'name': 'flashinfer', 'requirement': None, 'type': 'runtime', 'package': 'PYPI'}
2024-10-15 23:15:37,401 [PID: 1342873] process_libyear_dependency_metrics [get_release_date() in pypi_libyear_util.py:L80] [ERROR]: Used release of misc==0.0.0 has no upload time.
2024-10-15 23:16:01,573 [PID: 1342922] process_libyear_dependency_metrics [get_parsed_deps() in util.py:L79] [ERROR]: package.json for repo at path /mnt/repos/repos/hostedaugur/hosted/273831-github.com-microsoft-windows-containers-tutorials/windows-containers-tutorials is missing required key: 'dependencies'
 Skipping file...
2024-10-15 23:16:05,225 [PID: 1342873] process_libyear_dependency_metrics [get_parsed_deps() in util.py:L79] [ERROR]: package.json for repo at path /mnt/repos/repos/hostedaugur/hosted/273330-github.com-microsoft-vscode-essentials/vscode-essentials is missing required key: 'dependencies'
 Skipping file...
2024-10-15 23:16:05,269 [PID: 1342925] process_libyear_dependency_metrics [get_release_date() in pypi_libyear_util.py:L83] [ERROR]: Could not find an entry for version 0.24.0.1
2024-10-15 23:17:03,499 [PID: 1342919] process_libyear_dependency_metrics [get_deps_libyear_data() in util.py:L125] [ERROR]: Could not get current version of dependency for path /mnt/repos/repos/hostedaugur/hosted/267089-github.com-getsentry-snuba/snuba.
  Dependency: {'name': 'index-url', 'requirement': None, 'type': 'runtime', 'package': 'PYPI'}
2024-10-15 23:17:03,606 [PID: 1342922] process_libyear_dependency_metrics [get_deps_libyear_data() in util.py:L146] [ERROR]: Could not get latest version of dependency for path /mnt/repos/repos/hostedaugur/hosted/265385-github.com-sourcegraph-srclib-javascript/srclib-javascript.
  Dependency: {'name': 'find-files-excluding-dirs', 'requirement': 'alexsaveliev/find-files-excluding-dirs', 'type': 'runtime', 'package': 'NPM'}
2024-10-15 23:26:46,498 [PID: 1342906] process_libyear_dependency_metrics [get_parsed_deps() in util.py:L79] [ERROR]: package.json for repo at path /mnt/repos/repos/hostedaugur/hosted/273826-github.com-microsoft-webnn-developer-preview/webnn-developer-preview is missing required key: 'dependencies'
 Skipping file...
IsaacMilarky commented 1 month ago

10/23:

sgoggins commented 3 weeks ago

@IsaacMilarky : I'm still getting a couple of errors: 

    Traceback (most recent call last):   File "/home/sean/github/virtualenv/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 451, in trace_task     R = retval = fun(*args, **kwargs)                  ^^^^^^^^^^^^^^^^^^^^   File "/home/sean/github/virtualenv/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 734, in __protected_call__     return self.run(*args, **kwargs)            ^^^^^^^^^^^^^^^^^^^^^^^^^   File "/home/sean/github/rh-k12/augur/tasks/git/dependency_libyear_tasks/tasks.py", line 12, in process_libyear_dependency_metrics     deps_libyear_model(logger, repo_git)   File "/home/sean/github/rh-k12/augur/tasks/git/dependency_libyear_tasks/core.py", line 21, in deps_libyear_model     generate_deps_libyear_data(logger, repo.repo_id, absolute_repo_path)   File "/home/sean/github/rh-k12/augur/tasks/git/dependency_libyear_tasks/core.py", line 33, in generate_deps_libyear_data     deps = get_deps_libyear_data(path,logger)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/home/sean/github/rh-k12/augur/tasks/git/dependency_libyear_tasks/libyear_util/util.py", line 144, in get_deps_libyear_data     current_version = get_npm_current_version(data, dependency['requirement'])                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/home/sean/github/rh-k12/augur/tasks/git/dependency_libyear_tasks/libyear_util/npm_libyear_utils.py", line 78, in get_npm_current_version     if requirement[0] == '~':        ~~~~~~~~~~~^^^ IndexError: string index out of range
    --
sgoggins commented 3 weeks ago

@IsaacMilarky : another

Traceback (most recent call last):
  File "/home/sean/github/virtualenv/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 468, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/virtualenv/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 379, in on_error
    R = I.handle_error_state(
        ^^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/virtualenv/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 178, in handle_error_state
    return {
           ^
  File "/home/sean/github/virtualenv/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 225, in handle_failure
    task.backend.mark_as_failure(
  File "/home/sean/github/virtualenv/hosted/lib/python3.11/site-packages/celery/backends/base.py", line 177, in mark_as_failure
    self.on_chord_part_return(request, state, exc)
  File "/home/sean/github/virtualenv/hosted/lib/python3.11/site-packages/celery/backends/redis.py", line 539, in on_chord_part_return
    return self.chord_error_from_stack(callback, exc)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/virtualenv/hosted/lib/python3.11/site-packages/celery/backends/base.py", line 309, in chord_error_from_stack
    return backend.fail_from_current_stack(callback.id, exc=exc)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/virtualenv/hosted/lib/python3.11/site-packages/celery/backends/base.py", line 316, in fail_from_current_stack
    self.mark_as_failure(task_id, exc, exception_info.traceback)
  File "/home/sean/github/virtualenv/hosted/lib/python3.11/site-packages/celery/backends/base.py", line 172, in mark_as_failure
    self.store_result(task_id, exc, state,
  File "/home/sean/github/virtualenv/hosted/lib/python3.11/site-packages/celery/backends/base.py", line 528, in store_result
    self._store_result(task_id, result, state, traceback,
  File "/home/sean/github/virtualenv/hosted/lib/python3.11/site-packages/celery/backends/base.py", line 956, in _store_result
    current_meta = self._get_task_meta_for(task_id)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/virtualenv/hosted/lib/python3.11/site-packages/celery/backends/base.py", line 978, in _get_task_meta_for
    meta = self.get(self.get_key_for_task(task_id))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/virtualenv/hosted/lib/python3.11/site-packages/celery/backends/base.py", line 856, in get_key_for_task
    return key_t('').join([
           ^^^^^^^^^^^^^^^^
TypeError: sequence item 1: expected a bytes-like object, NoneType found
sgoggins commented 3 weeks ago 
Traceback (most recent call last):
  File "/home/sean/github/virtualenv/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/virtualenv/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/rh-k12/augur/tasks/git/dependency_libyear_tasks/tasks.py", line 12, in process_libyear_dependency_metrics
    deps_libyear_model(logger, repo_git)
  File "/home/sean/github/rh-k12/augur/tasks/git/dependency_libyear_tasks/core.py", line 21, in deps_libyear_model
    generate_deps_libyear_data(logger, repo.repo_id, absolute_repo_path)
  File "/home/sean/github/rh-k12/augur/tasks/git/dependency_libyear_tasks/core.py", line 33, in generate_deps_libyear_data
    deps = get_deps_libyear_data(path,logger)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/rh-k12/augur/tasks/git/dependency_libyear_tasks/libyear_util/util.py", line 111, in get_deps_libyear_data
    dependencies = get_parsed_deps(path,logger)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/rh-k12/augur/tasks/git/dependency_libyear_tasks/libyear_util/util.py", line 54, in get_parsed_deps
    dependency_list.extend(parse_pipfile(file_handle))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sean/github/rh-k12/augur/tasks/git/dependency_libyear_tasks/libyear_util/pypi_parser.py", line 80, in parse_pipfile
    return map_dependencies_pipfile(manifest['packages'],'runtime') + map_dependencies_pipfile(manifest['dev-packages'], 'develop')
                                                                                               ~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'dev-packages'
sgoggins commented 3 weeks ago 

Traceback (most recent call last):   File "/home/sean/github/virtualenv/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 451, in trace_task     R = retval = fun(*args, **kwargs)                  ^^^^^^^^^^^^^^^^^^^^   File "/home/sean/github/virtualenv/hosted/lib/python3.11/site-packages/celery/app/trace.py", line 734, in __protected_call__     return self.run(*args, **kwargs)            ^^^^^^^^^^^^^^^^^^^^^^^^^   File "/home/sean/github/rh-k12/augur/tasks/git/dependency_libyear_tasks/tasks.py", line 12, in process_libyear_dependency_metrics     deps_libyear_model(logger, repo_git)   File "/home/sean/github/rh-k12/augur/tasks/git/dependency_libyear_tasks/core.py", line 21, in deps_libyear_model     generate_deps_libyear_data(logger, repo.repo_id, absolute_repo_path)   File "/home/sean/github/rh-k12/augur/tasks/git/dependency_libyear_tasks/core.py", line 33, in generate_deps_libyear_data     deps = get_deps_libyear_data(path,logger)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/home/sean/github/rh-k12/augur/tasks/git/dependency_libyear_tasks/libyear_util/util.py", line 111, in get_deps_libyear_data     dependencies = get_parsed_deps(path,logger)                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/home/sean/github/rh-k12/augur/tasks/git/dependency_libyear_tasks/libyear_util/util.py", line 69, in get_parsed_deps     dependency_list.extend(parse_conda(file_handle))                            ^^^^^^^^^^^^^^^^^^^^^^^^   File "/home/sean/github/rh-k12/augur/tasks/git/dependency_libyear_tasks/libyear_util/pypi_parser.py", line 164, in parse_conda     dependencies = contents.get('dependencies', [])                    ^^^^^^^^^^^^ AttributeError: 'list' object has no attribute 'get'
--