Cachito does not appear to support npm workspaces

[mturley]

In https://github.com/konveyor/forklift-ui we're attempting to use the npm workspaces feature (new since npm v7) to manage our monorepo in the style of lerna / yarn workspaces but with simplified tooling. Our repo has a pkg subdirectory containing web, api, and qe-tests packages with their own package.json files, and the root package-lock.json file ends up with these lines:

   "node_modules/forklift-ui-api": {
      "resolved": "pkg/api",
      "link": true
    },
    "node_modules/forklift-ui-qe-tests": {
      "resolved": "pkg/qe-tests",
      "link": true
    },
    "node_modules/forklift-ui-web": {
      "resolved": "pkg/web",
      "link": true
    },

When Cachito encounters this lockfile, it fails with the following error:

[2021-12-22 17:56:38,297 cachito.workers.pkg_managers.npm INFO npm._get_deps] The dependency {'version': 'file:pkg/api', 'requires': {'@types/ejs': '^3.0.6', '@types/express': '^4.17.12', '@types/simple-oauth2': '^4.1.0', 'ejs': '^3.1.6', 'express': '^4.17.1', 'get-ssl-certificate': '^2.3.3', 'http-proxy-middleware': '^2.0.0', 'simple-oauth2': '^4.2.0'}} is not from the npm registry
[2021-12-22 17:56:38,297 cachito.workers.tasks.npm ERROR npm.fetch_npm_source] Failed to fetch npm dependencies for request 187413
Traceback (most recent call last):
  File "/src/cachito/workers/tasks/npm.py", line 143, in fetch_npm_source
    package_and_deps_info = resolve_npm(
  File "/src/cachito/workers/pkg_managers/npm.py", line 296, in resolve_npm
    package_and_deps_info = get_package_and_deps(package_json_path, package_lock_path)
  File "/src/cachito/workers/pkg_managers/npm.py", line 216, in get_package_and_deps
    name_to_deps, top_level_replacements = _get_deps(
  File "/src/cachito/workers/pkg_managers/npm.py", line 77, in _get_deps
    nexus_replacement = convert_to_nexus_hosted(name, info)
  File "/src/cachito/workers/pkg_managers/npm.py", line 142, in convert_to_nexus_hosted
    dep_in_nexus = process_non_registry_dependency(dep)
  File "/src/cachito/workers/pkg_managers/general_js.py", line 625, in process_non_registry_dependency
    raise CachitoError(
cachito.errors.CachitoError: The dependency forklift-ui-api@file:pkg/api is hosted in an unsupported location
[2021-12-22 17:56:38,298 cachito.workers.tasks.utils INFO utils.set_request_state] Setting the state of request 187413 to "failed" with the reason "The dependency forklift-ui-api@file:pkg/api is hosted in an unsupported location"
[2021-12-22 17:56:38,666 celery.app.trace ERROR trace._log_error] Task cachito.workers.tasks.npm.fetch_npm_source[8259b05a-abb3-48d4-b425-c5794db985fa] raised unexpected: CachitoError('The dependency forklift-ui-api@file:pkg/api is hosted in an unsupported location')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 405, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 697, in __protected_call__
    return self.run(*args, **kwargs)
  File "/src/cachito/workers/tasks/utils.py", line 145, in task_with_state_check
    return task_fn(*args, **kwargs)
  File "/src/cachito/workers/tasks/npm.py", line 143, in fetch_npm_source
    package_and_deps_info = resolve_npm(
  File "/src/cachito/workers/pkg_managers/npm.py", line 296, in resolve_npm
    package_and_deps_info = get_package_and_deps(package_json_path, package_lock_path)
  File "/src/cachito/workers/pkg_managers/npm.py", line 216, in get_package_and_deps
    name_to_deps, top_level_replacements = _get_deps(
  File "/src/cachito/workers/pkg_managers/npm.py", line 77, in _get_deps
    nexus_replacement = convert_to_nexus_hosted(name, info)
  File "/src/cachito/workers/pkg_managers/npm.py", line 142, in convert_to_nexus_hosted
    dep_in_nexus = process_non_registry_dependency(dep)
  File "/src/cachito/workers/pkg_managers/general_js.py", line 625, in process_non_registry_dependency
    raise CachitoError(
cachito.errors.CachitoError: The dependency forklift-ui-api@file:pkg/api is hosted in an unsupported location

Cachito appears to support file:* version specs for dependencies via setting cachito_npm_file_deps_allowlist in /etc/cachito/celery.py to include these packages, but I don't think we have control over the environment this is running in and I am hoping there is a way to allow this from within our repo. Otherwise any repo that uses npm workspaces would need to have those workspace names listed in this allowlist on the host that is running Cachito. It seems like Cachito could just look at the workspaces array in the lockfile and ignore packages that use those paths in their file: specs as well. From the top of the lockfile:

{
  "name": "forklift-ui",
  "version": "0.0.1",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "name": "forklift-ui",
      "version": "0.0.1",
      "license": "Apache-2.0",
      "workspaces": [
        "pkg/api",
        "pkg/web",
        "pkg/qe-tests"
      ],
      "dependencies": {
...

Ideally, because pkg/api is listed there as a workspace, forklift-ui-api@file:pkg/api should be allowed without being specified in cachito_npm_file_deps_allowlist.

[serg-cymbaluk]

Hello @mturley,

Thanks for the feedback. We are already aware of that problem and going to introduce a support of workspaces soon. Until that time I would suggest to not use the npm v7 feature in package.json.

[mturley]

Thanks @serg-cymbaluk !

[gildub]

Hello @serg-cymbaluk, would you please be able to provide a time frame for the workspace support ? The reason is we have another project release in a couple of sprints and we would like to have the feature for our monorepo structure.

[serg-cymbaluk]

The corresponding changes are already in 'master': https://github.com/containerbuildsystem/cachito/commit/947ea277e78684213be4fbbec013188a3a69012b

[nirzari]

@gildub We plan to release the above mentioned changes on February 22nd.

[gildub]

@seg-cymbaluk, @nirzari,

That's great. Thank you for the update.