search

containerbuildsystem / cachito

Caching service for source code and external dependencies
GNU General Public License v3.0
56 stars 45 forks source link

readme

Coverage Status

Cachito

Cachito is a service to store (and serve) source code for applications. Upon a request, Cachito will fetch a specific revision of a given repository from the Internet and store it permanently in its internal storage. Namely, it stores the source code for a specific Git commit from a given Git repository, which could be from a forge such as GitHub or GitLab. This way, even if that repository (or that revision) is deleted, it is still possible to track the pristine source code for the original sources. In fact, if the sources have already been previously fetched, Cachito will simply serve the stored copy.

Cachito also supports identifying and permanently storing dependencies for certain package managers and making them available for building the application. Like it does for source code, future requests that utilize these same dependencies will be taken from Cachito's internal storage rather than be fetched from the Internet. See the Package Manager Feature Support section for the package managers that Cachito currently supports.

Cachito will produce bundles as the output artifact of a request. The bundle is a tarball that contains the source code of the application and all the sources of its dependencies. For some package managers, these dependencies can be used directly for building the application. Other package managers will provide an alternative mechanism for this (e.g. a custom npm registry with the declared npm dependencies). Regardless of if the dependencies in the bundle are used for building the application, they are always present so that the source of these dependencies can be published alongside the application for license compliance.

Table of Contents

More Documentation

Documents that outgrew this README can be found in the docs/ drectory.

Coding Standards

The codebase conforms to the style enforced by flake8 with the following exceptions:

In addition to flake8, docstrings are also enforced by the plugin flake8-docstrings with the following exceptions:

The format of the docstrings should be in the reStructuredText style such as:

Set the state of the request using the Cachito API.

:param int request_id: the ID of the Cachito request
:param str state: the state to set the Cachito request to
:param str state_reason: the state reason to set the Cachito request to
:return: the updated request
:rtype: dict
:raise CachitoError: if the request to the Cachito API fails

Additionally, black is used to enforce other coding standards.

To verify that your code meets these standards, you may run tox -e black,flake8.

Quick Start

Run the application locally (requires docker compose):

make run

Note: while running Cachito locally requires docker compose, that does not mean you have to use Docker! Podman 3.0 or greater can serve as a replacement, see https://www.redhat.com/sysadmin/podman-docker-compose.

Alternatively, you could also run the application with podman-compose by setting the CACHITO_COMPOSE_ENGINE variable to the path of the podman-compose script. Unfortunately, the latest release of podman-compose contains various bugs making it unusable for running Cachito locally. Use the script from the devel branch instead. To facilitate this, set CACHITO_COMPOSE_ENGINE to the special value podman-compose-auto. which will instruct the Makefile to download and use the correct version of podman-compose. Be sure to pre-install the dependencies required by podman-compose, currently PyYAML. The script is available in ./tmp/podman_compose.py. You may use this script to interact with the local deployment.

make run CACHITO_COMPOSE_ENGINE=podman-compose-auto

Verify in the browser at http://localhost:8080/

Use curl to make requests:

# List all requests
curl http://localhost:8080/api/v1/requests

# Create a new request
curl -X POST -H "Content-Type: application/json" http://localhost:8080/api/v1/requests -d \
    '{
        "repo": "https://github.com/release-engineering/retrodep.git",
        "ref": "e1be527f39ec31323f0454f7d1422c6260b00580",
        "pkg_managers": ["gomod"]
      }'

# Check the status of a request
curl http://localhost:8080/api/v1/requests/1

# Download the source archive for a completed request
curl http://localhost:8080/api/v1/requests/1/download -o source.tar.gz

Pre-built Container Images

Cachito container images are automatically built when changes are merged. There are two images: an httpd based image with the Cachito API and a Celery worker image with the Cachito worker code.

cachito-api quay.io/containerbuildsystem/cachito-api:latest

cachito-workers quay.io/containerbuildsystem/cachito-workers:latest

Prerequisites

This is built to be used with Python 3.

Some Flask dependencies are compiled during installation, so gcc and Python header files need to be present. For example, on Fedora:

dnf install gcc python3-devel

Development

Virtualenv

You may create a virtualenv with Cachito and its dependencies installed with the following command:

make venv

This installs Cachito in develop mode which allows modifying the source code directly without needing to reinstall Cachito. This is really useful for syntax highlighting in your IDE, however, it's not practical to use as a development environment since Cachito has dependencies on other services.

NOTE: you may need to ensure that you have some packages installed. In Fedora, you will need

yum install python3.11 python3-devel python3-virtualenv gcc krb5-devel

where python3.11 is the version of python required based on tox.ini.

Run a Containerized Development Environment

You may create and run the containerized development environment with docker compose (v2) with the following command:

make run-start

The will automatically create and run the following containers:

After the development environment is running, you can submit jobs to it with curl requests

curl -X POST -H "Content-Type: application/json" http://localhost:8080/api/v1/requests -d \
'{
"repo": "https://github.com/athos-ribeiro/cachito-sample-pip-package.git",
"ref": "51ffb9c2412d50953ed9732c67267e5d2ff9aa68",
"pkg_managers": ["pip"],
"packages": {"pip": [{"path": "."}, {"path": "subpackage"}]}
}'

The REST API and the worker will restart if the source code is modified. Please note that the REST API may stop restarting if there is a syntax error.

Rebuilding Images

If you suspect that the images used for docker compose are out of date, you can run the containerized development environment while forcing a rebuild of the images with the following command:

make run-build-start

If you just want to force a rebuild of the images without running them, you can use

make run-build

Unit Tests

To run the unit tests with tox, you may run the following command:

make test-unit

Integration Tests

To run the integration tests with tox, you may run the following command:

make test-integration

By default, some tests will require custom configuration and will run against your local development environment. Read the integration tests readme for more information.

NOTE: The containerized development environment needs to be running before the integration tests can pass.

Running Specific Tests

Instead of running the entire unit/integration test suite, you can also run a specific set of tests.

make test-suite TOX_ARGS=<test-suite-identifier>

The test-suite-identifier can be pulled from the test result in the tox output or constructed from the filepath filepath and test function. For example, if you want to run test_fetch_gomod_source, you would call:

make test-suite TOX_ARGS=tests/test_workers/test_tasks/test_gomod.py::test_fetch_gomod_source

Omitting the TOX_ARGS will run all tests without performing black/flake8 validation.

In addition to running specific tests, parameters can be passed into tox with TOX_ARGS and the environment can be configured with TOX_ENVLIST.

make test-suite TOX_ARGS="-x --no-cov tests/test_workers/test_tasks/test_gomod.py"

By default, TOX_ENVLIST is set to python3.11 indicating that it should run on that version. If adding environment parameters to tox, ensure that you are setting the Python version if needed.

Clean Up

To remove the virtualenv, built distributions, and the local development environment, you may run the following command:

make clean

If you are using podman, do not forget to set the CACHITO_COMPOSE_ENGINE variable:

make clean CACHITO_COMPOSE_ENGINE=podman-compose

Adding Dependencies

To add more Python dependencies, add them to the following files:

If you're wondering why you need to add dependencies to both files (setup.py and one of the requirements files), see install_requires vs requirements files.

Afterwards, pip-compile the dependencies via make pip-compile (you may need to run make venv first, unless the venv already exists).

Additionally, if any of the newly added dependencies in the generated requirements*.txt files need to be compiled from C code, please install any missing C libraries in the corresponding Dockerfile(s): requirements.txt is used in both, requirements-web.txt only in api.

Accessing Private Repositories

If your Cachito worker needs to access private repositories in your development environment, you may mount a .netrc file by adding the volume mount - /path/to/.netrc:/root/.netrc:ro,z in your docker-compose.yml file under the cachito-worker container.

Using Cachito Requests Locally

More details here.

This is how you would use the example request above locally (assuming it is request #1).

bin/cachito-download.sh localhost:8080/api/v1/requests/1 /tmp/cachito-test

cd /tmp/cachito-test/remote-source/
# sed will sometimes be needed for requests from the dev environment
sed 's/nexus:8081/localhost:8082/g' --in-place cachito.env app/requirements.txt

# you don't *have* to use a container but having a clean environment is usually desirable
podman run --net=host --rm -ti -v "$PWD:/remote-source:z" -w "/remote-source" fedora:33
# <inside the container>
dnf -y install python3-pip
source cachito.env
cd app
pip install -r requirements.txt
python3 setup.py install

You need to have jq installed for the script to work.

Database Migrations

Follow the steps below for database data and/or schema migrations:

Database migration optional data

There are arguments to add migration optional data while upgrading Cachito Database:

Run cachito db upgrade --help to get more info about additional arguments consumed by custom env.py scripts.

API Documentation

The documentation is generated from the API specification written in the OpenAPI 3.0 format.

It is available on Cachito's root URL.

Configuring Workers

To configure a Cachito Celery worker, create a Python file at /etc/cachito/celery.py. Any variables set in this file will be applied to the Celery worker when running in production mode (default).

Custom configuration for the Celery workers are listed below:

To configure the workers to use a Kerberos keytab for authentication, set the KRB5_CLIENT_KTNAME environment variable to the path of the keytab. Additional Kerberos configuration can be made in /etc/krb5.conf.

Configuring the API

Custom configuration for the API:

Additionally, to configure the communication with the Cachito Celery workers, create a Python file at /etc/cachito/celery.py, and set the broker_url configuration to point to your RabbitMQ instance.

If you are planning to deploy Cachito with authentication enabled, you'll need to use a web server that supplies the REMOTE_USER environment variable when the user is properly authenticated. A common deployment option is using httpd (Apache web server) with the mod_auth_gssapi module.

Flags

Nexus

Nexus For Java Script

The Java Script(JS) package managers (npm, yarn) functionality relies on Nexus Repository Manager 3 to store JS dependencies. The Nexus instance will have a JS group repository (e.g. cachito-js) which points to a JS hosted repository (e.g. cachito-js-hosted) and a JS proxy repository (e.g. cachito-js-proxy) that points to the npm/yarn registry (registry.npmjs.org and registry.yarnpkg.com, which points to the same registry server). The hosted repository will contain all non-registry dependencies and the proxy repository will contain all dependencies from the JS registry. The union of these two repositories gives the set of all the JS dependencies ever encountered by Cachito.

On each request, Cachito will create a proxy repository to the JS group repository (e.g. cachito-js). Cachito will populate this proxy repository to contain the subset of dependencies declared in the repository's lock file. Once populated, Cachito will block the repository from getting additional content. This prevents the consumer of the repository from installing something that was not declared in the lock file. This is further enforced by locking down the repository to a single user created for the request, which the consumer will use. Please keep in mind that for this to function properly, anonymous access needs to be disabled on the Nexus instance or at least not set to have read access on all repositories.

These repositories and users created per request are deleted when the request is marked as stale or the request fails.

Nexus For pip

The pip package manager functionality relies on Nexus Repository Manager 3 to store pip dependencies. The Nexus instance will have a PyPI proxy repository (e.g. cachito-pip-proxy) that points to pypi.org and a raw repository (e.g. cachito-pip-raw) which will be used to store external dependencies. The PyPI proxy repository will cache all PyPI packages that Cachito downloads through it and the raw repository will hold tarballs or zip archives of external dependencies that Cachito will upload after fetching them from the original locations.

On each request, Cachito will create a PyPI hosted repository and a raw repository, e.g. cachito-pip-hosted-1 and cachito-pip-raw-1. Cachito will upload all dependencies for the request to these repositories (dependencies from PyPI to the hosted repository, external dependencies to the raw one). Cachito will provide environment variables and configuration files that, when applied to the user's environment, will allow them to install their dependencies from the above-mentioned repositories. When installing dependencies from the Cachito-provided repositories, the user is inherently blocked from installing anything that they did not declare as a dependency, because the repositories will only contain content that Cachito has made available.

These repositories are created per request and deleted when the request is marked as stale or the request fails.

Nexus for RubyGems

The RubyGems package manager functionality relies on Nexus Repository Manager 3 to store RubyGems dependencies. The Nexus instance consists of two repositories that act as a long terms storage - RubyGems proxy repository (e.g. cachito-rubygems-proxy) that points to rubygems.org and raw repository (e.g. cachito-rubygems-raw) used for storing Git dependencies. The RubyGems proxy repository caches all RubyGems packages that Cachito downloads through it and the raw repository holds tarballs of Git dependencies that Cachito uploads there after fetching them from the original locations.

On each request, Cachito creates a RubyGems hosted repository (e.g. cachito-rubygems-hosted-1) and uploads there all GEM dependencies for the request. This repository is created per request and deleted when the request is marked as stale or the request fails. Redirecting Bundler to use this repository instead of a default RubyGems server is done by providing a configuration file. Note that there's no request specific repository for external dependencies as other package managers do, instead, dependencies are installed from the downloaded bundle (see Package Managers section for more details).

When installing dependencies from the Cachito-provided repositories, the user is inherently blocked from installing anything that they did not declare as a dependency, because the repositories will only contain content that Cachito has made available.

Nexus Common Configuration

Refer to the "Configuring Workers" section to see how to configure Cachito to use Nexus. Please note that you may choose to use two Nexus instances. One for hosting the permanent content and the other for the ephemeral repositories created per request. This is useful if your organization already has a shared Nexus instance but doesn't want Cachito to have near admin level access on it. In this case, you will need to configure the following additional settings that point to the Nexus instance that hosts the permanent content: cachito_nexus_hoster_username, cachito_nexus_hoster_password, and cachito_nexus_hoster_url.

Package Managers

Feature Support

The table below shows the supported package managers and their support level in Cachito.

Feature gomod npm pip yarn rubygems
Baseline ✓ ✓ ✓ ✓ ✓
Content Manifest ✓ ✓ ✓ ✓ ✓
Dependency Replacements ✓ x x x x
Dev Dependencies ✓ ✓ ✓ ✓ x
External Dependencies N/A ✓ ✓ ✓ ✓
Multiple Paths ✓ ✓ ✓ ✓ ✓
Nested Dependencies ✓ ✓ x ✓ ✓
Offline Installations ✓ x x x x

Feature Definitions

Current Tool Versions

Tool Version
Go* 1.20.7, 1.23.0 (no workspace vendoring support)
Npm 9.5.0
Node 18.16.1
Pip 22.3.1
Python 3.11.4
Git 2.41.0
Yarn* 1.x
Bundler* 2.x

gomod

The gomod package manager works by parsing the go.mod file present in the source repository to determine which dependencies are required to build the application. By default, the top level module is discovered, but optional paths can be provided to point Cachito to the module(s) to discover.

Cachito then downloads the dependencies through Athens so that they are permanently stored and at the same time create a Go module cache to be stored in the request's bundle.

Cachito will produce a bundle that is downloadable at /api/v1/requests/<id>/download. This bundle will contain the application source code in the app directory and Go module cache of all the dependencies in the deps/gomod directory.

Cachito will provide environment variables in the REST API to set for the Go tooling to use this cache when building the application.

gomod vendoring

When the user enables vendoring mode via the gomod-vendor[-check] flag, Cachito will not build the module cache. The deps/gomod directory will be empty. Instead, the vendored modules will be present in the main module's vendor directory. Check the official documentation about vendoring for more details.

One important thing to note is that only a subset of the module dependency graph will be vendored. As explained in the docs, only modules containing packages needed for building and testing the main module will be present. Commands that expect the entire dependency graph to be available may not work as expected, if at all. Notably, go mod tidy and other go mod commands ignore the vendor directory and instead try to download the modules or access the module cache (which is empty).

Go package level dependencies and the go-package Cachito package type

When reporting Go sources, Cachito differentiates between modules and packages. To simplify a bit, any directory that contains a go.mod file is a module and any directory that contains .go files is a package. A directory that contains both go.mod and .go files is both a module and a package. In Cachito, all packages should have parent modules (or be modules themselves).

In the JSON response at the /api/v1/requests/<id> endpoint, Go modules use the gomod type, Go packages use go-package. Packages can be matched to their parent modules based on name; package names always start with the module name. In the dependencies section of a Go package, Cachito will list only the packages that were imported by that package (a.k.a. package level deps). In the dependencies section of a Go module, Cachito will list all the modules specified as dependencies in go.mod. Submodules allowed to be replaced by a local module by default, no entry required in the cachito_gomod_file_deps_allowlist config variable.

In the Content Manifests shipped at the /api/v1/requests/<id>/content-manifest API endpoint, all top-level purls and the purls of all dependencies refer to Go packages. The purls for the parent Go modules of those dependencies are present in sources.

npm

The npm package manager works by parsing the npm-shrinkwrap.json or package-lock.json file present in the source repository to determine what dependencies are required to build the application.

Cachito then creates an npm registry in an instance of Nexus it manages that contains just the dependencies discovered in the lock file. The registry is locked down so that no other dependencies can be added. The connection information is stored in an .npmrc file accessible at the /api/v1/requests/<id>/configuration-files API endpoint.

Cachito will produce a bundle that is downloadable at /api/v1/requests/<id>/download. This bundle will contain the application source code in the app directory and individual tarballs of all the dependencies in the deps/npm directory. These tarballs are not meant to be used to build the application. They are there for convenience so that the dependency sources can be published alongside your application sources. In addition, they can be used to populate a local npm registry in the event that the application needs to be built without Cachito and the Nexus instance it manages.

Cachito can also handle dependencies that are not from the npm registry such as those directly from GitHub, a Git repository, or an HTTP(S) URL. Please note that if the dependency is from a private repository, set the .netrc and known_hosts files for the Cachito workers. If the dependency location is not supported, Cachito will fail the request. When Cachito encounters a supported location, it will download the dependency, modify the version in the package.json to be unique, upload it to Nexus, modify the top level project's package.json and lock files to use the dependency from Nexus instead. The modified files will be accessible at the /api/v1/requests/<id>/configuration-files API endpoint. If Cachito encounters this same dependency again in a future request, it will use it directly from Nexus rather than downloading it and uploading it again. This guarantees that any dependency used for a Cachito request can be used again in a future Cachito request.

pip

The pip package manager works by parsing the requirements.txt and requirements-build.txt files present in the source repository to determine what dependencies are required to build the application. It is possible to specify different file path(s) for the requirements files as long as the files use the expected format.

Cachito then creates two repositories in an instance of Nexus it manages that contain just the dependencies discovered in the requirements files. PyPI dependencies are uploaded to a PyPI hosted repository, external dependencies are uploaded to a raw repository. Connection information for the hosted repository is provided as the PIP_INDEX_URL environment variable accessible at the /api/v1/requests/<id>/environment-variables endpoint. To make external dependencies available, Cachito modifies the requirements files for the request by replacing relevant entries with their corresponding URLs from the raw repository. The modified requirements files are accessible at the /api/v1/requests/<id>/configuration-files endpoint.

Note that the PIP_INDEX_URL variable exposes the username and password of the temporary user created for your request. This should not be a security concern, the user only has read access for the repositories and the only reason why we do not allow anonymous read access is due to a technical limitation in Nexus.

Cachito will produce a bundle that is downloadable at /api/v1/requests/<id>/download. This bundle will contain the application source code in the app directory and individual source archives of all the dependencies in the deps/pip directory. These archives are not meant to be used to build the application. They are there for convenience so that the dependency sources can be published alongside your application sources. In addition, they can be used to to install packages directly from the filesystem with pip install --no-index --no-deps <path/to/archive> (for each individual source archive) in the event that the application needs to be built without Cachito and the Nexus instance it manages.

As mentioned above, Cachito can also handle dependencies that are not from PyPI, such as those from a Git repository or an HTTP(S) URL. After downloading such a dependency, Cachito will upload it to the Nexus instance used for hosting permanent content. If Cachito encounters this same dependency again in a future request, it will use it directly from Nexus rather than downloading it and uploading it again. This guarantees that any dependency used for a Cachito request can be used again in a future Cachito request.

Compared to gomod and npm, Cachito support for pip has restrictions and limitations that users may not expect. For more details, see the Cachito pip documentation.

git-submodule

With git-submodule as a package manager, Cachito is able to fetch git submodules within given Cachito requested repo and make them available in the Cachito API request response. The git submodules are fetched before any other package managers are processed.

Cachito will produce a bundle that is downloadable at /api/v1/requests/<id>/download. This bundle will contain the application source code in the app directory. When git-submodule is passed as a pkg_managers argument for any Cachito request, the available git submodules within the requested repo will also become available as part of the downloadable bundle. If the repo contains multiple submodules, Cachito will fetch them all. Although, recursion is not supported and hence only one level of submodules will be fetched.

The git submodules information will be included in the Cachito API request response at the /api/v1/requests/<id> endpoint as packages with the git-submodule type.

Finally, the packages information will be used to compose Content Manifests shipped at the /api/v1/requests/<id>/content-manifest API endpoint.

Examples:

curl -X POST -H "Content-Type: application/json" http://localhost:8080/api/v1/requests \
-d '{
      "repo": "https://github.com/nirzari/retrodep.git",
      "ref": "18002daac67f82f1a0f3b1f41beb3469f23116ea",
      "pkg_managers": ["gomod", "git-submodule"]
    }'

In the above case, submodules tour and go-github within specified retrodep repo are fetched as part of the downloadable bundle. They would also be available as packages for Cachito API request response. Further, they become part of the Content Manifest.

If paths to specific git submodules are provided as part of the packages configuration, Cachito would fetch the submodules and then process them as regular packages.

curl "localhost:8080/api/v1/requests" \
    -X POST \
    -H 'content-type: application/json' \
    -d '{
          "repo": "https://github.com/chmeliik/cachito-sample-pip-package/",
          "ref": "1ca07be3001450dbc4f0224e0f763c60353d0f01",
          "pkg_managers": ["git-submodule", "pip", "npm"],
          "packages": {
            "pip": [
              {"path": "cachito-pip-with-deps"}
            ],
            "npm": [
              {"path": "cachito-npm-test"}
            ]
          }
        }'

In the above case, Cachito would fetch the submodules cachito-pip-with-deps, cachito-npm-test and then process them as a regular pip and npm package respectively.

yarn

Cachito handles the yarn package manager in much the same way as the npm package manager. The yarn package manager works by parsing the yarn.lock file present in the source repository to determine what dependencies are required to build the application.

All requests for the yarn package manager with package-lock.json, npm-shrinkwrap.json files in the root directory will fail because those files are dedicated for npm.

After parsing, Cachito creates a yarn registry in an instance of Nexus it manages that contains just the dependencies discovered in the lock file. The registry is locked down so that no other dependencies can be added. The connection information is stored in an .npmrc file accessible at the /api/v1/requests/<id>/configuration-files API endpoint. Cachito also generates a .yarnrc file in the same directory as the .npmrc file, overwriting any existing .yarnrc files if they exist.

Cachito will produce a bundle that is downloadable at /api/v1/requests/<id>/download. This bundle will contain the application source code in the app directory and individual tarballs of all the dependencies in the deps/yarn directory. These tarballs are not meant to be used to build the application. They are there for convenience so that the dependency sources can be published alongside your application sources. In addition, they can be used to populate a local yarn registry in the event that the application needs to be built without Cachito and the Nexus instance it manages.

Cachito can also handle dependencies that are not from the yarn registry such as those directly from GitHub, a Git repository, or an HTTP(S) URL. Please note that if the dependency is from a private repository, set the .netrc and known_hosts files for the Cachito workers. If the dependency location is not supported, Cachito will fail the request. When Cachito encounters a supported location, it will download the dependency, modify the version in the package.json to be unique, upload it to Nexus, modify the top level project's package.json and yarn.lock to use the dependency from Nexus instead. The modified files will be accessible at the /api/v1/requests/<id>/configuration-files API endpoint. If Cachito encounters this same dependency again in a future request, it will use it directly from Nexus rather than downloading it and uploading it again. This guarantees that any dependency used for a Cachito request can be used again in a future Cachito request.

RubyGems (Bundler)

The Bundler package manager works by parsing the Gemfile.lock file present in the source repository to determine what dependencies are required to build the application.

Cachito then creates a RubyGems repository in an instance of Nexus it manages that contains just the GEM dependencies discovered in the lock file. Also, Cachito produces a bundle downloadable at /api/v1/requests/<id>/download containing app/ directory with the application source code (including PATH dependencies) and /deps/rubygems directory with all GEM and GIT dependencies.

Since multiple packages in a single repo are supported, for each of these packages a configuration file is provided at /api/v1/requests/<id>/configuration-files endpoint. This file redirects Bundler to use Nexus proxy for downloading GEM dependencies and contains an entry for every Git dependency to be overridden by the corresponding dependency from deps/rubygems (instead of downloading it from the internet, see local Git repos for more details). If a GIT dependency is specified with branch: in the Gemfile, this branch is checked out so that local GIT repo redirection works.

Note that configuration files expose the username and password of the temporary user created for your request. This should not be a security concern, the user only has read access for the repositories and the only reason why we do not allow anonymous read access is due to a technical limitation in Nexus.

Requirements for RubyGems repos

There are several constraints on RubyGems packages that are enforced by Cachito and not meeting them raises an exception sooner or later:

cachito_rubygems_file_deps_allowlist = {
    "cachito-rubygems-multiple/first_pkg": ["pathgem"] 
}

Note that the name of the package (the key in the dictionary) is the last component of its repo URL. If the package isn't located in the root of the repo, then its /subpath is appended to the name (/first_pkg in the example above). The value in the dictionary is an array of all PATH dependencies of the given package, where the names are parsed from their .gemspec files (= names which are listed in Gemfile.lock).

Using Cachito Without Package Managers

Cachito can be used without specifying a package manager in a request. In that case, only the source code present in the specified commit in a repository will be downloaded and cached.

Even if there are package manager definitions in the source code (such as a package.json or a requirements.txt file), they'll be ignored using this approach. Besides not being cached, the dependencies will also be absent from the content manifest.

This approach can be useful in case there's need to cache and use only the actual source code for that commit, which will then be present in the tarball served by Cachito. Here's how to create a request without package managers:

curl "localhost:8080/api/v1/requests" \
    -X POST \
    -H 'content-type: application/json' \
    -d '{
          "repo": "https://github.com/cachito-testing/cachito-pip-with-deps/",
          "ref": "56efa5f7eb4ff1b7ea1409dbad76f5bb378291e6",
          "pkg_managers": []
        }'

It is important to use an empty array in the pkg_managers key, since omitting it will make Cachito fallback to a default package manager.

By default, the Git history is omitted from the tarball, but it can be included in case the include-git-dir flag is used.