search

apigee / registry

The Registry API allows teams to track and manage machine-readable descriptions of APIs.
https://apigee.github.io/registry
Apache License 2.0
147 stars 33 forks source link

Registry tool: Task (worker pool) improvements #445

Open seaneganx opened 2 years ago

seaneganx commented 2 years ago

Background

The registry tool makes heavy use of the worker pool for concurrent execution of subtasks associated with various commands. Currently if a task returns an error, the worker pool will terminate the command entirely. Some tasks intentionally avoid returning errors to prevent this.

Typically commands will define a specialized struct that implements the Task interface and generate instances of these tasks to handle operations like uploading OpenAPI specs, uploading Discovery specs, uploading Protobuf specs, uploading specs from a CSV row, uploading specs with new annotations, or uploading specs with new labels. Commands that work with a variety of resource types often define one task per resource type, e.g. annotating APIs, annotating API Versions, and annotating API Specs. In other cases tasks are used for computation, executing external applications, and more.

When commands are generating tasks for the worker pool, they're often produced from resources in the registry or from files in one or more directories. When several resource types from the registry are permitted, commands usually determine the argument's resource type before fetching those resources.

Ideas

  1. Configurable worker pools so commands can ignore, log, or immediately halt when tasks return an error. See #435.
  2. Centralized logging for tasks to improve consistency of command output.
  3. Define reusable tasks for common operations like creating/updating resources.
  4. Define reusable task generators for common arguments like files and resource names.
timburks commented 2 years ago

435 addresses the functional issue of controlling worker pool execution. Other improvements seem speculative and might be best to watch until we have more experience.

seaneganx commented 2 years ago

435 addresses the functional issue of controlling worker pool execution.

Right, I remembered the discussion and forgot it was tracked.

timburks commented 2 years ago

Let's consider putting worker pool support (with some or all of these improvements) in a new package under /pkg.