chore: Task Workers Revamp

jsantell commented 8 months ago

Decoupled Worker Tasks

Worker tasks are now abstracted such that routes now send jobs via a JobClient decoupled from where the work is performed. A standalone noosphere_gateway::jobs::worker_queue module provides the work distribution logic, with each processor receiving a GatewayJobContext that provides access to sphere contexts, IPFS, and name resolvers when processing a job.

Worker jobs can now queue up additional tasks on success (IPFS Syndication -> Name Record Publish) (https://github.com/subconsciousnetwork/noosphere/issues/606).
Jobs can be retried upon failure.
Some timeout logic has been implemented, but not yet ready.
Republishing name records is now its own job type. Eliminates the need for the scheduler to resolve the latest name record.

Minimize external queries

GatewayManager and routes were updated to minimize GM queries (https://github.com/subconsciousnetwork/noosphere/issues/753).

In addition to a single, cacheable baseline query for scope, unauthenticated routes now has one less external query, and authenticated routes also has one less external query, with another query moved behind authorization.

Ultimately handler logic should not be changed, other than moving Push handler's queueing of name record publishing behind IPFS syndication.

Caveats/Concerns/Enhancements

We should do some profiling to measure lock contention across workers where appropriate amongst shared resources like KuboClient and the name resolver.
Job processors may need to be rethought in order to better take advantage of retries e.g. classifying a failure between retriables, invalid state/unretriable, and failed-but-ok-to-lazily-try-again-later.
~~Implement timeouts in worker queue.~~ Timeouts now implemented
Ipfs syndication revision parameter now optional (unused? to confirm)
Ensure name record provided to PUSH is still valid after Ipfs syndication completes due to record expiry

github-actions[bot] commented 8 months ago

Test flake analysis

No flakes detected 🎉

status	platform	features	toolchain
🟢	`ubuntu-latest`	`test-kubo,headers`	`stable`
🟢	`ubuntu-latest`	`test-kubo,headers,rocksdb`	`stable`
🟢	`ubuntu-latest`	`test-kubo,headers`	`nightly`
🟢	`macos-13`	`test-kubo,headers,rocksdb`	`stable`
🟢	`macos-13`	`test-kubo,headers`	`stable`
🟢	`windows-latest`	`test-kubo,headers`	`stable`

jsantell commented 8 months ago

Added explicit checks on existence of name records after syncing changes in tests, possibly clearing up a handful of intermittent failures.

Windows builds are having an unrelated issue: https://github.com/subconsciousnetwork/noosphere/pull/805

jsantell commented 8 months ago

Added time out support, recreating the hanging worker thread and attempting to resubmit the job. Of note, the SingleTenantJobClient is set to only 1 attempt for now, until we normalize what an error is in the job processors

jsantell commented 7 months ago

Rebasing on #815

cdata commented 7 months ago

Ipfs syndication revision parameter now optional (unused? to confirm)

IIRC we now syndicate full spheres at a time to IPFS, so it isn't necessary to track this.

jsantell commented 7 months ago

Updates (commit):

Changes/cleanups from feedback
Remove vestigial revision in IpfsSyndication job
Reduce indirection from cloning (e.g. use outer Arcs, not inner)
s/ContextResolver/SphereContextResolver
s/NameResolverExt/NameResolverPoller
s/WorkerQueueThread/WorkerQueueOrchestrator
More documentation for worker_queue
Updated retries to explicitly be "retrying" (defaulting to 0) rather than "number of attempts" (defaulting to 1)

subconsciousnetwork / noosphere