Hangfire 2.0 roadmap

[odinserj]

2021 Update

The list below is more like a dream than a roadmap – it contains significant changes no one except new users would like to see. It's a great list of features for the new product, but I don't see how it can be possible to move forward these items while maintaining backward compatibility, the essential feature in Hangfire.

Programming Model

In short: generalization of a background job, more Task-like semantics for background jobs.

• Pending, Completed, Faulted, Canceled states for regular background jobs.
• Result of an antecedent job is available in continuations.
• Continuations can be run synchronously, i.e. on the same worker without any storage call.
• Recurring jobs (and batches, name will be changed) are regular jobs with just a special processing.
• Continuously-running processes are exposed as regular jobs also.
• Support for Job.WaitAny and Job.WaitAll background job types.
• Queues can be used to isolate different code bases without creating additional storage.
• Internals know nothing about user types, to allow shared monitoring.
• Full async/await programming model support for background jobs via dedicated threads.
• IoC container scope starts much earlier in the processing pipeline.

Goals:

• Even simpler understanding of background jobs due to well-known Task-like states.
• Better composition for jobs with support for results, ability to use map-reduce model.
• Simplified continuously-running processes without need for distributed locks.
• Support for microservices architecture by queue-based isolation.
• More performance for recurring jobs without any iterations.

Storage Model

In short: support for coarse-grained operations, idempotent writes to support eventually-consistent storages.

1. Two general aspects of a storage are message queues and background jobs, other types are removed.
2. Storage transactions are exposed directly to user code to support atomic and batched operations.
3. All storage operations are coarse-grained to support batch processing.
4. Each command in a storage transaction is idempotent to support eventually-consistent storages.
5. Storages that support automatic sharding can use it by implementing custom transaction log.
6. Built-in support for time-based scheduling in message queues for faster delayed jobs.
7. Storage abstractions know nothing about user types, data is serialized before passing to thee storage API.
8. Full support for asynchronous I/O for storages in every query.
9. Binary data and payloads for variable-length fields to reduce payload size.

Goals:

• Better network and storage utilization by using batched operations.
• Get rid of any bottlenecks to support nearly linear scaling for background processing.
• Simplified storage implementations for eventually consistent storages without implementing rollbacks via Two-Phase Commit protocol.
• Better performance for scheduled (time-based) processing.
• Redis Cluster and Cassandra can be utilized with any number of nodes with almost linear processing scale.

[ffMathy]

@odinserj any news on this? Especially the "better performance for scheduled processing" part is super important to us.

[odinserj]

Much faster scheduled jobs with less hops will be possible with the upcoming Hangfire 1.8.0 I'm planning to release at the end of this month. The new version allows to specify a queue explicitly for a background job, and in this case Hangfire will use less state changes for delayed jobs. You can even try to download the 1.8.0-beta3 version and use the new feature in the following way:

BackgroundJob.Schedule("default", () => Console.WriteLine("Hello"), TimeSpan.FromSeconds(15));

[usamanada]

Cassandra storage support is important feature .. we are moving from SQL Server to Cassandra. We don't Redis and will not be also to create a dedicated mongo cluster for HangFire.

We hope to see this feature soon. We love HangFire and don't want change technology after we based many things on it.

[odinserj]

Unfortunately there are no plans to implement Cassandra storage support, and I've also updated the original post, please see the top of the page.