dotnet new
Templates This repo hosts the source for Jet's dotnet new
templates.
These templates focus solely on Consistent Processing using Equinox Stores:
eqxweb
- Boilerplate for an ASP .NET Core 3 Web App, with an associated storage-independent Domain project using Equinox.eqxwebcs
- Boilerplate for an ASP .NET Core 3 Web App, with an associated storage-independent Domain project using Equinox, ported to C#.eqxtestbed
- Host that allows running back-to-back benchmarks when prototyping models using [Equinox]. (https://github.com/jet/equinox), using different stores and/or store configuration parameters.eqxPatterns
- Equinox Skeleton Deciders and Tests implementing various event sourcing patterns:
The following templates focus specifically on the usage of Propulsion
components:
proProjector
- Boilerplate for a Publisher application that
(default) --source cosmos
: an Azure CosmosDb ChangeFeedProcessor (typically unrolling events from Equinox.CosmosStore
stores using Propulsion.CosmosStore
)
-k --parallelOnly
schedule kafka emission to operate in parallel at document (rather than accumulated span of events for a stream) level--source eventStore
: Track an EventStoreDB >= 21.10 instance's $all
feed using the gRPC interface (via Propulsion.EventStoreDb
)
--source sqlStreamStore
: SqlStreamStore
's $all
feed
--source dynamo
-k
adds publishing to Apache Kafka using Propulsion.Kafka
.proConsumer
- Boilerplate for an Apache Kafka Consumer using Propulsion.Kafka
(typically consuming from an app produced with dotnet new proProjector -k
).
periodicIngester
- Boilerplate for a service that regularly walks the content of a source, feeding it into a propulsion projector in order to manage the ingestion process using Propulsion.Feed.PeriodicSource
Propulsion.DynamoStore.Indexer
and Propulsion.DynamoStore.Notifier
The bulk of the remaining templates have a consumer aspect, and hence involve usage of Propulsion
.
The specific behaviors carried out in reaction to incoming events often use `Equinox components
proReactor
- Boilerplate for an application that handles reactive actions ranging from publishing notifications via Kafka (simple, or summarising events through to driving follow-on actions implied by events (e.g., updating a denormalized view of an aggregate)
Input options are:
Propulsion.Cosmos
/Propulsion.DynamoStore
/Propulsion.EventStoreDb
/Propulsion.SqlStreamStore
depending on whether the program is run with cosmos
, dynamo
, es
, sss
arguments--source kafkaEventSpans
: changes source to be Kafka Event Spans, as emitted from dotnet new proProjector --kafka
The reactive behavior template has the following options:
EventStore
or a CosmosDB ChangeFeedProcessor to a Summary form in Cosmos
--blank
: remove sample Ingester logic, yielding a minimal projector--kafka
(without --blank
): adds Optional projection to Apache Kafka using Propulsion.Kafka
(instead of ingesting into a local Cosmos
store). Produces versioned Summary Event feed.--kafka --blank
: provides wiring for producing to Kafka, without summary reading logic etcNOTE At present, checkpoint storage when projecting from EventStore uses Azure CosmosDB - help wanted ;)
feedSource
- Boilerplate for an ASP.NET Core Web Api serving a feed of items stashed in an Equinox.CosmosStore
. See dotnet new feedConsumer
for the associated consumption logic
feedConsumer
- Boilerplate for a service consuming a feed of items served by dotnet new feedSource
using Propulsion.Feed
summaryConsumer
- Boilerplate for an Apache Kafka Consumer using Propulsion.Kafka
to ingest versioned summaries produced by a dotnet new proReactor --kafka
.
trackingConsumer
- Boilerplate for an Apache Kafka Consumer using Propulsion.Kafka
to ingest accumulating changes in an Equinox.Cosmos
store idempotently.
proSync
- Boilerplate for a console app that that syncs events between Equinox.Cosmos
and Equinox.EventStore
stores using the relevant Propulsion
.* libraries, filtering/enriching/mapping Events as necessary.
proArchiver
- Boilerplate for a console app that that syncs Events from relevant Categories from a Hot container and to an associated warm Equinox.Cosmos
stores archival container using the relevant Propulsion
.* libraries.
proPruner
- Boilerplate for a console app that that inspects Events from relevant Categories in an Equinox.Cosmos
store's Hot container and uses that to drive the removal of (archived) Events that have Expired from the associated Hot Container using the relevant Propulsion
.* libraries.
While a Pruner does not consume a large amount of RU capacity from either the Hot or Warm Containers, running one continually is definitely optional; a Pruner only has a purpose when there are Expired events in the Hot Container; running periodically during low-load periods may be appropriate, depending on the lifetime profile of the events in your system
Reducing the traversal frequency needs to be balanced against the primary goal of deleting from the Hot Container: preventing it splitting into multiple physical Ranges.
It is necessary to reset the CFP checkpoint (delete the checkpoint documents, or use a new Consumer Group Name) to trigger a re-traversal if events have expired since the lsat time a traversal took place.
proIndexer
- Derivative of proReactor
template. :pray: @ragiano215
Specific to CosmosDB, though it would be easy to make it support DynamoDB
For applications where the reactions using the same Container, credentials etc as the one being Monitored by the change feed processor (simpler config wiring and less argument processing)
includes full wiring for Prometheus metrics emission from the Handler outcomes
Demonstrates notion of an App
project that hosts common wiring common to a set of applications without having the Domain layer reference any of it.
Implements sync
and snapshot
subcommands to enable updating snapshots and/or keeping a cloned database in sync
eqxShipping
- Example demonstrating the implementation of a Process Manager using Equinox
that manages the enlistment of a set of Shipment
Aggregate items into a separated Container
Aggregate as an atomic operation. :pray: @Kimserey.
Shipment
s cannot be Reserved
, those that have been get Revoked
, and the failure is reported to the callerWatchdog
console app (based on dotnet new proReactor --blank
) responsible for concluding abandoned transaction instances (e.g., where processing is carried out in response to a HTTP request and the Clients fails to retry after a transient failure leaves processing in a non-terminal state).proHotel
)proHotel
- Example demonstrating the implementation of a Process Manager using Equinox
that coordinates the merging of a set of GuestStay
s in a Hotel as a single GroupCheckout
activity that coves the payment for each of the stays selected.
MessageDb
or DynamoDb
.As dictated by the design of dotnet's templating mechanism, consumption is ultimately via the .NET Core SDK's dotnet new
CLI facility and/or associated facilities in Visual Studio, Rider etc.
To use from the command line, the outline is:
dotnet new --list
to view your current list)Use dotnet new
to expand the template in a given directory
dotnet new
s list of available templates so it can be picked up bydotnet new
, Rider, Visual Studio etc.dotnet new -i Equinox.Templates
dotnet new eqxweb -t --help
dotnet new eqxwebcs -t
start readme.md
md -p ../DirectIngester | Set-Location dotnet new proReactor
md -p ../Projector | Set-Location
dotnet new proProjector -k start README.md
md -p ../Consumer | Set-Location dotnet new proConsumer start README.md
md -p ../Ingester | Set-Location dotnet new proReactor --source kafkaEventSpans
md -p ../SummaryProducer | Set-Location dotnet new proReactor --kafka start README.md
md -p ../SummaryProducer | Set-Location dotnet new proReactor --kafka --blank start README.md
SummaryProducer
)md -p ../SummaryConsumer | Set-Location dotnet new summaryConsumer start README.md
md -p ../My.Tools.Testbed | Set-Location
dotnet new eqxtestbed -c -e start README.md
dotnet run -p Testbed -- run -d 1 -f 10000 memory
dotnet run -p Testbed -- run -f 2000 es
dotnet run -p Testbed -- run -d 2 cosmos
md -p ../My.Tools.Sync | Set-Location
dotnet new proSync -m start README.md
md -p ../Shipping | Set-Location dotnet new eqxShipping
md -p ../Indexer | Set-Location dotnet new proIndexer
md -p ../ProHotel | Set-Location dotnet new proHotel
There's integration tests in the repo that check everything compiles before we merge/release
dotnet build build.proj # build Equinox.Templates package, run tests \/
dotnet pack build.proj # build Equinox.Templates package only
dotnet test build.proj -c Release # Test aphabetically newest file in bin/nupkgs only (-c Release to run full tests)
One can also do it manually:
Generate the package (per set of changes you make locally)
a. ensuring the template's base code compiles (see runnable templates concept in dotnet new
docs)
b. packaging into a local nupkg
$ cd ~/dotnet-templates
$ dotnet pack build.proj
Successfully created package '/Users/me/dotnet-templates/bin/nupkg/Equinox.Templates.3.10.1-alpha.0.1.nupkg'.
Test, per variant
(Best to do this in another command prompt in a scratch area)
a. installing the templates into the dotnet new
local repo
$ dotnet new -i /Users/me/dotnet-templates/bin/nupkg/Equinox.Templates.3.10.1-alpha.0.1.nupkg
b. get to an empty scratch area
$ mkdir -p ~/scratch/templs/t1
$ cd ~/scratch/templs/t1
c. test a variant (i.e. per symbol
in the config)
$ dotnet new proReactor -k # an example - in general you only need to test stuff you're actually changing
$ dotnet build # test it compiles
$ # REPEAT N TIMES FOR COMBINATIONS OF SYMBOLS
uninstalling the locally built templates from step 2a:
$ dotnet new -u Equinox.Templates
Wherever possible, the samples strongly type identifiers, particularly ones that might naturally be represented as primitives, i.e. string
etc.
FSharp.UMX
is useful to transparently pin types in a message contract cheaply - it works well for a number of contexts:
There are established conventions documented in Equinox's module Aggregate
overview
All the templates herein attempt to adhere to a consistent structure for the composition root module
(the one containing an Application’s main
), consisting of the following common elements:
type Configuration
Responsible for: Loading secrets and custom configuration, supplying defaults when environment variables are not set
Wiring up retrieval of configuration values is the most environment-dependent aspect of the wiring up of an application's interaction with its environment and/or data storage mechanisms. This is particularly relevant where there is variance between local (development time), testing and production deployments. For this reason, the retrieval of values from configuration stores or key vaults is not managed directly within the module Args
section
The Configuration
type is responsible for encapsulating all bindings to Configuration or Secret stores (Vaults) in order that this does not have to be complected with the argument parsing or defaulting in module Args
module Args
’s Arguments
wrappers should do that as applicable as part of the wireup process)module Args
Responsible for: mapping Environment Variables and the Command Line argv
to an Arguments
model
module Args
fulfils three roles:
argv
to values per argument, providing good error and/or help messages in the case of invalid inputsbuild
or start
functions can use to succinctly wire up the dependencies without needing to touch Argu
, Configuration
, or any concrete Configuration or Secrets storage mechanismsNOTE: there's a medium term plan to submit a PR to Argu extending it to be able to fall back to environment variables where a value is not supplied, by means of declarative attributes on the Argument specification in the DU, including having the --help
message automatically include a reference to the name of the environment variable that one can supply the value through
type Logging
Responsible for applying logging config and setting up loggers for the application
Args.Arguments
or values from it)type Logging() =
[<Extension>]
static member Configure(configuration : LoggingConfiguration, ?verbose) =
configuration
.Enrich.FromLogContext()
|> fun c -> if verbose = Some true then c.MinimumLevel.Debug() else c
// etc.
start
functionThe start
function contains the specific wireup relevant to the infrastructure requirements of the microservice - it's the sole aspect that is not expected to adhere to a standard layout as prescribed in this section.
let start (args : Args.Arguments) =
…
(yields a started application loop)
run
, main
functionThe run
function formalizes the overall pattern. It is responsible for:
run
- any such logic should live within the start
and/or build
functionsint
from run
; let main
define the exit codes in one placelet run args = async {
use consumer = start args
return! consumer.AwaitWithStopOnCancellation()
}
[<EntryPoint>]
let main argv =
try let args = Args.parse EnvVar.tryGet argv
try Log.Logger <- LoggerConfiguration().Configure(verbose=args.Verbose).CreateLogger()
try run args |> Async.RunSynchronously; 0
with e when not (e :? System.Threading.Tasks.TaskCanceledException) -> Log.Fatal(e, "Exiting"); 2
finally Log.CloseAndFlush()
with :? Argu.ArguParseException as e -> eprintfn "%s" e.Message; 1
| e -> eprintf "Exception %s" e.Message; 1
Please don't hesitate to create a GitHub issue for any questions, so others can benefit from the discussion. For any significant planned changes or additions, please err on the side of reaching out early so we can align expectations - there's nothing more frustrating than having your hard work not yielding a mutually agreeable result ;)
See the Equinox repo's CONTRIBUTING section for general guidelines wrt how contributions are considered specifically wrt Equinox.
The following sorts of things are top of the list for the templates:
While there is no rigid or defined limit to what makes sense to add, it should be borne in mind that dotnet new eqx/pro*
is sometimes going to be a new user's first interaction with Equinox and/or [asp]dotnetcore. Hence there's a delicate (and intrinsically subjective) balance to be struck between:
encouraging good design practices
In other words, there's lots of subtlety to what should and shouldn't go into a template - so discussing changes before investing time is encouraged; agreed changes will generally be rolled out across the repo.