microsoft / durabletask-mssql

Microsoft SQL storage provider for Durable Functions and the Durable Task Framework
MIT License
87 stars 32 forks source link

No support for custom schema name #64

Closed moldovangeorge closed 2 years ago

moldovangeorge commented 2 years ago

The existing DB setup/migration capabilities do not offer the possibility to customize the schema name for DTF. So when multiple services are using the same database, you only have the option to use the dt schema in the multitenancy setup, but don't have the option to use a schema per service approach.

Is there a plan for supporting this in the future?

cgillum commented 2 years ago

No plans for this currently. The idea is that the existing login-based multi-tenancy architecture should allow multiple services to share the same database with less overhead. Is separate schemas something you need?

moldovangeorge commented 2 years ago

Each service is independently deployed and is independently managing its DTF dependencies. Is the multi-tenancy architecture built for supporting multiple versions of DTF running on the same schema or it's built just for data segregation between services that have the same DTF version but share the DB resource? If one service will update their SQL provider dependency, the schema will be upgraded, while the other services remain running with the old SqlProvider version. Do you recommend using the multitenancy feature with this setup?

cgillum commented 2 years ago

Yes, the multitenancy feature is designed such that all tenants in a single database have their schema updated together. If you wanted to isolate schema updates, you'd need to put tenants in separate databases (which also enables things like resource isolation, etc.). Are you in a situation where each tenant could simply have its own database?

mercer commented 2 years ago

@cgillum does durabletask-mssql handle migrations?

What happens in this example where service A and service B share schema

  1. service A upgrades durabletask-mssql to a new version
  2. the new durabletask-mssql changes dt schema when DurableTask.SqlServer.SqlOrchestrationService.CreateIfNotExistsAsync is called
  3. service A schema migration is deployed

At this point, neither service A, nor B were upgraded. (A will have the new version of durabletask-mssql, but not yet deployed).

  1. will service A continue to work? (deployment would happend in a matter of minutes)
  2. will service B continue to work? (next version of service B may not be even planned at this moment)
cgillum commented 2 years ago

@mercer the intent is to have all DB schema upgrades be backwards-compatible, allowing old services to continue working with newer schemas. Does this answer your question?

mercer commented 2 years ago

What about forwards-compatible?

Given this scenario:

  1. Service A upgrades schema to v2
  2. Service A gets deployed with v2
  3. Service B still has v1 of library, and downgrades the schema to v1
  4. Service A now has v2 library working with v1 schema
cgillum commented 2 years ago

Downgrading schema versions is not supported. They can only move forward. Service B in your scenario would use the existing v2 schema when deployed.

mercer commented 2 years ago

I see.

But in my scenario service B will execute, as a part of schema migration, so before the service is deployed, DurableTask.SqlServer.SqlOrchestrationService.CreateIfNotExistsAsync, with v1.

What will v1 CreateIfNotExistsAsync() do when it sees v2 schema (upgraded by Service A)?

  1. Silently continue (forwards compatibility)?
  2. Fail (does not accept a newish schema)?
  3. Change anything in dt.Versions?
image
cgillum commented 2 years ago

The v1 CreateIfNotExistsAsync code will do nothing if it sees a newer schema version exists in the database. You can find the implementation for this here:

https://github.com/microsoft/durabletask-mssql/blob/24fdfe6979f5af3d57b4f52aaa3eb30a2ba5038f/src/DurableTask.SqlServer/SqlDbManager.cs#L54-L59

In this snippet, DTUtils.ExtensionVersion represents the schema version included in the currently running service ("V1" in your scenario) and currentSchemaVersion represents the version found in the database (in the [SemanticVersion] table - e.g. V2).

moldovangeorge commented 2 years ago

Thank you for all the info @cgillum, our concerns were settled, will close this issue now 🥇