Upgradeable and Backward Compatible

[sbwalker]

As we get closer to the 1.0 release, another area of focus needs to be on reliability and serviceability of the framework. One of the most successful attributes of DNN was that it provided users with an upgrade path from one version to the next and took backward compatibility for APIs very seriously so that it did not affect third party modules. This is how DNN was able to create an ecosystem with hundreds of thousands of consumers and producers - because they could trust the reliability of the core framework itself. Oqtane will also take this approach once it reaches the 1.0 release stage next month. This includes first class support for installing/upgrading modules, themes, and the core framework itself.

One of the pain points with DNN was that although it provided the ability to update the core framework it was far from a seamless process. This was due to a variety of technical reasons related to limitations in the behavior of the .NET Framework and also because the framework was not well encapsulated - it had many different types of disparate assets which were required for the overall solution. Oqtane does not have these limitations... it is built on .NET Core and as much as possible we have tried to keep all assets encapsulated.

Oqtane already contains a seamless upgrade capability. It was developed some time ago but has not been well tested yet because we have not reached the 1.0 milestone where it would be applicable. Basically it will be possible for a host user to visit the Upgrade Service in the Admin Dashboard. They will be able to interactively upload a new Oqtane nuget package or download it from Nuget.org. The framework will extract the contents of the nuget package into the appropriate locations, restart the application and complete the upgrade process automatically. This is the purpose of the Oqtane.Upgrade project which is already in the core repo. This should hopefully make it very straightforward to upgrade Oqtane to new versions - even for non-technical users. This should provide serviceability which is on par with other popular framework/CMS solutions.

The reason I am noting this as an issue is because we need to formalize a 0.9.0 version soon so that we can have a baseline for testing upgrade scenarios. Once 0.9.0 is established, any additional fixes/enhancements would become part of a 0.9.1 release and it should be possible to seamlessly upgrade from 0.9.0 to 0.9.1. This may reveal some unexpected issues which we need to solve now before we get to the 1.0 milestone.

[sbwalker]

One of the challenges I have observed with backward compatibility during development of Oqtane is related to Blazor. The Oqtane framework has dependencies on specific .NET Core packages:

< ItemGroup > < PackageReference Include="Microsoft.AspNetCore.Components.WebAssembly" Version="3.2.0-preview3.20168.3" / > < PackageReference Include="Microsoft.AspNetCore.Components.WebAssembly.Build" Version="3.2.0-preview3.20168.3" PrivateAssets="all" / > < PackageReference Include="Microsoft.AspNetCore.Components.Authorization" Version="3.1.2" / > < PackageReference Include="System.Net.Http.Json" Version="3.2.0-preview3.20175.8" / > < /ItemGroup >

And external modules/themes seem to require dependencies on the same package versions or else there are problems. For example, the current Blog module is build against .NET Core 3.2 Preview3 but when we upgrade the core framework to .NET Core 3.2 Preview4 ( which we should do asap ) I believe the Blog module will no longer function properly unless it is upgraded as well.

I am hoping this is only because of the pre-release nature of Blazor itself - ie, Microsoft is not maintaining backward compatibility for each Preview release. Perhaps once we get to the final Blazor release this will not be a problem.

Does anyone have any thoughts on this topic?

[sbwalker]

Oqtane needs to support install/upgrade for the core framework as well as modules. It is impossible to imagine every requirement a custom module might need so the install/upgrade capability needs to be flexible. Currently Oqtane supports upgrades at the database level. This is definitely part of the overall solution but it is not comprehensive enough to cover all use cases. There will be many install/upgrade scenarios which can only be handled through code ( ie. updating config files, managing files or folders, calling core APIs, etc... ).

One of the first examples which comes to mind is for the core framework. Once 1.0 released there may be a need to add an additional admin page. The SiteTemplate handles this well when creating new sites but it does not handle sites that already exist. There needs to be a mechanism during the upgrade process to call a method based on a version# and execute some code logic.

In order to support this concept we need a few enhancements:

1. We need to track the currently installed version of upgradeable assets. Currently this exists in a few different places, but probably not the right places. At the very least we should be storing the version in the ModuleDefinition table.

2. We need an interface which allows us to execute version specific upgrade/install logic. It would be a simple interface such as:

namespace Oqtane.Infrastructure
{
    public interface IInstallable
    {
        bool Install(string Version);
        bool Uninstall();
    }
}

The framework would use the currently installed version# of an asset in the database ( #1 ) and it would iterate through each subsequent version until it reached the latest version (which is stored internally in the asset itself). Each iteration would execute the Install(version) method and perform whatever logic is needed which is specific to that version.

3. The IModule interface would need an additional property for "ReleaseVersions" which would contain a comma delimited list of every version number of the module which was ever officially released. This list will be used by #2 above to know how to iterate through the versions. The core framework would also introduce a similar constant to track official release versions.

4. Install/Upgrade activities should be logged in the standard logging system for each tenant that they occur in. A new "function" enum value could be added for Install so that it is easy to identify the events.

5. The is the most controversial item on the list, but DBUp does not fit well into the requirements above. It works well for database scripts however it controls the entire upgrade process and does not allow you to execute only a single script for a version. In reality, the internal logic of DBUp basically does the same thing described above in #2 - but is limited to databases. So the question is if we should consider moving away from DBUp in favor of our own install/upgrade logic? Based on the goal to move towards a more code-focused migration approach, it seems to make sense. Also one of the guiding principles is to minimize third party dependencies, so it makes sense from this perspective as well. However, this would be the highest risk change of the items on this list.

6. Uninstall would need to change and leverage the IInstallable interface method. This would provide a looser coupling and a lot more flexibility.

7. Themes are another extensibility point of Oqtane and could also be enhanced to support upgrades in a consistent manner.

[PavelVsl]

ad 5. Most of this things can be done by SQL migrations. And DbUp is very versatile tool, which allows run anything, including custom functions and methods on .net level. take look here: https://dbup.readthedocs.io/en/latest/usage/#code-based-scripts

[sbwalker]

I think what you mean is that EF migrations can handle any database related changes in code - which is true and would seem to be the best long term option as the industry moves away from scripts. I was not aware DBUp had the capability to call code - which is quite interesting - I am just not sure if it’s a good idea to build a critical aspect of the framework around a third party component?

[sbwalker]

I am not opposed to retaining DBUp for now though as it is working fine.

[PavelVsl]

I'm not advocate of DBUP, it's third party component, I'm only pointing to additional functionality and possiblities. And I thing than standard EF migrations would be better.

ad 1. and 3. Central registry of installed versions would be better. Pls do not store comma delimited list.

ad 3. Module should know 1. installed version 2. current version. All upgrade scripts from 0 to module current version should be included in module. It's linear sequence and you need only know where start.

I have bad feeling about doing it at module (theme) level. What about installing class library with themes and modules (eshop, invoicing system, gallery with different views and photo processing modules ....) here is no reason to make every module in separate class. Better is develop it as monolithic library of related classes. This upgrade mechanism should be done at assembly level. And standard assembly version fields should be used.

[PavelVsl]

With EF migrations you can also make no database related changes:

Empty migrations Sometimes it's useful to add a migration without making any model changes. In this case, adding a new migration creates code files with empty classes. You can customize this migration to perform operations that don't directly relate to the EF Core model.

[sbwalker]

I am thinking my earlier post was not clear... and I also did not mention that the approach I explained above is the approach implemented in DNN - which has worked well for almost 20 years now and was quite simple for developers to follow in their modules.

1. I agree that we need a registry to store the versions of assets. This naturally belongs with the assets that the version is associated with. So the module version property should be stored in the ModuleDefinition table as that is the module registry. In DNN we went down the path of trying to make things more generic and having a global assets table but this made all of the code way more complicated.

2. When I was referring to a comma delimited list for ReleaseVersions I was referring to a new property in the IModule interface ( which is based on ModuleDefinition ). This property would not be stored in the database - it is only used by upgrades to know which releasable versions exist for a specific module. This is needed for the exact reason I mentioned in my earlier post and you mentioned above - because not all upgrades require a database script - some upgrades are pure code and have nothing to do with the database. So you cant rely on database scripts to be the mechanism to determine the list of upgrade versions. And you also cant realistically iterate through versions using a an int counter because people use very different numbering schemes ( ie. Windows 10 is now at version 10.0.18363.0 ).

    public class ModuleInfo : IModule
    {
        public ModuleDefinition ModuleDefinition => new ModuleDefinition
        {
            Name = "Some Module",
            Version = "2.0.0",
            ReleaseVersions = "1.0.0,1.1.0,2.0.0"
        };
    }

So the framework would get the ReleaseVersions and split them based on comma and then iterate through them in the order they were specified. This is the most explicit way for a developer to be able to tell the framework about their version release history. Their implementation of the IInstallable interface would look like below ( note that for each version that a developer specifies in the ReleaseVersions list there should be a corresponding entry in the Install() method implementation.

    public class InstallInfo : IInstallable
    {
        public bool Install(string Version)
        {
        switch (Version)
        {
            case "1.0.0":
                // do whatever logic is required for this version and return success/failure
                break;
            case "1.1.0":
                // do whatever logic is required for this version and return success/failure
                break;
            case "2.0.0":
                // do whatever logic is required for this version and return success/failure
                break;
        };
    }

Although EF migrations supports empty migrations it feels like a hack as EF's main focus is on databases - so why would you use something focused on databases to run code not related to databases such as updating config settings, dealing with files/folder, etc... I think EF migrations are the right technology for doing DB schema upgrades in code, but not for other upgrade activities.

I will have to think more about your comment about being able to create a consolidated group of modules. In theory it should be possible to do it now as the core framework is essentially a consolidated group of modules. When you create an external module it is fully possible to create it in such a way that it has multiple modules within the Client project that are compiled into a single assembly for deployment. In the install/upgrade process the framework would need to be able to iterate through the modules that are in an assembly and execute their logic. This is not difficult as the framework already does this for other purposes.

[sbwalker]

@chlupac I have been working through the implementation of the IInstallable interface and it has led me back to your comment above about consolidated groups of modules. It should be possible for someone to create a project which contains multiple modules that are compiled into the same assembly. In fact, the core framework is built this way already - there are a variety of default modules which are part of the Oqtane assembly.

One of the modules which is part of the core framework is the HtmlText module - and it is different from the other Admin modules in the sense that it is an isolated module with its own front-end, back-end, etc... ( it is an example of how to build an Internal module ). However the one aspect which is not currently isolated is the installation - the HtmlText database table definition is embedded in the framework Tenant script. This is not the right approach as it is tightly coupled. The HtmlText module should manage its own installation concerns. This includes having its own database installation scripts and the ability to execute whatever install/upgrade logic it might require. I already have this working locally and it seems to be a much cleaner approach.

Another benefit of this approach is that if someone uses the Internal module template in the ModuleCreator, it is possible for those modules to be compiled into the Oqtane assembly but still manage their own install/upgrade concerns. Also for External modules, it is possible to group a number of modules into a single assembly but allow each module within the assembly to manage their own install/upgrade concerns. Each module within the assembly can even have their own version number as they may be evolved on different schedules. The module is the lowest level of granularity and the assembly is just a container which can contain one or more modules.

[PavelVsl]

I'm thinking about set of attributes to mark-up installable elements on assembly or class level. What do you thing about it? Also here is much more possibilities to query application domain different than assembly name convention..

We should left on developer which level of granularity will suit his implementation. Lot of implementations will have common dbcontext and database implementation.

Instalation (or upgrade) is iteration of steps between currently installed version and installed assembly version. It should ideally work in both ways (upgrade, downgrade).

And at last, I think than installer should be one per assembly (assembly set, nuget).

[sbwalker]

1. You are referring to how Oqtane assemblies are currently identified based on naming convention ( ie. they need to have .Module. or .Theme. in their name ). I am fine with considering an alternate approach as long as it is very simple for developers to specify and it does not result in worse performance than File.IO ( ie. I believe reflection will be much slower ).

2. I agree that it needs to be up to the developer to decide their level of granularity. That being said, we need to build the framework based on the lowest level of granularity. Once we support that level then developers can aggregate or group the elements however they want. The lowest level of granularity which Oqtane needs to support is at the Module level. The assembly level is a higher level grouping which can contain one or more modules depending on a developers preference.

3. Install/upgrade is an iteration of steps between the currently installed version of a module and the new version. It should not be at the assembly level - assembly is a higher level of granularity. Upgrade and downgrade would be nice but I don't feel it is required at this stage ( DNN has never supported downgrade ).

4. A nuget package contains assemblies. There is nothing preventing a nuget package from containing assemblies for one or more modules. The system is simply going to extract the assemblies into the /bin and the framework will complete the installation from there by loading the assemblies and processing their contents.

[PavelVsl]

1. It's one time action during app startup, speed is not critical here.
2. I agree with it.
3. Yes, I do not imagine scenario when downgrade is needed (exception is buggy module version). It's too complicated.
4. I'm overthink this point. In general level -> explore assembly for install/migration classes/scripts -> execute them in correct order.

[hishamco]

I have few notes to add may be some already added, may be not

1. Modules

• Avoid internal module as much as we can, except the one who are shipped with core

• Add Version & RuntimeVersion properties to each module, so this will help us if a particular module is compatible with a certain Oqtane version or not, so the module will be upgraded or downgraded with the respect with the RuntimeVersion aka Oqtane version

• Modules can be installed though packaging system, NuGet for instance

• Modules may have assets or need to add some tables, so this can be done through

a) Coded Migration

b) SQL Migration

Perhaps we can go with the second option for now and avoid 3rd party dependencies will be better

• Need to have a manifest to know which assets have been added. to remove them during uninstall, or we could introduce Up and Down methods for each module, which will allow the module creators have a control on intstalling and uninstalling processes

2. Core

• We could have a nothing of IInstallable to install or uninstall a particular version but this is hard little bit, because the version may have dependencies on other assemblies, IMHO may we can do installing and uninstalling for minor & patch release but not for major once

• Unify the assets location for the runtime version

• Provide a migration mechanism like what I mentioned in modules, provide proper way for SQL migrations to install and uninstall, I'm not sure if the current SQL scripts provide a way to remove the tables