dotnetcore / sharding-core

high performance lightweight solution for efcore sharding table and sharding database support read-write-separation .一款ef-core下高性能、轻量级针对分表分库读写分离的解决方案,具有零依赖、零学习成本、零业务代码入侵
https://xuejmnet.github.io/sharding-core-doc/
Apache License 2.0
1.17k stars 171 forks source link

按年分表后,有一个数据的分表字段是明年的,就会报错找不到表 #262

Closed jifengmm closed 11 months ago

jifengmm commented 11 months ago

请问,怎么当有数据是未来的时间的数据时,如何创建未来的表。我数据库是sqlserver

ShardingCore.Exceptions.ShardingCoreException:“sharding key route not match Project.Model.Test.Order -> [CreationTime] -> [2024/12/21 10:52:29] -> sharding key to tail :[2024] -> all tails ->[2021,2022,2023]”

xuejmnet commented 11 months ago

@jifengmm 可以重写掉2个方法


        protected override List<string> CalcTailsOnStart()
        {
            var beginTime = ShardingCoreHelper.GetCurrentMonthFirstDay(GetBeginTime());

            var tails=new List<string>();
            //提前创建表
            var nowTimeStamp =ShardingCoreHelper.GetCurrentMonthFirstDay(DateTime.Now);//修改掉这个比如用未来的时间
            if (beginTime > nowTimeStamp)
                throw new ArgumentException("begin time error");
            var currentTimeStamp = beginTime;
            while (currentTimeStamp <= nowTimeStamp)
            {
                var tail = ShardingKeyToTail(currentTimeStamp);
                tails.Add(tail);
                currentTimeStamp = ShardingCoreHelper.GetNextMonthFirstDay(currentTimeStamp);
            }
            return tails;
        }

 public virtual Task ExecuteAsync()
        {
            var logger=RouteShardingProvider
                .GetService<ILogger<AbstractShardingAutoCreateOperatorVirtualTableRoute<TEntity, TKey>>>();
            logger.LogDebug($"get {typeof(TEntity).Name}'s route execute job ");

            var entityMetadataManager = RouteShardingProvider.GetRequiredService<IEntityMetadataManager>();
            var tableCreator = RouteShardingProvider.GetRequiredService<IShardingTableCreator>();
            var virtualDataSource = RouteShardingProvider.GetRequiredService<IVirtualDataSource>();
            var dataSourceRouteManager = RouteShardingProvider.GetRequiredService<IDataSourceRouteManager>();
            var now = DateTime.Now.AddMinutes(IncrementMinutes);//重写掉这个或者重写IncrementMinutes方法属性
            var tail = ConvertNowToTail(now);
//必须先执行AddPhysicTable在进行CreateTable
            Append(tail);
            ISet<string> dataSources = new HashSet<string>();
            if (entityMetadataManager.IsShardingDataSource(typeof(TEntity)))
            {
                var virtualDataSourceRoute = dataSourceRouteManager.GetRoute(typeof(TEntity));
                foreach (var dataSourceName in virtualDataSourceRoute.GetAllDataSourceNames())
                {
                    dataSources.Add(dataSourceName);
                }
            }
.........            
}

以上解决方案是固定未来时间比如未来最多一年的可以这样,如果未来时间或者过期时间不确定的可以用demo里面的另外一个路由 源码项目Sample.AutoCreateIfPresent 要什么加什么你可以按自己的需求改写掉即可

image
jifengmm commented 11 months ago

@xuejmnet 我只改写了这个方法,貌似没问题,你上面回答的第二个方法是做什么的,还需要改写吗?然后比如我程序一直运行,从23年到24年肯定没问题,那么在我程序不重启的情况下,在24年的时候会自动建立25年的数据表吗?还是需要重启一下才会触发这个方法? protected override List CalcTailsOnStart() { DateTime dateTime = new DateTime(GetBeginTime().Year, 1, 1); List list = new List(); DateTime dateTime2 = new DateTime(DateTime.Now.Year + 1, 1, 1);//多加了个1 if (dateTime > dateTime2) { throw new ArgumentException("begin time error"); }

DateTime dateTime3 = dateTime;
while (dateTime3 <= dateTime2)
{
    string item = ShardingKeyToTail(dateTime3);
    list.Add(item);
    dateTime3 = dateTime3.AddYears(1);
}

return list;

}

xuejmnet commented 11 months ago

@jifengmm 那么在我程序不重启的情况下,在24年的时候会自动建立25年的数据表吗? 会创建但是因为是当前时间+10分钟所以在2024年的12月31号晚上23:50分后的定时任务触发了,触发的代码是datetime.now.addminiutes(10)才能知道是2025,后续还是在上个月创建下个月,所以第二个方法的代码还是要在datetime.now处额外addyear(1),两边保持一致这样启动tails和定时的tails都是提前一年往后递推

xuejmnet commented 11 months ago

框架的tails有两个地方会自动添加如果是时间分表默认路由

jifengmm commented 11 months ago

第二个方法没有override关键字,我应该怎么去复写他?

xuejmnet commented 11 months ago

@jifengmm 他是virtual方法啊可以重写的

jifengmm commented 11 months ago

我写了一个这个类,继承了AbstractSimpleShardingYearKeyDateTimeVirtualTableRoute,然后我的表的route继承了我的类,是不是这样就好了? 我的XXX表的路由类: public class FaultDiagnosisRealData_ForUIVirtualTableRoute : YearKeyTableRoute 我改写的复写方法类: public abstract class YearKeyTableRoute: AbstractSimpleShardingYearKeyDateTimeVirtualTableRoute where TEntity : class { protected override List CalcTailsOnStart() { DateTime dateTime = new DateTime(GetBeginTime().Year, 1, 1); List list = new List(); DateTime dateTime2 = new DateTime(DateTime.Now.Year + 1, 1, 1); if (dateTime > dateTime2) { throw new ArgumentException("begin time error"); }

    DateTime dateTime3 = dateTime;
    while (dateTime3 <= dateTime2)
    {
        string item = ShardingKeyToTail(dateTime3);
        list.Add(item);
        dateTime3 = dateTime3.AddYears(1);
    }

    return list;
}

public virtual Task ExecuteAsync()
{
    ILogger<AbstractShardingAutoCreateOperatorVirtualTableRoute<TEntity, DateTime>> service = base.RouteShardingProvider.GetService<ILogger<AbstractShardingAutoCreateOperatorVirtualTableRoute<TEntity, DateTime>>>();
    service.LogDebug("get " + typeof(TEntity).Name + "'s route execute job ");
    IEntityMetadataManager requiredService = base.RouteShardingProvider.GetRequiredService<IEntityMetadataManager>();
    IShardingTableCreator requiredService2 = base.RouteShardingProvider.GetRequiredService<IShardingTableCreator>();
    IVirtualDataSource requiredService3 = base.RouteShardingProvider.GetRequiredService<IVirtualDataSource>();
    IDataSourceRouteManager requiredService4 = base.RouteShardingProvider.GetRequiredService<IDataSourceRouteManager>();
    DateTime now = DateTime.Now.AddYears(1).AddMinutes(IncrementMinutes);
    string text = ConvertNowToTail(now);
    Append(text);
    ISet<string> set = new HashSet<string>();
    if (requiredService.IsShardingDataSource(typeof(TEntity)))
    {
        foreach (string allDataSourceName in requiredService4.GetRoute(typeof(TEntity)).GetAllDataSourceNames())
        {
            set.Add(allDataSourceName);
        }
    }
    else
    {
        set.Add(requiredService3.DefaultDataSourceName);
    }

    service.LogInformation("auto create table data source names:[" + string.Join(",", set) + "]");
    if (AutoCreateTableByTime())
    {
        foreach (string item in set)
        {
            try
            {
                service.LogInformation($"begin table tail:[{text}],entity:[{typeof(TEntity).Name}]");
                requiredService2.CreateTable(item, typeof(TEntity), text);
                service.LogInformation($"succeed table tail:[{text}],entity:[{typeof(TEntity).Name}]");
            }
            catch (Exception exception)
            {
                service.LogInformation($"warning table tail:[{text}],entity:[{typeof(TEntity).Name}]");
                if (DoLogError)
                {
                    service.LogError(exception, item + " " + typeof(TEntity).Name + "'s create table error ");
                }
            }
        }
    }

    return Task.CompletedTask;
}

}