apache / shardingsphere

Distributed SQL transaction & query engine for data sharding, scaling, encryption, and more - on any database.
Apache License 2.0
19.87k stars 6.73k forks source link

Configuration property name 'spring.shardingsphere.rules.sharding.sharding-algorithms.XXX.props' is not valid #8300

Closed DishJi closed 3 years ago

DishJi commented 3 years ago

Question Caused by: org.springframework.boot.context.properties.source.InvalidConfigurationPropertyNameException: Configuration property name 'spring.shardingsphere.rules.sharding.sharding-algorithms.user_record_standard.props' is not valid

Version Springboot2.2.9 | mybatis-plus| shardingsphere-jdbc-core-spring-boot-starter 5.0.0-alpha

Config

spring:
    shardingsphere:
    rules:
      sharding:
        default-data-source-name: ds0
        tables:
          user_record:
            actual-data-nodes: ds0.user_record,ds0.user_record_$->{2019..2022}
            table-strategy:
              standard:
                sharding-column: user_id
                sharding-algorithm-name: user_record_standard
        sharding-algorithms:
          user_record_standard:
            type: UserRecordShardingAlgorithm
            props:
              allow-range-query-with-record-sharding: false

Exception

Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
    at org.springframework.util.ReflectionUtils.rethrowRuntimeException(ReflectionUtils.java:147)
    at org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:810)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:325)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)
    at cn.wx.provider.SmartlearningApplication.main(SmartlearningApplication.java:19)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.shardingsphere.spring.boot.util.PropertyUtil.v2(PropertyUtil.java:109)
    at org.apache.shardingsphere.spring.boot.util.PropertyUtil.handle(PropertyUtil.java:75)
    at org.apache.shardingsphere.spring.boot.registry.AbstractAlgorithmProvidedBeanRegistry.lambda$registerBean$1(AbstractAlgorithmProvidedBeanRegistry.java:57)
    at java.lang.Iterable.forEach(Iterable.java:75)
    at org.apache.shardingsphere.spring.boot.registry.AbstractAlgorithmProvidedBeanRegistry.registerBean(AbstractAlgorithmProvidedBeanRegistry.java:55)
    at org.apache.shardingsphere.sharding.spring.boot.algorithm.ShardingAlgorithmProvidedBeanRegistry.postProcessBeanDefinitionRegistry(ShardingAlgorithmProvidedBeanRegistry.java:38)
    at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:280)
    at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:126)
    at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:707)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:533)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
    ... 3 more
Caused by: org.springframework.boot.context.properties.source.InvalidConfigurationPropertyNameException: Configuration property name 'spring.shardingsphere.rules.sharding.sharding-algorithms.user_record_standard.props' is not valid
    at org.springframework.boot.context.properties.source.ConfigurationPropertyName.elementsOf(ConfigurationPropertyName.java:522)
    at org.springframework.boot.context.properties.source.ConfigurationPropertyName.elementsOf(ConfigurationPropertyName.java:499)
    at org.springframework.boot.context.properties.source.ConfigurationPropertyName.of(ConfigurationPropertyName.java:490)
    at org.springframework.boot.context.properties.source.ConfigurationPropertyName.of(ConfigurationPropertyName.java:478)
    at org.springframework.boot.context.properties.bind.Binder.bind(Binder.java:199)
    at org.springframework.boot.context.properties.bind.Binder.bind(Binder.java:186)
    ... 21 more
DishJi commented 3 years ago

sharding algorithm code

`public class UserRecordShardingAlgorithm implements StandardShardingAlgorithm {

private UserService userService = null;

private static final String REFER = "2019-01-02";

private boolean allowRangeQuery;

private Properties props = new Properties();

private boolean isAllowRangeQuery() {
    return Boolean.parseBoolean(this.props.getOrDefault("allow-range-query-with-record-sharding", Boolean.FALSE.toString()).toString());
}

@Override
public Collection<String> doSharding(Collection<String> collection, RangeShardingValue<Integer> rangeShardingValue) {
    if (this.allowRangeQuery) {
        return collection;
    } else {
        throw new UnsupportedOperationException("Since the property of `allow-range-query-with-user_record-sharding` is false, inline sharding algorithm can not tackle with range query.");
    }
}

@SneakyThrows
@Override
public String doSharding(Collection<String> collection, PreciseShardingValue<Integer> preciseShardingValue){
    //collection:["t_order_1","t_order_2"],preciseShardingValue:{"logicTableName":"t_order","columnName":"order_id","value":396416249350848512}
    //name为两张订单表 t_order_1 和 t_order_2
    if(null == userService){
        synchronized (UserRecordShardingAlgorithm.class){
            if(null == userService){
                userService = (UserService) SpringTools.getBeanByClass(UserServiceImpl.class);
            }
        }
    }
    String tableName = preciseShardingValue.getLogicTableName();
    int userId = preciseShardingValue.getValue();
    User user = userService.queryById(userId);

    Date loginTime = user.getLogintime();

    SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
    //参考时间
    Date referTime = sdf.parse(REFER);

    if (loginTime.after(referTime)) {
        Calendar loginCal = Calendar.getInstance();
        //设置登陆时间
        loginCal.setTime(loginTime);
        //获取登陆年份
        int year = loginCal.get(Calendar.YEAR);
        StringBuilder sb = new StringBuilder(tableName);
        sb.append("_");
        sb.append(year);
        return sb.toString();
    }
    return tableName;
}

@Override
public void init() {
    this.allowRangeQuery = this.isAllowRangeQuery();
}

@Override
public String getType() {
    return "UserRecordShardingAlgorithm";
}

@Override
public Properties getProps() {
    return props;
}

@Override
public void setProps(Properties props) {
    this.props = props;
}

}`

terrymanu commented 3 years ago

Did you add UserRecordShardingAlgorithm into SPI?

luckleemon commented 3 years ago

how to add UserRecordShardingAlgorithm into SPI?

thank you.

hope your reply.

DishJi commented 3 years ago

how to add UserRecordShardingAlgorithm into SPI?

thank you.

hope your reply.

image image

Is this to add SPI?

It still doesn't work properly,thanks @terrymanu @luckleemon

DishJi commented 3 years ago

image image

This is the SPI test result

terrymanu commented 3 years ago

Please summit your source code into a github repo, I will have a look

DishJi commented 3 years ago

@terrymanu

Sorry, because of my log settings, the log is not printed completely, and the following content is missing image

So, when I changed user_record_standard to user-record-standard, the error no longer reappeared. But another error occurred

Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
    at org.springframework.util.ReflectionUtils.rethrowRuntimeException(ReflectionUtils.java:147)
    at org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:810)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:325)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)
    at cn.wx.provider.SmartlearningApplication.main(SmartlearningApplication.java:19)

Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.shardingsphere.spring.boot.util.PropertyUtil.v2(PropertyUtil.java:111)
    at org.apache.shardingsphere.spring.boot.util.PropertyUtil.handle(PropertyUtil.java:75)
    at org.apache.shardingsphere.spring.boot.registry.AbstractAlgorithmProvidedBeanRegistry.registerBean(AbstractAlgorithmProvidedBeanRegistry.java:50)
    at org.apache.shardingsphere.sharding.spring.boot.algorithm.KeyGenerateAlgorithmProvidedBeanRegistry.postProcessBeanDefinitionRegistry(KeyGenerateAlgorithmProvidedBeanRegistry.java:38)
    at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:280)
    at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:126)
    at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:707)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:533)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
    ... 3 more

Caused by: java.util.NoSuchElementException: No value bound
    at org.springframework.boot.context.properties.bind.BindResult.get(BindResult.java:56)
    ... 19 more

I saw other same issues, and changed the config as suggested

image

But did not solve the problem

terrymanu commented 3 years ago

This maybe spring boot compatibility problem, I just assign to @yu199195

yu199195 commented 3 years ago

@DishJi hi,can you show you full config , i will give you some advice 。

DishJi commented 3 years ago

bootstrap.yml

server:
  port: 2004
  servlet:
    context-path: /smartlearning
    encoding:
      charset: UTF-8
      force: true
      enabled: true
  tomcat:
    uri-encoding: UTF-8
spring:
  profiles:
    active: dev
  application:
    name: provider-smartlearning
  shardingsphere:
    rules:
      sharding:
        default-data-source-name: ds0
        tables:
          user_record:
            actual-data-nodes: ds0.user_record,ds0.user_record_$->{2019..2022}
            table-strategy:
              standard:
                sharding-column: user_id
                sharding-algorithm-name: user-record-standard
        sharding-algorithms:
          user-record-standard:
            type: UserRecordShardingAlgorithm
            props:
              allow-range-query-with-record-sharding: false

  #表示不加入默认的静态资源路径处理,解决返回404默认页面问题
  resources:
    addMappings: false
    #让其抛出异常,这样我们就能处理这个异常,解决返回404默认页面问题
  mvc:
    throw-exception-if-no-handler-found: true

mybatis-plus:
  mapper-locations: classpath:mapper/smartlearning/*.xml
  configuration:
    #log-impl: org.apache.ibatis.logging.stdout.StdOutImpl
    map-underscore-to-camel-case: false

application-dev.yml

spring:
  shardingsphere:
    datasource:
      common:
        type: com.alibaba.druid.pool.DruidDataSource
        driver-class-name: com.mysql.jdbc.Driver
        username:
        password:
      names: ds0
      ds0:
        jdbc-url:

@yu199195 This is my full config.If it is not detailed enough, please refer to https://github.com/DishJi/sharding-demo.Thank you for your help!

yu199195 commented 3 years ago

@DishJi hi you also add this config . but It looks like your configuration is correct. user-record-standard is good ,user_record_standard is error.

spring.shardingsphere.rules.sharding.sharding-algorithms.user-record-standard.props.workId=123

and this type UserRecordShardingAlgorithm register SPI ?

DishJi commented 3 years ago

image image

This is the SPI test result

image

@yu199195 Hi is this the correct registration to SPI? And I added the workId parameter, but it did not solve the problem. Thanks

yu199195 commented 3 years ago

@DishJi i will fork you code ,help you solve the problem ,we will fix it ,see https://github.com/apache/shardingsphere/pull/8329

yu199195 commented 3 years ago

@DishJi hi,i debug you project code。 you will add config this:

spring:
  shardingsphere:
    rules:
      sharding:
        key-generators:
          snowflake:
            type: SNOWFLAKE
            props:
              work-id: 123

and if you config datasource by druid, the ds0 not config jdbc-url, is url。