[Feature] 支持 JDBC 导入 DATE 数据类型, 并存在自动强制转化成文本问题

homealim2012 commented 2 years ago

Feature Description (功能描述)

执行基于jdbc的数据导入功能，报错：

1、这是设置Date类型类型时报的错误： java.lang.IllegalStateException: Only accept FileSource when convert String value to Date, but got 'com.baidu.hugegraph.loader.source.jdbc.JDBCSource' at com.google.common.base.Preconditions.checkState(Preconditions.java:532) ~[guava-29.0-jre.jar:?] at com.baidu.hugegraph.util.E.checkState(E.java:68) ~[hugegraph-common-2.0.1.jar:2.0.1.0] at com.baidu.hugegraph.loader.util.DataTypeUtil.parseSingleValue(DataTypeUtil.java:140) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.util.DataTypeUtil.convert(DataTypeUtil.java:69) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.builder.ElementBuilder.convertPropertyValue(ElementBuilder.java:213) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.builder.ElementBuilder.addProperties(ElementBuilder.java:151) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.builder.ElementBuilder$VertexPkKVPairs.buildVertices(ElementBuilder.java:550) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.builder.VertexBuilder.build(VertexBuilder.java:59) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.task.ParseTaskBuilder.lambda$buildTask$0(ParseTaskBuilder.java:98) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.task.ParseTaskBuilder$ParseTask.get(ParseTaskBuilder.java:160) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.executeParseTask(HugeGraphLoader.java:289) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.loadStruct(HugeGraphLoader.java:263) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.loadStructs(HugeGraphLoader.java:219) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.loadInputs(HugeGraphLoader.java:201) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.loadInputs(HugeGraphLoader.java:173) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.load(HugeGraphLoader.java:110) [hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.main(HugeGraphLoader.java:73) [hugegraph-loader-0.12.0.jar:?]

2、如果设置成Text类型的，报强制转化的错误 2022-07-01 14:39:39 [main] [ERROR] c.b.h.l.t.ParseTaskBuilder [] - Parse VERTEX error com.baidu.hugegraph.loader.exception.ParseException: The value(key='CCRQ') '2014-03-01 14:00:00.0'(class java.sql.Timestamp) is not match with data type TEXT and can't convert to it at com.baidu.hugegraph.loader.task.ParseTaskBuilder.lambda$buildTask$0(ParseTaskBuilder.java:121) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.task.ParseTaskBuilder$ParseTask.get(ParseTaskBuilder.java:160) [hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.executeParseTask(HugeGraphLoader.java:289) [hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.loadStruct(HugeGraphLoader.java:263) [hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.loadStructs(HugeGraphLoader.java:219) [hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.loadInputs(HugeGraphLoader.java:201) [hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.loadInputs(HugeGraphLoader.java:173) [hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.load(HugeGraphLoader.java:110) [hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.main(HugeGraphLoader.java:73) [hugegraph-loader-0.12.0.jar:?]

所以应该加入对Date类型的支持，使数据导入功能更方便好用。

homealim2012 commented 2 years ago

另外也不支持数字转字符串

2022-07-01 12:14:16 [main] [ERROR] c.b.h.l.u.Printer [] - Failed to load, cause: The target data type must be number java.lang.IllegalStateException: The target data type must be number at com.google.common.base.Preconditions.checkState(Preconditions.java:532) ~[guava-29.0-jre.jar:?] at com.baidu.hugegraph.util.E.checkState(E.java:68) ~[hugegraph-common-2.0.1.jar:2.0.1.0] at com.baidu.hugegraph.loader.util.DataTypeUtil.parseNumber(DataTypeUtil.java:212) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.util.DataTypeUtil.checkDataType(DataTypeUtil.java:319) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.util.DataTypeUtil.parseSingleValue(DataTypeUtil.java:149) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.util.DataTypeUtil.convert(DataTypeUtil.java:69) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.builder.ElementBuilder.convertPropertyValue(ElementBuilder.java:213) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.builder.ElementBuilder.access$500(ElementBuilder.java:58) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.builder.ElementBuilder$VertexPkKVPairs.buildVertices(ElementBuilder.java:536) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.builder.VertexBuilder.build(VertexBuilder.java:59) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.task.ParseTaskBuilder.lambda$buildTask$0(ParseTaskBuilder.java:98) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.task.ParseTaskBuilder$ParseTask.get(ParseTaskBuilder.java:160) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.executeParseTask(HugeGraphLoader.java:289) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.loadStruct(HugeGraphLoader.java:263) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.loadStructs(HugeGraphLoader.java:219) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.loadInputs(HugeGraphLoader.java:201) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.loadInputs(HugeGraphLoader.java:173) ~[hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.load(HugeGraphLoader.java:110) [hugegraph-loader-0.12.0.jar:?] at com.baidu.hugegraph.loader.HugeGraphLoader.main(HugeGraphLoader.java:73) [hugegraph-loader-0.12.0.jar:?]

javeme commented 2 years ago

@homealim2012 欢迎贡献代码。

对于第1个问题，可以考虑改进方法DataTypeUtil.parseSingleValue()：

        else if (dataType.isDate()) {
            String dateFormat = Constants.DATE_FORMAT;
            String timeZone = Constants.TIME_ZONE;
            if (source instanceof FileSource) {
                dateFormat = ((FileSource) source).dateFormat();
                timeZone = ((FileSource) source).timeZone();
            }
            return parseDate(key, value, dateFormat, timeZone);
        }

对于第2个问题，可以考虑在方法DataTypeUtil.parseDate()中增加对java.sql.Timestamp转换的支持：if (value instanceof Timestamp)。

对于第3个问题，看起来是个Bug，需要修复下DataTypeUtil.checkDataType()方法，在if (value instanceof Number)的基础增加dataType的判断value instanceof Number && dataType.isNumber()；并且增强方法DataTypeUtil.parseSingleValue()：

else if (dataType == DataType.TEXT)) {
    if (!(rawValue instanceof String)) {
        value = rawValue.toString();
    }
}

liuxiaocs7 commented 1 year ago

Hi, @simon824, i want to have a try, please assign to me, thanks!

apache / incubator-hugegraph-toolchain