bytedance / bitsail

BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.
https://bytedance.github.io/bitsail/
Apache License 2.0
1.63k stars 335 forks source link

[BitSail][Improve]Add more debugging log for the NativeFlinkTypeInfoUtil. #417

Open hk-lrzy opened 1 year ago

hk-lrzy commented 1 year ago

Description

public class NativeFlinkTypeInfoUtil {
...
  public static RowTypeInfo getRowTypeInformation(List<ColumnInfo> columnInfos,
                                                  TypeInfoConverter typeInfoConverter) {

    String[] fieldNames = new String[columnInfos.size()];
    TypeInformation<?>[] fieldTypes = new TypeInformation[columnInfos.size()];

    for (int index = 0; index < columnInfos.size(); index++) {
      String type = StringUtils.lowerCase(columnInfos.get(index).getType());
      String name = columnInfos.get(index).getName();

      TypeInfo<?> typeInfo = typeInfoConverter.fromTypeString(type);

      fieldNames[index] = name;
      fieldTypes[index] = toNativeFlinkTypeInformation(typeInfo);
    }

    return new RowTypeInfo(fieldTypes, fieldNames);
  }
...
}

TypeInfo<?> typeInfo = typeInfoConverter.fromTypeString(type); and

public class ColumnFlinkTypeInfoUtil {
  public static RowTypeInfo getRowTypeInformation(TypeInfoConverter converter,
                                                  List<ColumnInfo> columnInfos) {

    String[] fieldNames = new String[columnInfos.size()];
    TypeInformation<?>[] fieldTypes = new TypeInformation[columnInfos.size()];
    for (int index = 0; index < columnInfos.size(); index++) {
      String type = StringUtils.lowerCase(columnInfos.get(index).getType());
      String name = columnInfos.get(index).getName();

      TypeInfo<?> typeInfo = converter.fromTypeString(type);
      fieldNames[index] = name;
      fieldTypes[index] = toColumnFlinkTypeInformation(typeInfo);
    }

    return new RowTypeInfo(fieldTypes, fieldNames);
  }
}

TypeInfo<?> typeInfo = converter.fromTypeString(type);

Both the classes have the same issue of lack the validation for the result of the converter.fromTypeString(type) and cause if there return null value the logic will throw NPE. IMO it is not clearly enough for all users to know what's hapened and how to solve it.

Your use scenarios

No response

Your alternatives

No response

Are you willing to submit PR?

Code of Conduct

beyond-up commented 1 year ago

please assign this issue to me, thanks!

hk-lrzy commented 1 year ago

@beyond-up Assigned it to you, look foward to your PR later~

liuxiaocs7 commented 1 year ago

@hk-lrzy @lichang-bd This issue seems can be closed.

swikarpat commented 3 days ago

Is this issue resolved already? I can work on this issue if not done yet.