doris无法加载表结构数据

janfengye commented 2 weeks ago

测试链接是成功的，也能成功保存，就是不能同步表数据 2024-08-26 21:12:23.037 [http-nio-0.0.0.0-2024-exec-11] INFO datart.data.provider.JdbcDataProvider : jdbc source '5ad4570508f544b8930156b8ef95339d-Doris' updated, source has been reset 2024-08-26 21:12:23.043 [pool-2-thread-2] INFO d.data.provider.jdbc.DataSourceFactoryDruidImpl : druid data source created (DataSource-1111789274) 2024-08-26 21:12:23.049 [pool-2-thread-2] INFO com.alibaba.druid.pool.DruidDataSource : {dataSource-6} inited 2024-08-26 21:12:23.063 [http-nio-0.0.0.0-2024-exec-11] WARN com.alibaba.druid.pool.DruidAbstractDataSource : discard long time none received connection. , jdbcUrl : jdbc:mysql://localhost:3333/datart?&allowMultiQueries=true&characterEncoding=utf-8, version : 1.2.4, lastPacketReceivedIdleMillis : 406943 2024-08-26 21:12:23.077 [http-nio-0.0.0.0-2024-exec-11] INFO datart.server.service.impl.SourceServiceImpl : jdbc source schema job has been created 5ad4570508f544b8930156b8ef95339d - Doris - interval 60 2024-08-26 21:12:24.896 [pool-2-thread-2] ERROR datart.server.service.impl.SourceServiceImpl : source schema sync error java.lang.NullPointerException: null at datart.data.provider.jdbc.DataTypeUtils.jdbcType2DataType(DataTypeUtils.java:61) at datart.data.provider.jdbc.adapters.JdbcDataProviderAdapter.readTableColumn(JdbcDataProviderAdapter.java:193) at datart.data.provider.jdbc.adapters.JdbcDataProviderAdapter.readTableColumn(JdbcDataProviderAdapter.java:176) at datart.data.provider.JdbcDataProvider.readTableColumns(JdbcDataProvider.java:85) at datart.data.provider.ProviderManager.readTableColumns(ProviderManager.java:114) at datart.server.service.impl.DataProviderServiceImpl.readTableColumns(DataProviderServiceImpl.java:140) at datart.server.job.SchemaSyncJob.execute(SchemaSyncJob.java:94) at datart.server.service.impl.SourceServiceImpl.lambda$updateJdbcSourceSyncJob$2(SourceServiceImpl.java:464) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-08-26 21:12:24.976 [DatartScheduleCluster_Worker-4] ERROR datart.server.job.SchemaSyncJob : source schema sync error java.lang.NullPointerException: null at datart.data.provider.jdbc.DataTypeUtils.jdbcType2DataType(DataTypeUtils.java:61) at datart.data.provider.jdbc.adapters.JdbcDataProviderAdapter.readTableColumn(JdbcDataProviderAdapter.java:193) at datart.data.provider.jdbc.adapters.JdbcDataProviderAdapter.readTableColumn(JdbcDataProviderAdapter.java:176) at datart.data.provider.JdbcDataProvider.readTableColumns(JdbcDataProvider.java:85) at datart.data.provider.ProviderManager.readTableColumns(ProviderManager.java:114) at datart.server.service.impl.DataProviderServiceImpl.readTableColumns(DataProviderServiceImpl.java:140) at datart.server.job.SchemaSyncJob.execute(SchemaSyncJob.java:94) at datart.server.job.SchemaSyncJob.execute(SchemaSyncJob.java:72) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded 2024-08-26 20:55:52.592 [DatartScheduleCluster_Worker-3] WARN com.alibaba.druid.pool.DruidAbstractDataSource : discard long time none received connection. , jdbcUrl : jdbc:mysql://localhost:3333/datart?&allowMultiQueries=true&characterEncoding=utf-8, version : 1.2.4, lastPacketReceivedIdleMillis : 272659 2024-08-26 20:55:52.604 [DatartScheduleCluster_Worker-3] INFO d.data.provider.jdbc.DataSourceFactoryDruidImpl : druid data source created (DataSource-1830622635) 2024-08-26 20:55:52.608 [DatartScheduleCluster_Worker-3] INFO com.alibaba.druid.pool.DruidDataSource : {dataSource-5} inited 2024-08-26 20:55:53.286 [DatartScheduleCluster_Worker-3] ERROR datart.server.job.SchemaSyncJob : source schema sync error java.lang.NullPointerException: null at datart.data.provider.jdbc.DataTypeUtils.jdbcType2DataType(DataTypeUtils.java:61) at datart.data.provider.jdbc.adapters.JdbcDataProviderAdapter.readTableColumn(JdbcDataProviderAdapter.java:193) at datart.data.provider.jdbc.adapters.JdbcDataProviderAdapter.readTableColumn(JdbcDataProviderAdapter.java:176) at datart.data.provider.JdbcDataProvider.readTableColumns(JdbcDataProvider.java:85) at datart.data.provider.ProviderManager.readTableColumns(ProviderManager.java:114) at datart.server.service.impl.DataProviderServiceImpl.readTableColumns(DataProviderServiceImpl.java:140) at datart.server.job.SchemaSyncJob.execute(SchemaSyncJob.java:94) at datart.server.job.SchemaSyncJob.execute(SchemaSyncJob.java:72) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

fohjseiou commented 2 weeks ago

我这边也是的，请问解决了吗

link2fun commented 1 week ago

改下源码就是了

public static ValueType jdbcType2DataType(int jdbcType) {
        SqlTypeName sqlTypeName = SqlTypeName.getNameForJdbcType(jdbcType);
        SqlTypeFamily family;
        if (sqlTypeName == null) {
            family = SqlTypeFamily.getFamilyForJdbcType(jdbcType);
        } else {
            family = sqlTypeName.getFamily();
        }
// add start ======================
        if (family == null) {
            return ValueType.STRING;
        }
// add end ======================
        switch (family) {
            case NUMERIC:
                return ValueType.NUMERIC;
            case DATE:
            case TIME:
            case TIMESTAMP:
            case DATETIME:
                return ValueType.DATE;
            default:
                return ValueType.STRING;
        }
    }

running-elephant / datart

doris无法加载表结构数据 #2344