search

aliyun / aliyun-odps-jdbc

JDBC Driver for ODPS
Other
125 stars 38 forks source link

如何使用JAVA编码方式将ODPS表数据全量同步到本地MySql表，数据量大约在100万[QUESTION] #71

Closed Malaban closed 4 years ago

Malaban commented 4 years ago

A clear and concise description of what the problem is.

cornmonster commented 4 years ago

比较好的方式是使用odps-sdk的tunnel读取odps 数据，之后通过mysql jdbc写入mysql

odps-sdk tunnel的example：

    TableTunnel tunnel = new TableTunnel(odps);
    try {
        DownloadSession downloadSession = tunnel.createDownloadSession(project, table);
        System.out.println("Session Status is : "+ downloadSession.getStatus().toString());
        long count = downloadSession.getRecordCount();
        System.out.println("RecordCount is: " + count);
        RecordReader recordReader = downloadSession.openRecordReader(0, count);
        Record record;
        while ((record = recordReader.read()) != null) {
            consumeRecord(record, downloadSession.getSchema());
        }
        recordReader.close();
    } catch (TunnelException e) {
        e.printStackTrace();
    } catch (IOException e1) {
        e1.printStackTrace();
    }