aliyun / aliyun-odps-python-sdk

ODPS Python SDK and data analysis framework
http://pyodps.readthedocs.io
Apache License 2.0
434 stars 97 forks source link

使用tunnel时,如何用多个block啊? #40

Closed thesby closed 6 years ago

thesby commented 6 years ago
from odps.tunnel import TableTunnel

table = o.get_table('my_table')

tunnel = TableTunnel(odps)
upload_session = tunnel.create_upload_session(table.name, partition_spec='pt=test')

with upload_session.open_record_writer(0) as writer:
    record = table.new_record()
    record[0] = 'test1'
    record[1] = 'id1'
    writer.write(record)

    record = table.new_record(['test2', 'id2'])
    writer.write(record)

upload_session.commit([0])

这里使用只有一个block_id,感觉速度不够快,怎么能够加快这个操作呢?因为record非常多,需要的时间较长。

qinxuye commented 6 years ago
open_record_writer(0) 

这个 0 就是 block_id,你可以多开几个,每个对应一个 block_id