分块上传的demo - Githubissues

2shou commented 8 years ago

有时间写个demo，大概是将文件分块，计算每一块的MD5 precreate 对于服务器上的某个文件需要上传的块 upload_tmpfile 上传临时文件（块） upload_superfile 合并块（在precreate返回所需块为空时调用本函数可合并文件）注意，百度服务器上文件是分块保存的，块不会消失

求完善这个demo，云主机内存有限，稍微大一点的文件就报MemoryError了。希望上传的是一个mp4文件，不是按行分隔的文本文件。

ly0 commented 8 years ago

:smile: 把MP4文件分成一堆块，每传一块记录一下返回的MD5，然后释放块占用的内存。最后调用合并文件就好了呗

morefreeze commented 8 years ago

#coding: utf-8
import os,json,sys,tempfile
from baidupcsapi import PCS

pcs = PCS('username','password')
chinksize = 1024*1024*16
fid = 1
md5list = []
tmpdir = tempfile.mkdtemp('bdpcs')
with open(sys.argv[1],'rb') as infile:
    while 1:
        data = infile.read(chinksize)
        if len(data) == 0: break
        smallfile = os.path.join(tmpdir, 'tmp%d' %fid)
        with open(smallfile, 'wb') as f:
            f.write(data)
        print('chunk%d size %d' %(fid, len(data)))
        fid += 1
        print('start uploading...')
        ret = pcs.upload_tmpfile(open(smallfile, 'rb'))
        md5list.append(json.loads(ret.content)['md5'])
        print('md5: %s' %(md5list[-1]))
        os.remove(smallfile)

os.rmdir(tmpdir)
ret = pcs.upload_superfile('/'+os.path.basename(sys.argv[1]), md5list)
print ret.content

写了个简单的，不嫌丑可以拿着先用。python upload.py huge_file @ly0

morefreeze commented 8 years ago

@2shou 已更新到README fix in #33

ly0 / baidupcsapi

分块上传的demo #26