对超大文件的处理，会造成内存被耗尽，或者缓冲区溢出

xbee commented 8 years ago

你拷贝的这段代码有明显的问题，一点都不优雅，而且原本应该要解决的问题也没有解决，你自己思考一下有啥更好的方式？

blocks := uint64(math.Ceil(float64(filesize) / float64(filechunk)))  // <----此处仍然有问题 
hash := sha1.New()

for i := uint64(0); i < blocks; i++ {
    blocksize := int(math.Min(filechunk, float64(filesize-int64(i*filechunk))))
    buf := make([]byte, blocksize)
    file.Read(buf)
    io.WriteString(hash, string(buf))
}

programmerZhou commented 8 years ago

首先，这个不是我的代码，至少不实完全一样。

对超大文件的处理会导致内存耗尽或缓冲区溢出，您实际测试过吗，这几行代码不会导致这种情况，io.WriteString(hash, string(buf)), 是否会导致，依赖于hash.Hash.Write函数的实现，我看了下也没有问题。如果您测试过有问题，请详细说明。

这种计算md5的方式就是golang官方的计算方式，example代码如下：

package main

import ( "crypto/md5" "fmt" "io" )

func main() { h := md5.New() io.WriteString(h, "The fog is getting thicker!") io.WriteString(h, "And Leon's getting laaarger!") fmt.Printf("%x", h.Sum(nil)) }

blocks := uint64(math.Ceil(float64(filesize) / float64(filechunk))) // <----此处仍然有问题

这一行没有问题，math.Ceil是天花板除，舍弃小数点后的数，如果小数点后不为零，则整数加一。优雅怎么评价，不能有链式表达式吗。这个也比较好看懂吧。。

On Oct 17, 2016, at 9:32 PM, xbee notifications@github.com wrote:

你拷贝的这段代码有明显的问题，一点都不优雅，而且原本应该要解决的问题也没有解决，你自己思考一下有啥更好的方式？

blocks := uint64(math.Ceil(float64(filesize) / float64(filechunk))) // <----此处仍然有问题 hash := sha1.New()

for i := uint64(0); i < blocks; i++ { blocksize := int(math.Min(filechunk, float64(filesize-int64(i*filechunk)))) buf := make([]byte, blocksize) file.Read(buf) io.WriteString(hash, string(buf)) } — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/programmerZhou/gotree/issues/3, or mute the thread https://github.com/notifications/unsubscribe-auth/APmse2OxR3XuRhaGG-K0pnsAnBHvaRFKks5q03jbgaJpZM4KYoEP.

programmerZhou commented 8 years ago

我的程序计算的md5sum和md5sum 计算的比较过，结果一样

On Oct 17, 2016, at 9:32 PM, xbee notifications@github.com wrote:

你拷贝的这段代码有明显的问题，一点都不优雅，而且原本应该要解决的问题也没有解决，你自己思考一下有啥更好的方式？

blocks := uint64(math.Ceil(float64(filesize) / float64(filechunk))) // <----此处仍然有问题 hash := sha1.New()

for i := uint64(0); i < blocks; i++ { blocksize := int(math.Min(filechunk, float64(filesize-int64(i*filechunk)))) buf := make([]byte, blocksize) file.Read(buf) io.WriteString(hash, string(buf)) } — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/programmerZhou/gotree/issues/3, or mute the thread https://github.com/notifications/unsubscribe-auth/APmse2OxR3XuRhaGG-K0pnsAnBHvaRFKks5q03jbgaJpZM4KYoEP.

programmerZhou / gotree

对超大文件的处理，会造成内存被耗尽，或者缓冲区溢出 #3