andelf / go-curl

golang curl(libcurl) binding.
Apache License 2.0
478 stars 130 forks source link

curled zip files are roughly double in size #29

Closed miroswan closed 9 years ago

miroswan commented 9 years ago

Here is the header info from verbose output:

< HTTP/1.1 200 OK
* Server Apache-Coyote/1.1 is not blacklisted
< Server: Apache-Coyote/1.1
< Set-Cookie: cq-authoring-mode=CLASSIC; Path=/
< Last-Modified: Wed, 26 Nov 2014 22:33:43 GMT
< Content-Type: application/zip
< Content-Length: 68339
< Date: Fri, 28 Nov 2014 18:04:21 GMT

After curling the file size is 129168, which is about double. Here is the WRITEFUNCTION I have:

// callback function for OPT_WRITEFUNCTION. See libcurl docs.
func WriteData(ptr []byte, userdata interface{}) bool {
    // Create ptr reference to tempfile to which we are writing
    writePtr := userdata.(*os.File)

    // Create a bufio reader and writer
    reader := bufio.NewReader(bytes.NewReader(ptr))
    writer := bufio.NewWriter(writePtr)

    // Create a buffer of 1024 bytes
    buf := make([]byte, 1024)

    // Loop until break
    for {
        // read 1024 bytes and return actual number of bytes read into n
        nin, err := reader.Read(buf)
        if err != nil && err != io.EOF {
            panic(err)
        }
        // If we have nothing left to read, then break
        if nin == 0 {
            break
        }

        // write a buffer data from 0 to number of bytes read
        nout, err := writer.Write(buf[:nin])
        if err != nil {
            panic(err)
        }
        fmt.Printf("%d bytes written\n", nout)
    }

    // Flush the writer after we're done writing
    err := writer.Flush()
    if err != nil {
        panic(err)
    }

    return true
}

I tried copying and pasting a WRITEFUNCTION from the code examples in the the project, but they give me the same results.

Another thing that I've noticed, is that when I double click the downloaded zip file, it creates a file of the same name, with .cpgz appended to it. Double click it again and it creates a file of the same original name. This creates an infinite loop: http://osxdaily.com/2013/02/13/open-zip-cpgz-file/

unziping with the unzip binary works okay.

What could be causing the file to have extra data written to it?

andelf commented 9 years ago

I tried and couldn't reproduce your error. Would you post your full code here?

miroswan commented 9 years ago

Ahh. You are right. The bug is on my end. Will close this issue.