Do chunk wise transfer

[rishiloyola]

Note: Do not merge this. I test my code locally but still need to test on a large amount of file.

Reference

1. https://gist.github.com/cryptix/9dd094008b6236f4fc57
2. https://garbagecollected.org/2015/05/30/io-with-go-io-pipe/

[vkuznet]

from the code I don't see yet how you read part of the file. And, your goroutine reads c.Bytes which means the whole file.

On 0, Rishi notifications@github.com wrote:

Note: Do not merge this. I test my code locally still need to test on a large amount of file. You can view, comment on, or merge this pull request online at:

https://github.com/vkuznet/transfer2go/pull/37

-- Commit Summary --

• Do chunck wise transfer

-- File Changes --

M core/request.go (71)

-- Patch Links --

https://github.com/vkuznet/transfer2go/pull/37.patch https://github.com/vkuznet/transfer2go/pull/37.diff

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/vkuznet/transfer2go/pull/37

[rishiloyola]

I tried to test this code on 1.6 GB file. But this logic is not working fine. Any suggestions?

[rishiloyola]

@vkuznet I opened one issue on Golang repo to solve our problem. link - https://github.com/golang/go/issues/21272

[rishiloyola]

@vkuznet Please check out my latest commit. I solved this problem and successfully transferred 1.3 GB file. Here is the screenshot which is indicating that destination has received 1.3 GB data but source file has only 7 bytes in size. The source file is only 7 bytes in size because the client is reading the entire file in memory and due to lack of RAM it can't upload 1.3 GB file. So registered a fake file and then changed the file data.

[vkuznet]

Rishi, I don't understand what you wrote. Either you transfer successfully or not. I don't understand how you end-up with 7 bytes, does it because your RAM haven't had 7 bytes left? Can you transfer 1.2 GB file in that case? And, why client need to read entire file in its RAM? Why you can't just dump it to the file on file-system?

On 0, Rishi notifications@github.com wrote:

@vkuznet Please check out my latest commit. I solved this problem and successfully transferred 1.3 GB file. Here is the screenshot which is indicating that destination has received 1.3 GB data but source file has only 7 bytes in size. The source file is only 7 bytes in size because the client is reading the entire file in memory and due to lack of RAM it can't upload 1.3 GB file. So registered a fake file and then changed the file data.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/vkuznet/transfer2go/pull/37#issuecomment-320444922

[rishiloyola]

why client need to read entire file in its RAM? While uploading the records to the agent catalog our client loads the entire file into the RAM. Check out this code - link I need to change this logic.

how you end-up with 7 bytes? While uploading the big file(1.2 GB) to the agent catalog using this client the process is running out of memory because we are not doing chunk wise transfer. So I registered a fake file which is 7 bytes in size and then change its content on disk. So the physical path would be the same but content will be different. I put around 1.3 GB data in that file and then tried to transfer over HTTP. I successfully transferred that file without running out of memory using Go's io.pipe.

[rishiloyola]

@vkuznet check out my last commit. Now we can calculate the hash of large files without running out of memory. You can merge this pr.

Error thrown by old code

Successfully registered new records by updated code

Showing the size of registered file - 1.3Gb

[vkuznet]

Merged.