search

fog / fog-rackspace

Rackspace provider gem for Fog ecosystem
MIT License
8 stars 35 forks source link

Download from Rackspace Doesn't always get all Segments #33

Open homerlex opened 4 years ago

homerlex commented 4 years ago

We are using fog (fog-rackspace (0.1.6) to do offsite backups from Heroku to Rackspace. Our files are 6GB+ so we need to do the segmented uploads as described here: upload-large-files.

The upload works perfectly every time. When we manually download the file from the Rackspace UI it is fully intact.

The problem we are having is when we try downloading the file using fog we sometimes only get the first segment of the file. We can tell because the size of the file we see download is exactly 4GB (our UPLOAD_SEGMENT_LIMIT is set to 4GB). We are downloading the file with the suggested code from the above mentioned blog:

File.open('downloaded-file.jpg', 'w') do | f |
  directory.files.get("my_big_file.jpg") do | data, remaining, content_length |
    f.syswrite data
  end
end

The problem with the download is intermittent. Has anyone else experienced this issue?

homerlex commented 4 years ago

I was just testing with an 11MB file with 3MB chunks and has the same issue. The problem is actuall that the last segment was not downloaded. I'm getting the feeling that perhaps we are trying to download the file too soon after updating it. Perhaps we need to sleep a bit in order for Rackspace to get its act together before we try the download? I'll experiment with this...