search

fog / fog-openstack

Fog for OpenStack Platform
MIT License
68 stars 130 forks source link

Question: How do I determine if the object's checksum was compared and is valid? #308

Closed jeffreyguenther closed 7 years ago

jeffreyguenther commented 7 years ago

I'm working on implementing an OpenStack service for ActiveStorage, the new file attachment gem the Rails community is working on.

Is there a way to determine if a file matches it's checksum after upload? I notice there's no exception thrown if they don't match. You can see my work at https://github.com/jeffreyguenther/activestorage

seanhandley commented 7 years ago

Hi @jeffreyguenther,

Yes, the attribute you want is file.etag - which should give you the md5 sum for the file/object in question.

Hope that helps and all the best with your ActiveStorage work!

jeffreyguenther commented 7 years ago

To clarify, this is the checksum that OpenStack calculates upon upload? It's not the checksum we pass to container.files.create(key: key, body: io, etag: checksum) as the expected checksum?

Is there any reason to pass the checksum on create? It doesn't seem to be doing anything... I feel like I'm missing something.

seanhandley commented 7 years ago

I believe the expected behaviour is for Swift (the OpenStack object storage service) to return an error if the uploaded file's checksum differs from the one you specify prior to the upload, though TCP should ensure this anyway so I'm not sure why you'd need it during the upload itself.

However, Swift also calculates a checksum each time you access the file to ensure it hasn't changed during the time it's been in the store e.g. Bit Rot so perhaps that's the rationale! I'm not very au-fait with Swift so I'm not sure...

jeffreyguenther commented 7 years ago

How will that error be exposed by fog?

Sounds like there are a bunch of integrity checks built-in. For Active Storage, we want to verify that the file received is the one we thought we uploaded, so using the file.etag looks like it's the right thing to compare our pre-upload checksum against.

seanhandley commented 7 years ago

Since the API would return a non-success HTTP code, Fog would raise the error to the user's client code. The Swift API doc says:

If the MD5 checksum of the data that is written to the object store does not match the optional ETag value, the operation returns the Unprocessable Entity (422) response code.

You can handle this in your code by catching Excon::Error::UnprocessableEntity

See log output:

irb(main):073:0> fog.put_object 'foo', 'bar', 'test', etag: 'fail'
Excon::Error::UnprocessableEntity: Expected(201) <=> Actual(422 Unprocessable Entity)
excon.error.response
  :body          => "{\"Code\":\"UnprocessableEntity\",\"BucketName\":\"foo\",\"RequestId\":\"tx0000000000000009a77d3-005968905b-ea475f6-uk-sal01\",\"HostId\":\"ea475f6-uk-sal01-uk\"}"
  :cookies       => [
  ]
  :headers       => {
    "Accept-Ranges"             => "bytes"
    "Content-Length"            => "146"
    "Content-Type"              => "application/json; charset=utf-8"
    "Date"                      => "Fri, 14 Jul 2017 09:35:23 GMT"
    "Last-Modified"             => "Thu, 01 Jan 1970 00:00:00 GMT"
    "Server"                    => "Apache/2.4.7 (Ubuntu)"
    "Strict-Transport-Security" => "max-age=31536000"
    "X-Trans-Id"                => "tx0000000000000009a77d3-005968905b-ea475f6-uk-sal01"
    "etag"                      => "098f6bcd4621d373cade4e832627b4f6"
  }
  :host          => "storage.datacentred.io"
  :local_address => "138.68.164.200"
  :local_port    => 37004
  :path          => "/swift/v1/AUTH_a3196cda804744d7b5d38d903a79610f/foo/bar"
  :port          => 443
  :reason_phrase => "Unprocessable Entity"
  :remote_ip     => "185.43.218.55"
  :status        => 422
  :status_line   => "HTTP/1.1 422 Unprocessable Entity\r\n"

    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/middlewares/expects.rb:7:in `response_call'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/middlewares/response_parser.rb:9:in `response_call'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/docker-api-1.31.0/lib/excon/middlewares/hijack.rb:45:in `response_call'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/connection.rb:388:in `response'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/connection.rb:252:in `request'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/middlewares/idempotent.rb:27:in `error_call'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/middlewares/base.rb:11:in `error_call'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/middlewares/base.rb:11:in `error_call'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/middlewares/base.rb:11:in `error_call'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/connection.rb:272:in `rescue in request'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/connection.rb:215:in `request'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/middlewares/idempotent.rb:27:in `error_call'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/middlewares/base.rb:11:in `error_call'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/middlewares/base.rb:11:in `error_call'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/middlewares/base.rb:11:in `error_call'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/connection.rb:272:in `rescue in request'
... 2 levels...
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/middlewares/base.rb:11:in `error_call'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/middlewares/base.rb:11:in `error_call'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/middlewares/base.rb:11:in `error_call'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/connection.rb:272:in `rescue in request'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/excon-0.54.0/lib/excon/connection.rb:215:in `request'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/fog-core-1.43.0/lib/fog/core/connection.rb:81:in `request'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/fog-openstack-0.1.7/lib/fog/openstack/storage.rb:146:in `request'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/fog-openstack-0.1.7/lib/fog/openstack/requests/storage/put_object.rb:36:in `put_object'
    from (irb):73
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/railties-5.0.2/lib/rails/commands/console.rb:65:in `start'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/railties-5.0.2/lib/rails/commands/console_helper.rb:9:in `start'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/railties-5.0.2/lib/rails/commands/commands_tasks.rb:78:in `console'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/railties-5.0.2/lib/rails/commands/commands_tasks.rb:49:in `run_command!'
    from /home/rails/stronghold/vendor/bundle/ruby/2.3.0/gems/railties-5.0.2/lib/rails/commands.rb:18:in `<top (required)>'
    from bin/rails:4:in `require'
    from bin/rails:4:in `<main>'
irb(main):074:0> fog.put_object 'foo', 'bar', 'test'
=> #<Excon::Response:0x00000007454a28 @data={:body=>"", :cookies=>[], :host=>"storage.datacentred.io", :headers=>{"Date"=>"Fri, 14 Jul 2017 09:35:45 GMT", "Server"=>"Apache/2.4.7 (Ubuntu)", "etag"=>"098f6bcd4621d373cade4e832627b4f6", "Last-Modified"=>"Fri, 14 Jul 2017 09:35:45 GMT", "X-Trans-Id"=>"tx0000000000000009a88d3-0059689071-eb9c48c-uk-sal01", "Content-Type"=>"application/json; charset=utf-8", "Strict-Transport-Security"=>"max-age=31536000"}, :path=>"/swift/v1/AUTH_a3196cda804744d7b5d38d903a79610f/foo/bar", :port=>443, :status=>201, :status_line=>"HTTP/1.1 201 Created\r\n", :reason_phrase=>"Created", :remote_ip=>"185.43.218.55", :local_port=>37008, :local_address=>"138.68.164.200"}, @body="", @headers={"Date"=>"Fri, 14 Jul 2017 09:35:45 GMT", "Server"=>"Apache/2.4.7 (Ubuntu)", "etag"=>"098f6bcd4621d373cade4e832627b4f6", "Last-Modified"=>"Fri, 14 Jul 2017 09:35:45 GMT", "X-Trans-Id"=>"tx0000000000000009a88d3-0059689071-eb9c48c-uk-sal01", "Content-Type"=>"application/json; charset=utf-8", "Strict-Transport-Security"=>"max-age=31536000"}, @status=201, @remote_ip="185.43.218.55", @local_port=37008, @local_address="138.68.164.200">
jeffreyguenther commented 7 years ago

Wonderful! That's exactly what I was looking for!

seanhandley commented 7 years ago

Great 👍