kzk / webhdfs

Ruby client for Hadoop WebHDFS
Other
81 stars 46 forks source link

accept IO-like object as payload #11

Closed skaji closed 9 years ago

skaji commented 9 years ago

Currently webhdfs gem accepts IO objects as payloads. This pull request relaxes it so that it accepts IO-like objects (i.e. objects that have read and size methods).

Let me explain why I need this modification. I want fluent-plugin-webhdfs to support gzip compression. If webhdfs gem accepts this patch, then I can write the following code:

require 'zlib'
require 'tempfile'
require 'webhdfs'
client = WebHDFS::Client.new("host", 50070)
content = "this is sample"

tmp = Tempfile.new("webhdfs-")
begin
  w = Zlib::GzipWriter.new(tmp)
  w.puts(content)
  w.close
  tmp.close
  tmp.open
  client.create("/path/to/file.gz", tmp)
ensure
  tmp.close(true) rescue nil
end
tagomoris commented 9 years ago

Thank you! I've released webhdfs.gem v0.6.0 just now with this (and previous) patch!

skaji commented 9 years ago

Amazing! Thank you!