search

kzk / webhdfs

Ruby client for Hadoop WebHDFS
Other
81 stars 46 forks source link

Accept data stream #4

Closed jhsenjaliya closed 10 years ago

jhsenjaliya commented 10 years ago

current webhdfs gem does not take IO as payload type for PUT and POST request which puts limits on size of data that can be sent over to be less than available memory. This does not work well for big files that needs to be transferred to hdfs. This patch update the current method in minimal way that takes ruby IO as payload type, that reads the IO in predefined chunk size ( 1024 as hard coded in ruby net http module ), removing any memory limitation.

tagomoris commented 10 years ago

This looks great patch! @jay-altiscale Are you using this patched version in your environment?

jhsenjaliya commented 10 years ago

Yes, we are using it in production, and would be great if you can review and merge it to master branch. Thanks for taking a look.

tagomoris commented 10 years ago

Merged, and released v0.5.5. Thank you!