search

dask / hdfs3

A wrapper for libhdfs3 to interact with HDFS from Python
http://hdfs3.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
136 stars 40 forks source link

Fix bug- overriding the buffer size at file read / write #150

Closed WYishai closed 6 years ago

martindurant commented 6 years ago

Thank you, this looks like a useful addition. Is the intention that the module-level variables might be altered by users? I would appreciate some extra documentation about the choices here, and maybe a couple of small tests to show that the buffer variables are correctly set.

WYishai commented 6 years ago

I'm not sure I fully understand what you mean. The python code initialize the native file handler with the buff parameter, but during the reading or writing operation itself it uses a buffer with different size (which was arbitrarily determind with fixed number). I change the python code to use the self.buff instead of this value, but if the user has not defined the buff parameter (==0) it uses the default value (which i moved to const, for clearance and consistency). I have not changed these default values (except from making them consistent all over the code)...