search

oracle / oci-hdfs-connector

HDFS Connector for Oracle Cloud Infrastructure
https://cloud.oracle.com/cloud-infrastructure
Other
27 stars 26 forks source link

Missing fs.oci.buffer.dir #24

Closed dragon299 closed 4 years ago

dragon299 commented 4 years ago

There is a huge among of space needed for the buffer dir during copy process of big files from hdfs to object storage.

The s3 connector has the option to configure the buffer dir (fs.s3.buffer.dir). This is missing for oci I guess.

omkar07 commented 4 years ago

Hi @dragon299 , I would like to confirm whether you are asking us to add a property "fs.oci.buffer.dir" which will allow you to use any directory on the local system instead of hadoop.tmp.dir. Also, could you please state why can't you increase the space of hadoop.tmp.dir?

dragon299 commented 4 years ago

Hi @omkar07, correct. I use the oci-hdfs-connector to copy a huge among of data from hdfs to oci using distcp. Distcp will copy all data to tmp dir bevor start to transfer it to oci. In my case I need a lot of space for this buffer dir and it does not make sense for me to extend /tmp to a few terabytes.

omkar07 commented 4 years ago

Hi @dragon299 , In that case could you please state why can't you set the hadoop.tmp.dir in the core-site.xml to use any directory on the local system?

omkar07 commented 4 years ago

Hi @dragon299 , Could you please let us know if you are still facing the above issue? We will close this issue if we don't hear from you by next week.

omkar07 commented 4 years ago

Hi @dragon299 , We are closing this issue since we haven't heard from you. please reopen it in case you are still facing this issue.