Closed dragon299 closed 4 years ago
Hi @dragon299 , I would like to confirm whether you are asking us to add a property "fs.oci.buffer.dir" which will allow you to use any directory on the local system instead of hadoop.tmp.dir. Also, could you please state why can't you increase the space of hadoop.tmp.dir?
Hi @omkar07, correct. I use the oci-hdfs-connector to copy a huge among of data from hdfs to oci using distcp. Distcp will copy all data to tmp dir bevor start to transfer it to oci. In my case I need a lot of space for this buffer dir and it does not make sense for me to extend /tmp to a few terabytes.
Hi @dragon299 , In that case could you please state why can't you set the hadoop.tmp.dir in the core-site.xml to use any directory on the local system?
Hi @dragon299 , Could you please let us know if you are still facing the above issue? We will close this issue if we don't hear from you by next week.
Hi @dragon299 , We are closing this issue since we haven't heard from you. please reopen it in case you are still facing this issue.
There is a huge among of space needed for the buffer dir during copy process of big files from hdfs to object storage.
The s3 connector has the option to configure the buffer dir (fs.s3.buffer.dir). This is missing for oci I guess.