Closed piyushg closed 5 years ago
Thanks for reporting this issue. We'll look into reproducing this issue and addressing it.
Per the attached callstack, the createDirectory operation failed due to the service returning the below failure.
Caused by: com.oracle.bmc.model.BmcException: (404, ObjectNotFound, false) null (opc-request-id: 951de845-9b75-c5b1-7074-0a9fa1a51e3b)
The service here returns a 404, ObjectNotFound error. The connector does not attempt to detect, prevent or serialize access to objects from multiple threads. It is the responsibility of the caller to ignore expected failures (ex createDirectory failure if the parent directory was deleted by a separate thread).
https://github.com/oracle/oci-hdfs-connector/blob/master/hdfs-connector/src/main/java/com/oracle/bmc/hdfs/store/BmcDataStore.java
BmcDataStore -> createDirectory is not thread safe and throwing IOException. when using spark to store dataset to object storage using this connector we are hitting across this issue. data access to object storage through connector should be thread safe.
Stacktrace: ExceptionStack_1.txt
Code reference from where exception is getting thrown:
/**
if the operation cannot be completed. */ public void createDirectory(final Path path) throws IOException { // nothing to do for the "root" directory if (this.isRootDirectory(path)) { LOG.debug("Root directory specified, nothing to create"); return; }
}