Java binding smoking test failed

[wgzhao]

I want to ingrate opendal into my project. I wrote a simple java binding smoking test code but failed.

public class OpendalUtil {
    public static void main(String[] args)  {
        Map<String, String> params = new HashMap<>();
        params.put("bucket", "test");
        params.put("region", "us-east-1");
        params.put("access_key_id", "key");
        params.put("secret_access_key", "secret key");
        params.put("endpoint", "http://10.x.x.x:19000");
        params.put("root", "/");
        BlockingOperator operator =new BlockingOperator("s3", params);
        operator.write("test.txt", "hello, world!");
    }
}

It gives me the following error:

Exception in thread "main" org.apache.opendal.OpenDALException: Unsupported (permanent) at blocking_write => operation is not supported
    at org.apache.opendal.BlockingOperator.write(Native Method)
    at org.apache.opendal.BlockingOperator.write(BlockingOperator.java:49)
    at org.apache.opendal.BlockingOperator.write(BlockingOperator.java:45)
    at com.wgzhao.addax.plugin.writer.txtfilewriter.OpendalUtil.main(OpendalUtil.java:66)

I also try to use Operator to replace BlockingOperator ， it gives me the following error:

thread 'tokio-runtime-worker' panicked at 'A Tokio 1.x context was found, but IO is disabled. Call `enable_io` on the runtime builder to enable IO.', /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/net/tcp/stream.rs:160:18
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::option::expect_failed
   3: tokio::runtime::io::registration::Registration::new_with_interest_and_handle
   4: tokio::io::poll_evented::PollEvented<E>::new
   5: tokio::net::tcp::stream::TcpStream::new
   6: <hyper::client::connect::http::HttpConnector<R> as tower_service::Service<http::uri::Uri>>::call::{{closure}}
   7: <hyper_rustls::connector::HttpsConnector<T> as tower_service::Service<http::uri::Uri>>::call::{{closure}}
   8: reqwest::connect::Connector::connect_with_maybe_proxy::{{closure}}
   9: reqwest::connect::with_timeout::{{closure}}
  10: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
  11: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
  12: <futures_util::future::try_future::try_flatten::TryFlatten<Fut,<Fut as futures_core::future::TryFuture>::Ok> as core::future::future::Future>::poll
  13: <hyper::common::lazy::Lazy<F,R> as core::future::future::Future>::poll
  14: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  15: hyper::client::client::Client<C,B>::retryably_send_request::{{closure}}
  16: <hyper::client::client::ResponseFuture as core::future::future::Future>::poll
  17: <reqwest::async_impl::client::PendingRequest as core::future::future::Future>::poll
  18: <reqwest::async_impl::client::Pending as core::future::future::Future>::poll
  19: opendal::raw::http_util::client::HttpClient::send::{{closure}}
  20: opendal::services::s3::core::S3Core::send::{{closure}}
  21: <opendal::services::s3::writer::S3Writer as opendal::raw::oio::write::multipart_upload_write::MultipartUploadWrite>::write_once::{{closure}}
  22: <opendal::raw::oio::write::multipart_upload_write::MultipartUploadWriter<W> as opendal::raw::oio::write::api::Write>::write::{{closure}}
  23: <opendal::layers::error_context::ErrorContextWrapper<T> as opendal::raw::oio::write::api::Write>::write::{{closure}}
  24: <opendal::layers::complete::CompleteWriter<W> as opendal::raw::oio::write::api::Write>::write::{{closure}}
  25: opendal::types::operator::operator::Operator::write_with::{{closure}}::{{closure}}
  26: <opendal::types::operator::operator_futures::FutureWrite as core::future::future::Future>::poll
  27: opendal_java::operator::intern_write::{{closure}}
  28: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
  29: tokio::runtime::task::core::Core<T,S>::poll
  30: tokio::runtime::task::harness::Harness<T,S>::poll
  31: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  32: tokio::runtime::scheduler::multi_thread::worker::Context::run
  33: tokio::runtime::context::scoped::Scoped<T>::set
  34: tokio::runtime::context::runtime::enter_runtime
  35: tokio::runtime::scheduler::multi_thread::worker::run
  36: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
  37: tokio::runtime::task::core::Core<T,S>::poll
  38: tokio::runtime::task::harness::Harness<T,S>::poll
  39: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

here my pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>com.wgzhao.addax</groupId>
        <artifactId>addax-all</artifactId>
        <version>4.1.2-SNAPSHOT</version>
        <relativePath>../../../pom.xml</relativePath>
    </parent>

        <!-- integrate OpenDAL -->
        <dependency>
            <groupId>org.apache.opendal</groupId>
            <artifactId>opendal-java</artifactId>
            <version>${opendal.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.opendal</groupId>
            <artifactId>opendal-java</artifactId>
            <version>${opendal.version}</version>
            <classifier>${os.detected.classifier}</classifier>
        </dependency>
    </dependencies>

    <build>
        <extensions>
            <extension>
                <groupId>kr.motd.maven</groupId>
                <artifactId>os-maven-plugin</artifactId>
                <version>1.7.0</version>
            </extension>
        </extensions>
    </build>
</project>

[Xuanwo]

It's a known issue that s3 doesn't support blocking operation yet and we need to enable blocking layer to make it work.

cc @tisonkun, would you like to take a look over enable BlockingLayer for java binding by default?

[wgzhao]

I also try to test redis, sftp Operator, they both got the following error:

Exception in thread "main" org.apache.opendal.OpenDALException: Unsupported (permanent) at , context: { scheme: redis } => scheme is not enabled or supported
    at org.apache.opendal.BlockingOperator.constructor(Native Method)
    at org.apache.opendal.BlockingOperator.<init>(BlockingOperator.java:41)
    at com.wgzhao.addax.plugin.writer.txtfilewriter.OpendalUtil.main(OpendalUtil.java:71)

I also use Redis, REDIS, sftp, Sftp as the operator scheme.

Is there something wrong with the code I wrote or is there a problem with the Java binding itself at the current?

[Xuanwo]

Is there something wrong with the code I wrote or is there a problem with the Java binding itself at the current?

redis and sftp service requires enabling extra features, thus opendal-java v0.39 doesn't bundle them. We have enabled them by default at main branch so I believe this will be fixed in our next release.

And for s3 support, opendal-java v0.39 only support been used in async way.

[wgzhao]

for s3, I also try to use Operator class , the class comment indicates that it's async way, but it does not work.

[Xuanwo]

for s3, I also try to use Operator class , the class comment indicates that it's async way, but it does not work.

Oh, that's unexpected. Would you like to share your code snippet? I'm willing to take a look.

[wgzhao]

The code in the first comment is complete I also upload a complete maven project later.

[Xuanwo]

The code in the first comment is complete

Hi, BlockingOperator for s3 in opendal-java v0.39 is known to be non-working. But Operator should work with async way.

[tisonkun]

0.39.0 opendal-java contains several bugs:

• https://github.com/apache/incubator-opendal/commit/69a4f721e58210425a98779802486e7133c3fba5 enable_io which is exactly the error message you met and the root cause that async operator not work.

This should be fixed in the next release.

[wgzhao]

@Xuanwo @tisonkun I create a opendal java binding example project, please check it.