idea: Introduce Codegen into OpenDAL

[Xuanwo]

The main obstacles preventing our users from trying out bindings are the documentation, including API docs, examples, and more.

However, OpenDAL now supports over 50 services, with more being added continuously. Our bindings have also grown, now totaling 14. It's neither possible nor wise to attempt filling this gap manually. We have discussed about this many times, for example, https://github.com/apache/opendal/issues/3537.

So I think it's time for us to introduce codegen to help us:

• rust: Generate Config
• rust: Generate Builder's API that plays with Config
• rust: Generate Examples
• python: Generate API Stub
• python: Generate Examples
• ...

I have build a demo at https://github.com/apache/opendal/pull/4278.

During building the demo, I found it's useful even just for rust:

• The same mask_secret for all services
• The same sensitive, required metadata
• Remove the need to duplicate builder API

So what do you think about this?

[Xuanwo]

cc binding owners:

• java @tisonkun @G-XD
• nodejs @suyanhanx
• python @messense @Zheaoli

Other bindings are not released yet, let's keep the scope small for now.

[tisonkun]

I don't find a n * m complexity here, but the binding should wrap APIs. Once APIs are wrapped, all services can be used following the same configuration pattern.

[Xuanwo]

Once APIs are wrapped, all services can be used following the same configuration pattern.

Are you talking about exposing S3Config?

[tisonkun]

If you mean a structural config type, I agree. Using IDL protobuf or avro may help?

[tisonkun]

Are you talking about exposing S3Config?

Currently, we can configure service with a nonstructural map.

[Xuanwo]

Using IDL protobuf or avro may help? Or try use jsonnet.

How will we expose API to users in this way? Use the struct that generated by protobuf directly?

[tisonkun]

Yes. I'd prefer to avoid defining a handy DSL, which is likely to be error-prone. But instead using a battle-tested schemata solution like Avro - https://avro.apache.org/docs/1.11.1/idl-language.

Protobuf can be hard to use in the Java world if we don't need the serde functions.

[Xuanwo]

Our discussion seems to be going to far regarding config structure generation's detail. That's not the main issue I want to address. Let me explain my motivation and goal first.

---

OpenDAL's vision is to freely access data. Our rust core enables this for the rust language, and our language bindings extend this capability across various languages for all users.

Our language is designed to mirror native implementations, allowing users to use our bindings independently without consulting Rust documentation or having prior knowledge of Rust core.

However, it's currently impossible. Take java binding as an example.

OpenDAL Java features automatically generated JavaDoc available at javadoc.io, but the documentation lacks a lot of information I wish it included:

• Every detailed services description
• Docs/Examples for every services' config

I hope users can install the opendal-java package and learn to use it in their IDE by using ctrl+click to access our API documentation, similar to what we do for Rust core.

[AJIOB]

@Xuanwo, I think it will be more convenient to create code generation via rustc itself: just use the rust macros, similar to Serialize from serde, for example. It will remove useless python dependency for code compiling.

Or it is not possible for this case?

[Xuanwo]

just use the rust macros

Hi, @AJIOB. How can we achieve the goal of making it available to both Python and Java? I don't think it's currently possible. I mean, it's not just about the code itself, but also the documentation and API comments.