java: Add spring-boot-starter-opendal

Xuanwo commented 3 months ago

Spring is a crucial framework in the Java ecosystem. It would be beneficial for OpenDAL to develop a spring-boot-starter-opendal, enabling Java users to seamlessly integrate OpenDAL immediately.

Notes

We can place spring-boot-starter-opendal under integrations/spring-boot.

shoothzj commented 3 months ago

I can deep in that next week.

Xuanwo commented 3 months ago

I can deep in that next week.

Thanks a lot!

shoothzj commented 2 months ago

@Xuanwo Sorry for my late, recently I was kind of busy. But I still want to work on this :). I think in the short term, we will not be putting the spring starter opendal into the Spring repository. Therefore, we are facing the issue of maintaining OpenDAL with multiple versions of Spring in the OpenDAL repository. I suggest the following strategy:

opendal-spring-xx will adapt to the corresponding Spring Boot version. For example, opendal-33 will adapt to Spring Boot 3.3:

<dependency>
  <groupId>org.apache.opendal</groupId>
  <artifactId>opendal-spring-33</artifactId>
  <version>${latest.version}</version>
</dependency>

<dependency>
  <groupId>org.apache.opendal</groupId>
  <artifactId>opendal-spring-27</artifactId>
  <version>${latest.version}</version>
</dependency>

opendal-spring-boot-starter will by default depend on the adapted latest version of Spring Boot. If you need to run it in a lower version of a Spring Boot project, you can replace it in the following way:

<dependency>
  <groupId>org.apache.opendal</groupId>
  <artifactId>opendal-spring-boot-starter</artifactId>
  <version>${latest.version}</version>
  <exclusions>
      <exclusion>
          <groupId>org.apache.opendal</groupId>
          <artifactId>opendal-spring-33</artifactId>
      </exclusion>
  </exclusions>
</dependency>
<dependency>
  <groupId>org.apache.opendal</groupId>
  <artifactId>opendal-spring-27</artifactId>
  <version>${latest.version}</version>
</dependency>

Xuanwo commented 2 months ago

Hi @shoothzj, thanks for your efforts. However, I feel targeting multiple Spring versions might be too complex for a starter proejct. I'm concerned about its future maintainability. Perhaps we could focus on the latest version and provide guidance for older versions?

shoothzj commented 2 months ago

@Xuanwo I agree, we can focus on 3.3.x version.

shoothzj commented 1 month ago

@Xuanwo, SpringBoot now supports two types of starters: reactive for asynchronous operations and non-reactive for synchronous operations. Let's look at a Redis example:

spring-data-redis contains the core logic.
spring-boot-starter-redis includes auto-configuration for Spring WebMVC (synchronous).
spring-boot-starter-redis-reactive includes auto-configuration for Spring WebFlux (asynchronous).

If a project is popular, the spring-boot-autoconfigure repository will include auto-configuration in their main repo, and spring-boot-starter-redis and spring-boot-starter-redis-reactive can be empty or non-existent.

I think we should first maintain these three sub-projects with the latest SpringBoot version in our repo.

Regarding the artifact ID, I prefer using "opendal" as the suffix, which is common in the Spring ecosystem. What do you think?

Opendal as Prefix

opendal-spring
opendal-spring-starter
opendal-spring-starter-reactor

Opendal as Suffix

spring-opendal
spring-starter-opendal
spring-starter-opendal-reactor

Xuanwo commented 1 month ago

I'm happy to continue using the common behavior in the ecosystem. spring-opendal looks fine to me.

shoothzj commented 1 month ago

@Xuanwo I put some sample codes in https://gitcode.com/shoothzj/opendal-demo/overview. And here is some docs, would you please take a look?

Usage

Configuration

First, configure the application.yml file with the necessary OpenDAL settings:

spring:
  opendal:
    schema: "fs"
    conf:
      root: "/tmp"

Code Integration

Features

OpenDAL/OpenDALReactive Bean: Easily operate on data using OpenDAL/OpenDALReactive.
Auto Serialize/Deserialize Data: Leverage Jackson for automatic serialization and deserialization.

I plan to implement Auto Serialize/Deserialize Data in the next.

Implementation

Spring WebMVC

To use OpenDAL with Spring WebMVC:

Configure the application.yml file as shown above.
Autowire the OpenDAL bean in your controller.

import org.apache.opendal.Entry;
import org.apache.opendal.Metadata;
import org.apache.opendal.PresignedRequest;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;

import java.time.Duration;
import java.util.List;

@RestController
@RequestMapping("/api/v1/opendal")
public class OpenDALController {

    @Autowired
    private OpenDAL openDAL;

    @PostMapping("/write")
    public ResponseEntity<Void> write(@RequestParam String path, @RequestBody String content) {
        openDAL.write(path, content);
        return ResponseEntity.ok().build();
    }

    @PostMapping("/writeBytes")
    public ResponseEntity<Void> writeBytes(@RequestParam String path, @RequestBody byte[] content) {
        openDAL.write(path, content);
        return ResponseEntity.ok().build();
    }

    @PostMapping("/append")
    public ResponseEntity<Void> append(@RequestParam String path, @RequestBody String content) {
        openDAL.append(path, content);
        return ResponseEntity.ok().build();
    }

    @PostMapping("/appendBytes")
    public ResponseEntity<Void> appendBytes(@RequestParam String path, @RequestBody byte[] content) {
        openDAL.append(path, content);
        return ResponseEntity.ok().build();
    }

    @GetMapping("/stat")
    public ResponseEntity<Metadata> stat(@RequestParam String path) {
        Metadata metadata = openDAL.stat(path);
        return ResponseEntity.ok(metadata);
    }

    @GetMapping("/read")
    public ResponseEntity<byte[]> read(@RequestParam String path) {
        byte[] content = openDAL.read(path);
        return ResponseEntity.ok(content);
    }

    @GetMapping("/presignRead")
    public ResponseEntity<PresignedRequest> presignRead(@RequestParam String path, @RequestParam long duration) {
        PresignedRequest request = openDAL.presignRead(path, Duration.ofSeconds(duration));
        return ResponseEntity.ok(request);
    }

    @GetMapping("/presignWrite")
    public ResponseEntity<PresignedRequest> presignWrite(@RequestParam String path, @RequestParam long duration) {
        PresignedRequest request = openDAL.presignWrite(path, Duration.ofSeconds(duration));
        return ResponseEntity.ok(request);
    }

    @GetMapping("/presignStat")
    public ResponseEntity<PresignedRequest> presignStat(@RequestParam String path, @RequestParam long duration) {
        PresignedRequest request = openDAL.presignStat(path, Duration.ofSeconds(duration));
        return ResponseEntity.ok(request);
    }

    @DeleteMapping("/delete")
    public ResponseEntity<Void> delete(@RequestParam String path) {
        openDAL.delete(path);
        return ResponseEntity.ok().build();
    }

    @PostMapping("/createDir")
    public ResponseEntity<Void> createDir(@RequestParam String path) {
        openDAL.createDir(path);
        return ResponseEntity.ok().build();
    }

    @PostMapping("/copy")
    public ResponseEntity<Void> copy(@RequestParam String sourcePath, @RequestParam String targetPath) {
        openDAL.copy(sourcePath, targetPath);
        return ResponseEntity.ok().build();
    }

    @PostMapping("/rename")
    public ResponseEntity<Void> rename(@RequestParam String sourcePath, @RequestParam String targetPath) {
        openDAL.rename(sourcePath, targetPath);
        return ResponseEntity.ok().build();
    }

    @DeleteMapping("/removeAll")
    public ResponseEntity<Void> removeAll(@RequestParam String path) {
        openDAL.removeAll(path);
        return ResponseEntity.ok().build();
    }

    @GetMapping("/list")
    public ResponseEntity<List<Entry>> list(@RequestParam String path) {
        List<Entry> entries = openDAL.list(path);
        return ResponseEntity.ok(entries);
    }
}

Spring WebFlux

To use OpenDALReactive with Spring WebFlux:

Configure the application.yml file as shown above.
Autowire the OpenDALReactive bean in your controller.

import com.shoothzj.opendal.spring.OpenDALReactive;
import org.apache.opendal.Entry;
import org.apache.opendal.Metadata;
import org.apache.opendal.PresignedRequest;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;

import java.time.Duration;

@RestController
@RequestMapping("/api/v1/opendal/reactive")
public class OpenDALReactiveController {

    @Autowired
    private OpenDALReactive<byte[]> openDALReactive;

    @PostMapping("/write")
    public Mono<ResponseEntity<Void>> write(@RequestParam("path") String path, @RequestBody String content) {
        return openDALReactive.write(path, content).then(Mono.just(ResponseEntity.ok().<Void>build()));
    }

    @PostMapping("/writeBytes")
    public Mono<ResponseEntity<Void>> writeBytes(@RequestParam("path") String path, @RequestBody byte[] content) {
        return openDALReactive.write(path, content).then(Mono.just(ResponseEntity.ok().<Void>build()));
    }

    @PostMapping("/append")
    public Mono<ResponseEntity<Void>> append(@RequestParam("path") String path, @RequestBody String content) {
        return openDALReactive.append(path, content).then(Mono.just(ResponseEntity.ok().<Void>build()));
    }

    @PostMapping("/appendBytes")
    public Mono<ResponseEntity<Void>> appendBytes(@RequestParam("path") String path, @RequestBody byte[] content) {
        return openDALReactive.append(path, content).then(Mono.just(ResponseEntity.ok().<Void>build()));
    }

    @GetMapping("/stat")
    public Mono<ResponseEntity<Metadata>> stat(@RequestParam("path") String path) {
        return openDALReactive.stat(path).map(ResponseEntity::ok);
    }

    @GetMapping("/read")
    public Mono<ResponseEntity<byte[]>> read(@RequestParam("path") String path) {
        return openDALReactive.read(path).map(ResponseEntity::ok);
    }

    @GetMapping("/presignRead")
    public Mono<ResponseEntity<PresignedRequest>> presignRead(@RequestParam("path") String path, @RequestParam("duration") long duration) {
        return openDALReactive.presignRead(path, Duration.ofSeconds(duration)).map(ResponseEntity::ok);
    }

    @GetMapping("/presignWrite")
    public Mono<ResponseEntity<PresignedRequest>> presignWrite(@RequestParam("path") String path, @RequestParam("duration") long duration) {
        return openDALReactive.presignWrite(path, Duration.ofSeconds(duration)).map(ResponseEntity::ok);
    }

    @GetMapping("/presignStat")
    public Mono<ResponseEntity<PresignedRequest>> presignStat(@RequestParam("path") String path, @RequestParam("duration") long duration) {
        return openDALReactive.presignStat(path, Duration.ofSeconds(duration)).map(ResponseEntity::ok);
    }

    @DeleteMapping("/delete")
    public Mono<ResponseEntity<Void>> delete(@RequestParam("path") String path) {
        return openDALReactive.delete(path).then(Mono.just(ResponseEntity.ok().<Void>build()));
    }

    @PostMapping("/createDir")
    public Mono<ResponseEntity<Void>> createDir(@RequestParam("path") String path) {
        return openDALReactive.createDir(path).then(Mono.just(ResponseEntity.ok().<Void>build()));
    }

    @PostMapping("/copy")
    public Mono<ResponseEntity<Void>> copy(@RequestParam("sourcePath") String sourcePath, @RequestParam("targetPath") String targetPath) {
        return openDALReactive.copy(sourcePath, targetPath).then(Mono.just(ResponseEntity.ok().<Void>build()));
    }

    @PostMapping("/rename")
    public Mono<ResponseEntity<Void>> rename(@RequestParam("sourcePath") String sourcePath, @RequestParam("targetPath") String targetPath) {
        return openDALReactive.rename(sourcePath, targetPath).then(Mono.just(ResponseEntity.ok().<Void>build()));
    }

    @DeleteMapping("/removeAll")
    public Mono<ResponseEntity<Void>> removeAll(@RequestParam("path") String path) {
        return openDALReactive.removeAll(path).then(Mono.just(ResponseEntity.ok().<Void>build()));
    }

    @GetMapping("/list")
    public Flux<Entry> list(@RequestParam("path") String path) {
        return openDALReactive.list(path);
    }
}

Testing with `curl`

Use the following curl commands to test the endpoints of your Spring WebMVC and WebFlux controllers:

Write String Content

curl -X POST "http://localhost:8080/api/v1/opendal/write?path=/example/path" -H "Content-Type: application/json" -d "This is a test content"

Write Byte Content


curl -X POST "http://localhost:8080/api/v1/opendal/writeBytes?path=/example/path

Append String Content

curl -X POST "http://localhost:8080/api/v1/opendal/append?path=/example/path" -H "Content-Type: application/json" -d "This is additional content"

Append Byte Content

curl -X POST "http://localhost:8080/api/v1/opendal/appendBytes?path=/example/path" -H "Content-Type: application/octet-stream" --data-binary @/path/to/your/file

Stat

curl -X GET "http://localhost:8080/api/v1/opendal/stat?path=/example/path"

Read

curl -X GET "http://localhost:8080/api/v1/opendal/read?path=/example/path"

Presign Read

curl -X GET "http://localhost:8080/api/v1/opendal/presignRead?path=/example/path&duration=3600"

Presign Write

curl -X GET "http://localhost:8080/api/v1/opendal/presignWrite?path=/example/path&duration=3600"

Presign Stat

curl -X GET "http://localhost:8080/api/v1/opendal/presignStat?path=/example/path&duration=3600"

Delete

curl -X DELETE "http://localhost:8080/api/v1/opendal/delete?path=/example/path"

Create Directory

curl -X POST "http://localhost:8080/api/v1/opendal/createDir?path=/example/path"

Copy

curl -X POST "http://localhost:8080/api/v1/opendal/copy?sourcePath=/example/source&targetPath=/example/target"

Rename

curl -X POST "http://localhost:8080/api/v1/opendal/rename?sourcePath=/example/source&targetPath=/example/target"

Remove All

curl -X DELETE "http://localhost:8080/api/v1/opendal/removeAll?path=/example/path"

List

curl -X GET "http://localhost:8080/api/v1/opendal/list?path=/example/path"

Xuanwo commented 1 month ago

Hi, @tisonkun, would you like to take a look? I'm not a Java expert like you are.

Xuanwo commented 1 month ago

Some ideas in my mind for now:

We don't need to seperate writeString and writeBytes, we only need to have write which accept bytes.
We should remove append, it's an option for write
I don't need we need to play with presign at current stage.

tisonkun commented 1 month ago

Hi, @tisonkun, would you like to take a look? I'm not a Java expert like you are.

Looks like a good start. Perhaps create a repo for this starter and we can collaborate as normal issues + PRs.

I don't think we need to include these code into the main repo since it would cause extra release burden. Let @shoothzj be the maintainer of the starters and modifying anything necessary in binding-java should be fine.

shoothzj commented 1 month ago

We don't need to seperate writeString and writeBytes, we only need to have write which accept bytes.

We can only maintain writeBytes now, Java has powerful overload and generic, we can support write(Object obj) leverage jackson's serial.

We should remove append, it's an option for write

I need take a deep look, but why the java binding expose this method?

I don't need we need to play with presign at current stage. Ok.

shoothzj commented 1 month ago

Hi, @tisonkun, would you like to take a look? I'm not a Java expert like you are.

Looks like a good start. Perhaps create a repo for this starter and we can collaborate as normal issues + PRs.

I don't think we need to include these code into the main repo since it would cause extra release burden. Let @shoothzj be the maintainer of the starters and modifying anything necessary in binding-java should be fine.

I would be happy to help maintain the starters. :)

Xuanwo commented 1 month ago

Hi, @shoothzj, the subproject for spring-opendal has been started. Would you like to create a tracking issue for it?

shoothzj commented 1 month ago

@Xuanwo https://github.com/apache/opendal/issues/5015

Xuanwo commented 1 month ago

Thanks!

apache / opendal