fix: register object store with datafusion

apache / hudi-rs

A native Rust library for Apache Hudi, with bindings into Python

https://hudi.apache.org/

Apache License 2.0

149 stars 30 forks source link

fix: register object store with datafusion #107

Closed abyssnlp closed 3 months ago

abyssnlp commented 3 months ago

Description

Reading Hudi tables from an object store, s3 in this case, was failing due to the object store not being registered with Datafusion.

Before the changes in this PR, the following error was encountered:

Error: Internal("No suitable object store found for s3://test-hudi-rs/. See `RuntimeEnv::register_object_store`")

Datafusion requires the object store to be registered. Refer here.

Note

Storage and FileSystemView have public struct fields object_store and storage. There should be a better way to expose these to the datafusion crate to be able to register them.

How are the changes test-covered

[x] N/A
[ ] Automated tests (unit and/or integration tests)
[ ] Manual tests
- [ ] Details are described below

abyssnlp commented 3 months ago

This change should make it okay to proceed with #81

abyssnlp commented 3 months ago

Possible fix for #108

codecov[bot] commented 3 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 87.42%. Comparing base (b3ffa32) to head (ea9726c).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #107 +/- ## ========================================== + Coverage 87.24% 87.42% +0.18% ========================================== Files 14 14 Lines 690 700 +10 ========================================== + Hits 602 612 +10 Misses 88 88 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

xushiyan commented 3 months ago

@abyssnlp thanks for the fix. refactored a bit. can you verify this change e2e in your env please? would love to unblock the minio integ test!

abyssnlp commented 3 months ago

@xushiyan I verified it for 3 scenarios and it works as expected.

HudiDataSource with AWS S3 (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY , AWS_DEFAULT_REGION in the env vars)
HudiDataSource with local FS

HudiDataSource with minio (requires the AWS env var AWS_ENDPOINT and ALLOW_HTTP)

let hudi = HudiDataSource::new_with_options(
    "s3://test-bucket/v6_nonpartitioned",
    vec![
        ("allow_http", "true"),
    ],
)
.await?;