Create `HudiTableFactory` implementing DataFusions `TableProviderFactory`

[matthewmturner]

Is there an existing issue for this?

• [X] I have searched the existing issues

Description of the bug

This isnt a bug report - its a feature request but i didnt see a way to submit a feature request.

I would like to be able to register hudi tables with datafusion like so:

CREATE EXTERNAL TABLE my_table STORED AS HUDITABLE LOCATION '/path/to/table';

If hudi-rs provided a TableProviderFactory then we could register and use that (This is how i currently register deltalake tables and I would like to do something similar for hudi).

Steps To Reproduce

Not a bug

Expected behavior

I can register tables to datafusion like so:

CREATE EXTERNAL TABLE my_table STORED AS HUDITABLE LOCATION '/path/to/table';

Screenshots / Logs

No response

Software information

N/A

Additional context

No response

[xushiyan]

@matthewmturner thanks for raising this! would you be able to review the linked PR by @kazdy please?

[matthewmturner]

@xushiyan sure, checking it out

[matthewmturner]

Looks good but when i try integrating in my app i get the below error (which isnt surprising since we arent on the same version of datafusion). Of course thats not a blocker for this though :). Excited to see this

[xushiyan]

@matthewmturner we just landed this feature in main. can you pls test it out with your PR? note that we choose HUDI as the name to stay consistent. See https://github.com/apache/hudi-rs/pull/162#discussion_r1799914280. thanks.

[matthewmturner]

@xushiyan sure will do, do you have any recommendations for what test data to use? i just need something simple to test a basic query

[xushiyan]

@matthewmturner yes we keep a bunch of zipped test tables in https://github.com/apache/hudi-rs/tree/main/crates/tests/data/tables

you can unzip and use them directly.

[matthewmturner]

Okay, going to work on this tonight / tomorrow. I may ping you for a review if you dont mind.

[matthewmturner]

I must be doing something wrong, im getting error that hoodie.properties doesnt exist even though it does.

will pick this up tomorrow.

[kazdy]

Hi Matt,

I cloned your branch and got this:

Can it be something related to your file permissions maybe? edit: permissions look ok and same as mine

[matthewmturner]

sigh i got it to work. the repo used to be called datafusion-tui and i still have it as that locally pointing to datafusion-dft so my path was wrong.

[matthewmturner]

and now test passes :)