search

yobix-ai / extractous

Fast and efficient unstructured data extraction. Written in Rust with bindings for many languages.
Apache License 2.0
416 stars 17 forks source link

Tika Metadata - HashMap Issue #24

Closed s4zuk3 closed 1 week ago

s4zuk3 commented 2 weeks ago

Hello! I've been a bit busy, but now I have time to finish this. This Draft PR is mainly to ask for help, as after trying many different configurations, I still can't get Rust to read the Java HashMap. I suspect there might be some missing configuration in the native image, but I can't figure out what it is. Do you know what I'm missing? This draft PR contains a small portion of the final PR, with an example where it’s not working (extractous-core/src/tika/jni_utils.rs ln124).

Thank you very much!

s4zuk3 commented 2 weeks ago

@nmammeri

nmammeri commented 2 weeks ago

Thanks again for you valuable contribution. Yes, indeed you need to have some entries in the graalVM reflection metada. We'll help you on this. @KapiWow is looking at integrating your pull request. I think he started rebasing your previous pull request. Let's wait for his commit.

KapiWow commented 2 weeks ago

Hi @s4zuk3, Jni config is missing some methods needed for HashMap. I updated config in separate branch, you can take a look. https://github.com/yobix-ai/extractous/commit/e5afe08f804632ce22981e3f90b46b1e2047e442 I hope it will work for you, I only have tested it locally for now. Feel free to ask if you have any other questions.

Please let us know if you have enough time to finish the metadata ticket or if we should finish it ourselves.

s4zuk3 commented 1 week ago

@KapiWow, thank you so much for the help! With those changes, everything works perfectly! I really appreciate your assistance. In a few hours, I’ll have the final PR ready for review. Thank you very much.

By the way, could you explain how you generated the changes for the JNI config files? I’ve tried using AgentNative but haven’t had any success.

nmammeri commented 1 week ago

@s4zuk3 it's great to that @KapiWow changes worked for you. Most of reflection data is generated automatically using the tracing agent, but sometimes it's required to add manual entries to the jni_config.json.

We have a separate repo with a lot of data files that try to exercise the dynamic paths as much as possible. We decided to have it in a separate repo to not bloat the extractous repository with loads of data files. The reflection data is mostly static and doesn't change much.

I can grant you access to this repo if you want to.

KapiWow commented 1 week ago

Actually I added it manually, for some reasons this is not captured by agent in config.