search

activeloopai / deeplake

Database for AI. Store Vectors, Images, Texts, Videos, etc. Use with LLMs/LangChain. Store, query, version, & visualize any AI data. Stream data in real-time to PyTorch/TensorFlow. https://activeloop.ai
https://activeloop.ai
Apache License 2.0
8.2k stars 631 forks source link

[FEATURE] Open source the C++ backend and Python package build process #2985

Open zrnsm opened 2 weeks ago

zrnsm commented 2 weeks ago

Description

It seems like the current state of affairs with the 4.0.1 release on PyPi is that the backend is locked in a compiled shared library that simply must be trusted. Is the strategy going forward to keep that module closed source?

Use Cases

No response

khustup2 commented 2 weeks ago

Hi @zrnsm thanks for raising this question. The process of open sourcing the C++ backend is ongoing. We will keep you posted once it's done.

I'm curious, if you are interested in contributing to the C++ codebase?

Thanks.

zrnsm commented 2 weeks ago

@khustup2 I'd definitely be interested in contributing if deeplake works for my use case. I'm still just evaluating some options in this space.

How is the binding implemented? pybind11?

khustup2 commented 2 weeks ago

Great! We will keep you posted. Hopefully deeplake will work on your use case. In case it's not, please let us know what didn't go well.

Yes, bindings are implemented using pybind11.