search

splitgraph / seafowl

Analytical database for data-driven Web applications 🪶
https://seafowl.io
Apache License 2.0
386 stars 9 forks source link

`datafusion-remote-tables` crate #456

Open ccollie opened 11 months ago

ccollie commented 11 months ago

According to the blog post here, datafusion-remote-tables was released separately as a crate, but I don't see it on crates.io

gruuya commented 11 months ago

Good question. My idea was to first make it available independently of Seafowl, which was done by separating it into a standalone crate. You should be able to add it to your project by specifying a dependency that targets this git repo (or a clone of it): https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#specifying-dependencies-from-git-repositories. Does that work for you?

As for publishing, I was waiting for connector-x (the central dependency of datafusion-remote-tables) to do a proper release (but that didn't happen for quite some time, hence why we have a git dependency of our own here). The reason for this is that one can't publish a package with git dependencies if I'm not mistaken.

Consequently, if you intend to publish a package based on datafusion-remote-tables itself than the above method won't work for you. If that is the case we can try and see when the maintainers of connector-x think they can do a next release.

ccollie commented 11 months ago

Thanks for the quick response.

Since i intend to publish my project on crates.io, a git dependency won't work. I appreciate the option of reaching out to the connector-x folk, otherwise I can temporarily copy as a module locally until there is a release.

gruuya commented 11 months ago

@ccollie sure thing; I opened a PR to bump the dependencies over at connector-x: https://github.com/sfu-db/connector-x/pull/518

Hopefully this can result in a new connector-x release.

ccollie commented 11 months ago

BTW - my use-case is a promql compatible execution engine that uses traits for data access. datafusion-remote-tables would go a long way in making it useful.