search

pola-rs / r-polars

Bring polars to R
https://pola-rs.github.io/r-polars/
Other
405 stars 35 forks source link

r-polars long-term support #367

Open sorhawell opened 9 months ago

sorhawell commented 9 months ago

First of all many thanks!

I will stop main developing r-polars from the 1st September, because I personally need to start up another project. It has been 17 great months nearly full-time and I have learned so much, and I have gotten tremendous help and feedback.

For r-polars to keep up with py-polars / rust-polars breaking changes, it will likely require 2-10 rust/R developers. I have been working hard to "finish" r-polars, but I must admit that seems to be full time job onging. The py-polars is far from a thin wrapper and has about ~50-100k lines of code. r-polars is basically a re-implementation of py-polars, and every time a bug is fixed in py-polars it should be fixed in r-polars also. The rust-polars breaking changes are demanding too, but easier to deal with for a rust developer, as the rust compiler often clearly suggest what needs to be updated.

I promise to patch very critical errors and future R versions. Isolated bugs in various methods will not be fixed regularly by me at least. The r-polars project does have 5 admins and 15 contributors. Feel free to reach out to me, if you want to work on/with r-polars in some sense.

Thank you for so much help ! <3 @eitsupi , @etiennebacher , @vincentarelbundock , @Sicheng-Pan @grantmcdermott @yutannihilation @eddelbuettel @jeroen @paleolimbot @xendai66 @ritchie46 extendr-community: "iliak" "mosa" "multimeric", "barry" "daniellga" and more

paleolimbot commented 9 months ago

Thank you so much for your work on this! Polars is a fantastic engine and it's so exciting that R users will be able to benefit from the past and future work of polars and py-polars developers (not to mention the improvements in the rust--R bindings community resulting from your limit testing here!). Feel free to give me a mention if there is an issue where you think I may be able to help...I can't do much with Rust (yet!) but there may be other parts where I can help.

I'm not sure if you have time to do it in the next 3 days, but if you're willing to move any/all of your Polars--R vector conversion to use nanoarrow::as_nanoarrow_array() and/or nanoarrow::convert_array_stream(), I do have plans (and scheduled paid time) to make continuous improvements to those conversions. Similarly, if there are any plans to implement/improve the SQL in -> ArrowArrayStream out workflow, it might be possible to (optionally) use dbplyr to generate SQL and reduce the R-specific implementation surface.

etiennebacher commented 9 months ago

Thanks a lot @sorhawell ! I'm still motivated to help developing r-polars, but I'll only be able to do it on the R side. I can work on docs, cleaning the internal code and possibly adding new features/arguments as long as it's mostly following the Python implementation. But I don't have the skills to implement full new functions in Rust or updating the rust-polars crate as you do in #334.

vincentarelbundock commented 9 months ago

Thanks for all your work on this amazing package. I really appreciate all the hours you put it to help analysts!

xendai66 commented 9 months ago

Thanks a lot @sorhawell , I've been a bit busy with exams lately, but I'm excited to dive back into contributing and learning with you all! I'd love to help out with any tasks that align with my skills and abilities. If there's anything that you think I could assist with, please let me know and I'll do my best to make it happen.

grantmcdermott commented 8 months ago

Sorry, I'm late to this. But just to add my own note of gratitude for all the work that you've put in @sorhawell. Many thanks!

(I'm in a similar boat to several other people, insofar as my Rust skills are (currently) a bit lacking to promise much upside on the main codebase. But I'll certainly try to keep contributing where and when I can.)

eitsupi commented 6 months ago

if there are any plans to implement/improve the SQL in -> ArrowArrayStream out workflow, it might be possible to (optionally) use dbplyr to generate SQL and reduce the R-specific implementation surface.

A separate package was created to use Polars SQLContext via the DBI package (and dbplyr). https://github.com/rpolars/r-polarssql

It is written in pure R and does not require any Rust (It is sufficient to install the polars package from R-universe). Contributions are welcome.

eitsupi commented 4 months ago

https://github.com/pola-rs/r-polars/pull/693#issuecomment-1901283668

eitsupi commented 1 month ago

Just a note: Since there are only two people doing substantial maintenance these days, @etiennebacher and me, and getting one person's approval at merge time is burdensome, I have changed the repository settings so that we can bypass the mandatory approvals.