abstractqqq
commented
1 month ago Hey thank you for the interest!
This is actually my post in our public discord:
Some random rambles :
I am currently integrating my own kdtree into the package, instead of using the a third party rust crate. Took me 2 days to fix a segfault bug which happens in very rare cases....
The kdtree re-impl is taking very long. But I want to do it because Kdtrees are used in all kinds of exact KNN searches, and entropy calculations. The reason I want to develop my own implementation is simple: better speed + easier to extend + I can learn how to run approximate vector search along the process, which can be hugely useful. I am now also very open to the idea of providing basic models outside Polars DataFrames, which work on any 2d matrices. If we can make vector search in higher dimensions quick (approximate method, in-memory use case), oh wow the implication will be huge.. All that fun stuff keeps my mind bouncing...
Since I will be making models now, this opens up tons of new items to work on. For starters, some users have asked me to implement variations of linear regression, e.g. ridge, l2 regularized, etc., and a rolling regression engine. I am aware this is extremely useful for the econometrics and finance folks. I am still a little torn on this because polars_ols provides exactly this. The real problem is whether we should consolidate some efforts within the polars plugin community, which is a harder topic to pursue.
K-means has always been on my list but I haven't had enough time to implement. Actually, in my limited experience, I had better results with k-medoids which is a similar method and which doesn't rely on properties of Euclidean geometry. There is an excellent Rust crate on that.
The more I keep imagining the future the more I cannot fall asleep.. The future seems bright, but there are a lot of uncertainties in life too... Regarding marketing, I have tried some linkedin posts in earliers in the year and got some stars. I had terrible experience with reddit and X before, and thus I am not actively promoting the project on those platforms. I used to be more blunt and like using strong words. When I voice some of my unconventional opinions on programming, e.g. OOP not good for scientific computing, I got some terrible comments. Still learning how to navigate the online tech world.
With current Polars-ds, you can actually do MRMR feature selection with many options for correlation (4 different correlations readily available in the package rn). I cannot publish it because it is used in my company. But if you look up MRMR feature selection online, the logic isn't hard to implement.
Traditional ML pipelines are also available. But it strictly only applies to data transformations before being consumed by a model. Personally I think those two steps should be separate. You can find more in examples/.
PCA is also available in the package if you are not aware. See query_pca and query_principal_components.
Still a lot of work to do. Right now the focus is on regression. I am implementing Lasso with coordinate descent. Let's see how that one goes.
:)
firmai
Always funny how these projects start, one goes from like 100 users that understand the tremendous opportunity of fast in-memory computing --- and then 2-3 years later 10 million people heavily rely on your solutions.
There is a lot of potential in your project, it lacks a name, clarity (too much text on readme), marketing, and objectives. I think polars extensions and libraries built ontop of polars is the future of data science for the next 10 years.
Similar to rust-ml/linfa, I think you should aim for something much more grand. Why not search active open-source sponsorship, corporate or otherwise.
There is currently no polars sklearn. How could this be?. It is currently the most obvious missing link in data science. It is for a DS geek a wheeled suitcase moment - "We put wheels on bags after we put a man on the moon".
We need clustering, we need dimensionality reduction, feature selection, pairwise interactions, we need a polars machine learning and data-science project worth throwing yourself behind. Everybody is so busy creating LLMs to generate chicken casserol recipes, nobody is doing any actual work.
What are the hundreds and thousands of data scientists doing in their free time? Thanks for picking up the slack, you will be tomorrow's heros.