A library like Scipy

[johnjansen]

https://www.scipy.org

[marceloboeira]

@johnjansen 👍

[konovod]

It would be great imho, as Crystal's fast speed and agile syntax could make it pretty successful in scientific programming. I see two main problems\questions though:

• Scipy is a wrapper around a number of C and Fortran libraries. This would make crystal analogue pretty hard to distribute - instead of just installing shard you have to download binaries for your system or use a fortran compiler to build them from source. Of course, pure crystal implementation is more convenient, but given how much lines of code it will require and how much efforts to optimization was spent on Scipy it doesn't looks possible in near future.
• Scipy is a big library that handles many areas. In Ruby world (and in Crystal so far), single-purposed shards are used more often. Both approaches have it's pros and cons. Personally i would prefer installing single library instead of seeking for shards, it is easier to maintain and better from point of PR ("XX is a Scipy for Crystal"). But it have important drawback - Crystal compiler isn't very fast, and using the big library can increase compile times dramatically. Clever construction of library (requiring only needed files instead of all at once) and incremetal compilation could somewhat mitigate this, but it's still a drawback. So some kind of decision should be made - either (most of) community wants a big library or a specific shards for each area\problem. In a latter case no action is needed, just more shards that wraps libraries and provide high-level interface. In a former case community-maintained library should be created imho, so modules that duplicate scipy functionality could be eventually contributed by different people.

(all of that is a diletant point of view, I could be totally wrong and don't have much experience neither in scipy nor in scientific programming).

[RX14]

@konovod Actually requiring a large shard shouldn't slow down the crystal compiler much if at all. The speed of the compiler is proportional to the amount of code in use: methods that aren't called and code that isn't used barely have a cost overhead because they are never typed.

I haven't actually benchmarked this at all so take it with a grain of salt but i'm 90% sure that this is how it works from looking at compiler internals.

[konovod]

Good to hear. I haven't benchmarked too, but type inference was my most concern. The compiler still have to parse all code - i've started creating wrapper for LAPACK and the direct conversion of c-header is 2300 lines, and that's just one library. But parsing is perhaps fast enough.

[RX14]

@konovod if you use --stats you'll find even parsing all_spec.cr takes a fraction of a second.

[konovod]

Ok, now i have another question. License. Is GPL acceptable for most potential users of "Crystal SciPy"?

If GPL is acceptable, wrapper around GSL can provide most of the features Scipy has - optimization, ode, integrators, statistics, sparse linalg, splines etc etc. I've no idea about performance, but it can be improved later. There is also GLPK for linear programming and possibly more libraries.

If GPL is too strict and license should be close to SciPy - things will be more complex (but still solvable. after all, scipy itself has permissive license). Personally, i don't have anything against GPL (i'm closer to gpl fan actually), but if this will severely reduce popularity of the library, there is a little point to use it. Maybe this issue is not a best place to discuss it, but i don't know where to go with it.

[konovod]

So, status report

• Linear algebra: My LAPACK wrapper has more than half functions that Scipy.linalg has(checklist), with an (imho) convenient matrix library that is close to Numpy but two-dimensional only. There is also https://github.com/Exilor/matrix by @Exilor, I'm going to grab everything good from it.
  • Linear programming: Looks like GLPK isn't the fastest LP solver in the OpenSource: according to http://plato.asu.edu/ftp/lpsimp.html CLP performs much better, almost at the level of commercial solvers. So i created wrapper for SYMPHONY for linear and mixed integer programming. It is pretty basic for now (lacks things like warmstarting), but should be fast enough for not-very-big tasks.
  • Looks like @ruivieira and @dylandy already started GSL bindings. Only Statistics and Matrix parts but it's the start. Well, my matrix is better (imho :laughing: ) but there is still statistics part with beautiful examples.

I'm open to suggestions what should be the next thing. Maybe Numpy analogue - multidimensional arrays with convenient DSL's. Or more complete GSL bindings (my favorite parts are root-finding\minimization and ode solvers, so i'll perhaps start with them). Or maybe sparce matrices - there is a SuiteSparse, i think i can try to create wrappers, but as i've no experience with sparce matrices i'm not sure what things are more important in it and what can be skipped.

[konovod]

Huh, just found there is also https://github.com/mverzilli/crystalla by @mverzilli. Looks like without centralized packages list people will keep reinventing bicycles again and again. If i've found it before I would prefer making PR's to it, as it doesn't looks like my one-man-hobby, but now my library has more features and i'm opinionated towards it.

On the GSL - my impression is that library has poor quality. Yes, it covers a lot of areas, but I've found a bug at the day I touched it, google shows that it is already reported but never fixed. Besides that, of three minimization algorithms one is slow, one is "default" and one fails to converge on simplest example. So i don't think wrapping it worth efforts. I've started instead wrapper of NLopt for nonlinear optimization.

[mverzilli]

Hey @konovod! As you might have noticed, it's been a while since I last worked on crystalla, and now that I'm a bit more involved in Crystal itself it is very unlikely that I'll dedicate time to it without a strong reason. Feel free to "steal" anything if you feel there's something useful. Otherwise, I'm super happy that someone is tackling this: I agree that Crystal is a super interesting fit for scientific applications. This post shows some of the potential: https://manas.tech/blog/2015/10/30/linear-algebra-in-crystal-from-lapack.html. Given most of the heavyweight was actually done by LAPACK/BLAS, those 10x improvements stem directly from the OH introduced by Crystal vs. Python, with comparable code "complexity".

[christopherzimmerman]

Just adding bottle here. I've only gotten to a few of scipy's features, but they are in the roadmap, and I do implement an N-Dimensional Tensor which is really useful.