mratsim
commented
6 years ago Placeholder.
To avoid polluting this meta-thread with specific discussion on certain topics (say what I want in the random library), this will link to the discussion topics:
Multidimensional arrays, Linear-algebra
14, #17, #25, #50, #59
Plotting
17, #51, #70
Geospatial
13, #69
Image processing
69
Dataframes, columnar/tabular data processing
20, #47, #33
Random
40
Statistics
16
Machine learning
48
Deep learning
No issue open
andreaferretti
dom96
sdwfrost
Vindaar
narimiran
EelcoHoogendoorn
brentp
abudden
Araq
retsyo
sinkingsugar
ihendley
mantielero
stisa
jblindsay
lbartoletti
planetis-m
This is a meta-issue to keep track of discussion around Nim scientific libraries.
Primitive libraries
Decimal128: https://github.com/JohnAD/decimal128 Fixed-point: https://gitlab.com/lbartoletti/fpn
Multidimensional arrays, Linear-algebra
Multidimensional arrays are the basic block of scientific computing, it goes beyond the 2D or 3D vectors and matrices. Notable non-Nim implementations include Fortran, Julia, Matlab and Numpy.
Status: in-progress Libraries:
Support
Arraymancer supports dense multidimensional arrays of any type, on CPU (integers, floats, complex), Cuda and OpenCL (float only) and uses BLAS, CuBLAS and Clblast under the hood.
Flambeau is provide libtorch bindings and reproduces PyTorch functionality.
Manu is a pure Nim matrix library with no external dependencies
Neo supports dense and sparse float vectors and matrices, on CPU and Cuda (Nvidia GPUs) and also uses BLAS and LAPACK under the hood.
Status: stalled Libraries:
NimTorch supports most PyTorch features regarding multidimensional arrays, on CPU, Cuda, OpenCL and AMD ROCm provided you compiled PyTorch's Aten backend with the relevant features.
Plotting
Data analysis requires plotting, notable non-Nim implementations include Python matplotlib and seaborn, Plot.ly (Python, R, Javascript), R ggplot2, Matlab and Facebook Visdom (a simple interface to Plot.ly).
Note that there are a couple of approach to plotting, either having a charting library or having a high-level grammar library (similar to SQL) that hides low-level details of a chart.
Status: in-progress Libraries:
Proof-of-concepts:
Unmaintained:
ggplotnim is an implementation in pure Nim of the graphics of grammar. gnuplot.nim is a wrapper of gnuplot. Nim-Plotly uses the plot.ly charting library as a backend. Both MetaPlot and Monocle uses the Vega visualization grammar.
Image processing library
Computer vision is a thriving area of research. Vision scientists needs algorithms that works on images represented as a multidimensional arrays (different from say Photoshop), preferably multithreaded and GPU accelerated.
Notable non-Nim libraries include OpenCV, Matlab, Python scikit-image, scipy.ndimage and mahotas.
Status: in-progress
Libraries:
Unmaintained:
Nim-opencv provides rough low-level bindings of OpenCV functions.
Dataframe and columnar/tabular data processing
Dataframes are essential to process structured data (say Name, Age, number of products bought, last time of visit). They allow very efficient data manipulation, including easily creating new columns, joining dataframes, converting between types.
Notable non-Nim packages include Python Pandas and R datatable. When data does not fit in RAM, dataframe packages are interfaced with SQL or HDF5 datastores or even Spark for very large scale processing.
Status: in-progress Libaries:
Random library
Lots of scientific algorithms rely on stochastic processes or random distribution. At the very least pseudo-random generator that samples from a normal/gaussian distribution is needed.
Notable non-Nim library include Scipy
Status: in-progress Libraries:
Statistics library
Notable language: R
Status: standard lib statistics module
Machine learning
Machine learning is how to teach a computer to learn/generalize patterns from data.
Notable non-Nim libraries include: Python's Scikit-Learn and R's Caret. State-of-the-art C++ library to wrap: XGBoost
Status: in-progress
Deep learning & neural network.
Deep learning is machine learning with neural networks and arguably eating the world (or atleast Reddit, Hacker News and sponsors). In comparison to most traditional machine learning tools, neural networks can also learn very well from non-structured data (images, sounds, text ...).
Notable non-Nim libraries include: Facebook Torch, Google Tensorflow, Apache and Amazon Mxnet
Status: in-progress Libraries:
Proof-of-concept:
Non-linear optimization
Status: in-progress Libraries:
Linear programming
Status: in-progress Libraries:
Computational Physics
Status: in-progress Libraries:
Geometry
Computational geometry also require tuned algorithms for: geometry primitives, polygons and polyhedron, triangulations, distances, shape analysis ...
Noteable non-Nim library: CGAL
Status: no library
Scientific serialization format
There are many formats specific to science ot even science domains.
Libraries:
Geospatial library
Often scientist needs to deal with geospatial coordinate (latitude, longitude), maps and distances. This include efficient data-structures like KD-Tree or RTree to compute distances between points and distance formulas like Haversine to compute distance on a sphere.
Notable non-Nim libraries include Python's scipy.spatial, Geopy, Shapely
Status: in-progress R-tree forum thread.
Proof-of-concepts:
Scientific language bindings
Python:
Unmaintained