search

brightway-lca / brightway2-calc

The calculation engine for the Brightway2 life cycle assessment framework.
BSD 3-Clause "New" or "Revised" License
14 stars 16 forks source link

Investigate `scipy`/Apple Accelerate (for Sparse Solvers) #69

Open michaelweinold opened 1 year ago

michaelweinold commented 1 year ago

scipy in December 2018 dropped and in May 2021 re-introduced support for the Apple Accelerate library (compare also the SciPy roadmap and their GitHub repo wiki page).

This library

provides high-performance, energy-efficient computation on the CPU by leveraging its vector-processing capability.

It could therefore provide a highly optimized alternative to the current bw25nosolver workaround (compare the documentation).

Investigation

Implementation

Compare @cmutel's existing notebook "Compare speed of iterative solvers".

Documentation

Installation

Sparse Solvers

Discussion on Hackernews

michaelweinold commented 1 year ago

This affects the following sections in the documentation:

LCA > Normal Static LCA

Due to licensing conflicts, recent versions of SciPy do not include UMFpack. UMFpack is faster than SuperLU, especially for repeated calculations. Python wrappers for UMFpack must be installed separately using scikits.umfpack.

Setup > macOS (Apple Silicon/ARM)

Brightway runs on the new Apple Silicon ARM architecture. However, the super-fast linear algebra software library pypardiso is not compatible with the ARM processor architecture. To avoid critical errors during instruction that would break core functionality, a different version of Brightway (brightway_nosolver) must be installed, which includes a different linear algebra software library (scikit-umfpack).

cmutel commented 6 months ago

The documentation for Accelerate sparse matrices is here.

I also added a warning to let people know if they don't have a fast LA library.

Given that using Accelerate would mean a lot of development work on a platform with has scikit-umfpack already, this is a low priority.

michaelweinold commented 5 months ago

Numpy 2.0.0 now (again) has Apple Accelerate support out-of-the-box.

So I think this may not be the case:

Given that using Accelerate would mean a lot of development work on a platform with has scikit-umfpack already, this is a low priority.

And this might potentially be interesting for:

I'll try to check if I can compare performance on Apple Silicon and post the results here.

Aside from performance, we could further simplify/unify the installation instructions.

cmutel commented 5 months ago

Sure, I saw this too. We are currently pinning stuff to numpy<2 because we don't have enough testing to see if it works, but this looks promising.

michaelweinold commented 5 months ago

Having done some testing, I quickly realized that the actual magic happens in the sp.sparse.linalg.spsolve(technosphere_matrix, demand_array) function, which is called by bw2calc.lca_base.LCABase.solve_linear_system, which is called by bw2calc.lca.LCA.lci_calculation.

Unfortunately, the SciPy documentation (1.13.X) mentions that:

A recent version of OpenBLAS, MKL or ATLAS. The Accelerate BLAS library is no longer supported. \ - Scipy Toolchain Roadmap

An archived SciPy GitHub-wiki entry details the reasoning for this missing support.

There are some efforts:

However, it looks like this might still take a while 😢...

michaelweinold commented 4 months ago

Ok, before @romainsacchi posts his findings here:

For some reason I did not find this previously: SciPy from 1.14.0 DOES now have Apple Accelerate support.

Installing SciPy such that is is actually built against Accelerate is not yet straightforward. Conda failed to do so when I tried... I had better luck with just pip (yeah, yeah, @cmutel 😜).

I found performance did improve for some operations - a quick testing notebook is on this branch:

cmutel commented 4 months ago

a quick testing notebook is on this branch

Is it though?