Create a PyIodide package for Andes

[rwl]

Andes has a complex set of dependencies that potential users might not be willing to install locally. PyIoidide is an exciting new project that could allow Andes to be run within the browser using WebAssembly. Many of the packages that Andes requires have already been ported to PyIodide. If CVXOPT and SymPy were ported it might be possible to create an Andes package:

https://pyodide.readthedocs.io/en/latest/new_packages.html

This would allow users to create and share notebooks online. Andes running in WebAssembly would open the possibility to create a custom GUI such as that made with MATPOWER:

https://matpower.app/

This could, potentially, also be built into an Electron-based desktop application and made available to download by users wishing to work offline or on air-gaped machines.

[rwl]

I see now that it is already possible to try Andes online here:

https://mybinder.org/v2/gh/curent/andes/master

[cuihantao]

Got it. Right now, Andes runs online in Binder.

I was impressed by matpower.app's performance in WebAssembly. Will give it a shot at some later time.

[rwl]

Demo: https://andesapp.github.io/

[cuihantao]

Demo: https://andesapp.github.io/

This is amazing!!

With the latest version of ANDES, you create a customized distribution to ANDES source code to skip the code generation. The steps are as follows.

1. Obtain the ANDES source code.
2. Install the same version of ANDES (using either develop or install).
3. In the source code root folder, run andes prepare --pycode-path andes/pycode

A folder named pycode will be generated in the subfolder andes at the same level of the core and models folder. The new source code can be used for the webapp. It should skip the code generation every time.

The inconvenience is that the above steps need to be executed every time after updating ANDES models.

Let me know if any thoughts.

[cuihantao]

Hi @rwl , I wonder if you could share the build script meta.yaml for KVXOPT? I would like to update your demo andesapp to work with the latest ANDES. I haven't been able to link lapack and blas when building KVXOPT using Pyodide. Thanks!

[rwl]

For some reason, I didn't use the CLAPACK package that comes with Pyodide. I have a version that applies these patches:

diff -Naur a/BLAS/SRC/xerbla.c b/BLAS/SRC/xerbla.c
--- a/BLAS/SRC/xerbla.c 2010-04-27 21:23:16.000000000 +0200
+++ b/BLAS/SRC/xerbla.c 2021-03-25 19:58:44.754892352 +0100
@@ -10,6 +10,8 @@
        http://www.netlib.org/f2c/libf2c.zip
 */

+#include <stdio.h>
+
 #include "f2c.h"
 #include "blaswrap.h"

diff -Naur a/F2CLIBS/libf2c/ef1asc_.c b/F2CLIBS/libf2c/ef1asc_.c
--- a/F2CLIBS/libf2c/ef1asc_.c  2009-08-08 00:32:18.000000000 +0200
+++ b/F2CLIBS/libf2c/ef1asc_.c  2021-03-25 20:05:18.970632776 +0100
@@ -13,7 +13,7 @@
 extern VOID s_copy();
 ef1asc_(a, la, b, lb) ftnint *a, *b; ftnlen *la, *lb;
 #else
-extern void s_copy(char*,char*,ftnlen,ftnlen);
+extern int s_copy(char*,char*,ftnlen,ftnlen);
 int ef1asc_(ftnint *a, ftnlen *la, ftnint *b, ftnlen *lb)
 #endif
 {
diff -Naur a/F2CLIBS/libf2c/f2ch.add b/F2CLIBS/libf2c/f2ch.add
--- a/F2CLIBS/libf2c/f2ch.add   2009-08-08 00:32:18.000000000 +0200
+++ b/F2CLIBS/libf2c/f2ch.add   2021-03-25 20:05:18.970632776 +0100
@@ -124,9 +124,9 @@
 extern double r_sqrt(float *);
 extern double r_tan(float *);
 extern double r_tanh(float *);
-extern void s_cat(char *, char **, integer *, integer *, ftnlen);
+extern int s_cat(char *, char **, integer *, integer *, ftnlen);
 extern integer s_cmp(char *, char *, ftnlen, ftnlen);
-extern void s_copy(char *, char *, ftnlen, ftnlen);
+extern int s_copy(char *, char *, ftnlen, ftnlen);
 extern int s_paus(char *, ftnlen);
 extern integer s_rdfe(cilist *);
 extern integer s_rdue(cilist *);
diff -Naur a/F2CLIBS/libf2c/Makefile b/F2CLIBS/libf2c/Makefile
--- a/F2CLIBS/libf2c/Makefile   2010-04-27 21:09:35.000000000 +0200
+++ b/F2CLIBS/libf2c/Makefile   2021-04-08 15:33:03.414480609 +0200
@@ -20,14 +20,12 @@
 # compile, then strip unnecessary symbols
 .c.o:
    $(CC) -c -DSkip_f2c_Undefs $(CFLAGS) $*.c
-   ld -r -x -o $*.xxx $*.o
-   mv $*.xxx $*.o
 ## Under Solaris (and other systems that do not understand ld -x),
 ## omit -x in the ld line above.
 ## If your system does not have the ld command, comment out
 ## or remove both the ld and mv lines above.

-MISC = f77vers.o i77vers.o main.o s_rnge.o abort_.o exit_.o getarg_.o iargc_.o\
+MISC = f77vers.o i77vers.o s_rnge.o abort_.o exit_.o getarg_.o iargc_.o\
    getenv_.o signal_.o s_stop.o s_paus.o system_.o cabs.o ctype.o\
    derf_.o derfc_.o erf_.o erfc_.o sig_die.o uninit.o
 POW =  pow_ci.o pow_dd.o pow_di.o pow_hh.o pow_ii.o pow_ri.o pow_zi.o pow_zz.o
@@ -72,8 +70,8 @@
 all: f2c.h signal1.h sysdep1.h libf2c.a clapack_install

 libf2c.a: $(OFILES)
-   ar r libf2c.a $?
-   -ranlib libf2c.a
+   $(ARCH) r libf2c.a $?
+   $(RANLIB) libf2c.a

 ## Shared-library variant: the following rule works on Linux
 ## systems.  Details are system-dependent.  Under Linux, -fPIC
@@ -119,7 +117,7 @@

 install: libf2c.a
    cp libf2c.a $(LIBDIR)
-   -ranlib $(LIBDIR)/libf2c.a
+   $(RANLIB) $(LIBDIR)/libf2c.a

 clapack_install: libf2c.a
    mv libf2c.a ..
@@ -184,8 +182,8 @@
 arith.h: arithchk.c
    $(CC) $(CFLAGS) -DNO_FPINIT arithchk.c -lm ||\
     $(CC) -DNO_LONG_LONG $(CFLAGS) -DNO_FPINIT arithchk.c -lm
-   ./a.out >arith.h
-   rm -f a.out arithchk.o
+   node a.out.js >arith.h
+   rm -f a.out.js a.out.wasm

 check:
    xsum Notice README abort_.c arithchk.c backspac.c c_abs.c c_cos.c \
diff -Naur a/INSTALL/tstiee.c b/INSTALL/tstiee.c
--- a/INSTALL/tstiee.c  2009-08-08 00:32:18.000000000 +0200
+++ b/INSTALL/tstiee.c  2021-03-25 19:58:44.754892352 +0100
@@ -10,6 +10,8 @@
        http://www.netlib.org/f2c/libf2c.zip
 */

+#include <string.h>
+
 #include "f2c.h"
 #include "blaswrap.h"

diff -Naur a/Makefile b/Makefile
--- a/Makefile  2021-04-13 19:33:04.300487374 +0200
+++ b/Makefile  2021-04-13 19:33:04.320487362 +0200
@@ -15,9 +15,8 @@

 clean: cleanlib cleantesting cleanblas_testing 

-lapack_install:
-   ( cd INSTALL; $(MAKE); ./testlsame; ./testslamch; \
-     ./testdlamch; ./testsecond; ./testdsecnd; ./testversion )
+lapack_install: f2clib
+   ( cd INSTALL; $(MAKE) )

 blaslib:
    ( cd BLAS/SRC; $(MAKE) )
diff -Naur a/SRC/Makefile b/SRC/Makefile
--- a/SRC/Makefile  2010-04-27 21:09:35.000000000 +0200
+++ b/SRC/Makefile  2021-04-08 15:43:54.616133123 +0200
@@ -50,9 +50,9 @@
 #
 #######################################################################

-ALLAUX = maxloc.o ilaenv.o ieeeck.o lsamen.o xerbla.o xerbla_array.o iparmq.o  \
+ALLAUX = maxloc.o ilaenv.o ieeeck.o lsamen.o iparmq.o  \
     ilaprec.o ilatrans.o ilauplo.o iladiag.o chla_transtype.o \
-    ../INSTALL/ilaver.o ../INSTALL/lsame.o
+    ../INSTALL/ilaver.o

 ALLXAUX =

@@ -104,7 +104,7 @@
    sggrqf.o sggsvd.o sggsvp.o sgtcon.o sgtrfs.o sgtsv.o  \
    sgtsvx.o sgttrf.o sgttrs.o sgtts2.o shgeqz.o \
    shsein.o shseqr.o slabrd.o slacon.o slacn2.o \
-   slaein.o slaexc.o slag2.o  slags2.o slagtm.o slagv2.o slahqr.o \
+   slaein.o slaexc.o slag2.o  slags2.o slagtm.o slagv2.o \
    slahrd.o slahr2.o slaic1.o slaln2.o slals0.o slalsa.o slalsd.o \
    slangb.o slange.o slangt.o slanhs.o slansb.o slansp.o \
    slansy.o slantb.o slantp.o slantr.o slanv2.o \
@@ -176,7 +176,7 @@
    clacgv.o clacon.o clacn2.o clacp2.o clacpy.o clacrm.o clacrt.o cladiv.o \
    claed0.o claed7.o claed8.o \
    claein.o claesy.o claev2.o clags2.o clagtm.o \
-   clahef.o clahqr.o \
+   clahef.o \
    clahrd.o clahr2.o claic1.o clals0.o clalsa.o clalsd.o clangb.o clange.o clangt.o \
    clanhb.o clanhe.o \
    clanhp.o clanhs.o clanht.o clansb.o clansp.o clansy.o clantb.o \
@@ -236,7 +236,7 @@
    dggrqf.o dggsvd.o dggsvp.o dgtcon.o dgtrfs.o dgtsv.o  \
    dgtsvx.o dgttrf.o dgttrs.o dgtts2.o dhgeqz.o \
    dhsein.o dhseqr.o dlabrd.o dlacon.o dlacn2.o \
-   dlaein.o dlaexc.o dlag2.o  dlags2.o dlagtm.o dlagv2.o dlahqr.o \
+   dlaein.o dlaexc.o dlag2.o  dlags2.o dlagtm.o dlagv2.o \
    dlahrd.o dlahr2.o dlaic1.o dlaln2.o dlals0.o dlalsa.o dlalsd.o \
    dlangb.o dlange.o dlangt.o dlanhs.o dlansb.o dlansp.o \
    dlansy.o dlantb.o dlantp.o dlantr.o dlanv2.o \
@@ -310,7 +310,7 @@
    zlacgv.o zlacon.o zlacn2.o zlacp2.o zlacpy.o zlacrm.o zlacrt.o zladiv.o \
    zlaed0.o zlaed7.o zlaed8.o \
    zlaein.o zlaesy.o zlaev2.o zlags2.o zlagtm.o \
-   zlahef.o zlahqr.o \
+   zlahef.o \
    zlahrd.o zlahr2.o zlaic1.o zlals0.o zlalsa.o zlalsd.o zlangb.o zlange.o \
    zlangt.o zlanhb.o \
    zlanhe.o \

It uses this make.inc file:

# -*- Makefile -*-
####################################################################
#  LAPACK make include file.                                       #
#  LAPACK, Version 3.2.1                                           #
#  June 2009                                                       #
####################################################################
#
# See the INSTALL/ directory for more examples.
#
SHELL = /bin/sh
#
#  The machine (platform) identifier to append to the library names
#
# WA for WebAssembly
PLAT =
#
#  Modify the FORTRAN and OPTS definitions to refer to the
#  compiler and desired compiler options for your machine.  NOOPT
#  refers to the compiler options desired when NO OPTIMIZATION is
#  selected.  Define LOADER and LOADOPTS to refer to the loader
#  and desired load options for your machine.
#
#######################################################
# This is used to compile C libary
#CC        = gcc  # inherit $CC from emmake
# if no wrapping of the blas library is needed, uncomment next line
#CC        = gcc -DNO_BLAS_WRAP
CFLAGS    = -O3 -I$(TOPDIR)/INCLUDE -fPIC -DNO_BLAS_WRAP
LDFLAGS   = -O3
LOADER    = $(CC)
LOADOPTS  =
NOOPT     = -O0 -I$(TOPDIR)/INCLUDE -fPIC
DRVCFLAGS = $(CFLAGS)
F2CCFLAGS = $(CFLAGS)
#######################################################################

#
# Timer for the SECOND and DSECND routines
#
# Default : SECOND and DSECND will use a call to the EXTERNAL FUNCTION ETIME
# TIMER    = EXT_ETIME
# For RS6K : SECOND and DSECND will use a call to the EXTERNAL FUNCTION ETIME_
# TIMER    = EXT_ETIME_
# For gfortran compiler: SECOND and DSECND will use a call to the INTERNAL FUNCTION ETIME
# TIMER    = INT_ETIME
# If your Fortran compiler does not provide etime (like Nag Fortran Compiler, etc...)
# SECOND and DSECND will use a call to the Fortran standard INTERNAL FUNCTION CPU_TIME 
TIMER    = INT_CPU_TIME
# If neither of this works...you can use the NONE value... In that case, SECOND and DSECND will always return 0
# TIMER     = NONE
#
#  The archiver and the flag(s) to use when building archive (library)
#  If you system has no ranlib, set RANLIB = echo.
#
ARCH     = $(AR)
ARCHFLAGS= cr
#RANLIB   = ranlib
#
#  The location of BLAS library for linking the testing programs.
#  The target's machine-specific, optimized BLAS library should be
#  used whenever possible.
#
BLASLIB      = ../../blas$(PLAT).a
#
#  Location of the extended-precision BLAS (XBLAS) Fortran library
#  used for building and testing extended-precision routines.  The
#  relevant routines will be compiled and XBLAS will be linked only if
#  USEXBLAS is defined.
#
# USEXBLAS    = Yes
XBLASLIB     =
# XBLASLIB    = -lxblas
#
#  Names of generated libraries.
#
LAPACKLIB    = lapack$(PLAT).a
F2CLIB       = ../../F2CLIBS/libf2c.a
TMGLIB       = tmglib$(PLAT).a
EIGSRCLIB    = eigsrc$(PLAT).a
LINSRCLIB    = linsrc$(PLAT).a

Finally, the Makefile applies these commands:

    # Modify subroutines to return void instead of int
    # as expected by KVXOPT and SuiteSparse.
    sed -i 's|^/\* Subroutine \*/ int |/\* Subroutine \*/ void |' $(SRC)/BLAS/SRC/*.c
    sed -i 's|^/\* Subroutine \*/ int |/\* Subroutine \*/ void |' $(SRC)/SRC/*.c
    sed -i 's|^    extern /\* Subroutine \*/ int |    extern /\* Subroutine \*/ void |' $(SRC)/BLAS/SRC/*.c
    sed -i 's|^    extern /\* Subroutine \*/ int |    extern /\* Subroutine \*/ void |' $(SRC)/SRC/*.c
    sed -i 's|return 0;|return;|' $(SRC)/BLAS/SRC/*.c
    sed -i 's|return 0;|return;|' $(SRC)/SRC/*.c

The KVXOPT package required some changes to the setup.py file:

diff -Naur a/setup.py b/setup.py
--- a/setup.py  2020-11-01 20:06:27.000000000 +0100
+++ b/setup.py  2021-04-13 15:15:42.804671229 +0200
@@ -7,10 +7,10 @@
 from distutils.file_util import copy_file

 # Modifiy this if BLAS and LAPACK libraries are not in /usr/lib.
-BLAS_LIB_DIR = '/usr/lib'
+BLAS_LIB_DIR = '../../../F2CLAPACK/F2CLAPACK_WA'

 # Default names of BLAS and LAPACK libraries
-BLAS_LIB = ['blas']
+BLAS_LIB = ['blas', 'f2c']
 LAPACK_LIB = ['lapack']
 BLAS_EXTRA_LINK_ARGS = []

@@ -82,7 +82,7 @@
     FFTW_MACROS = []

 # Directory containing SuiteSparse source
-SUITESPARSE_SRC_DIR = ''
+SUITESPARSE_SRC_DIR = '../../../SuiteSparse/SuiteSparse-WA'

 # For SuiteSparse Versions before to 4.0.0 SuiteSparse_config does not exist
 # We can avoid the search and link with this flag

The meta.yaml file is just:

package:
  name: kvxopt
  version: 1.2.6.0
source:
  sha256: b67430a7434ce2eee94da217059d6f2903ed1930a29370cc479b559699cd0ff0
  url: https://files.pythonhosted.org/packages/52/61/e33d976e1954d18f8543a944ac6dc85899523739814aa01628f3a437ab06/kvxopt-1.2.6.0.tar.gz
  patches:
    - kvxopt.patch
requirements:
  run:
    - F2CLAPACK
    - SuiteSparse
test:
  imports:
    - kvxopt
    - kvxopt.amd
    - kvxopt.base
    - kvxopt.blas
    - kvxopt.dense
    - kvxopt.klu
    - kvxopt.lapack
    - kvxopt.sparse
    - kvxopt.umfpack

I found that dill required a small change, but this may not needed anymore:

diff -Naur a/dill/_dill.py b/dill/_dill.py
--- a/dill/_dill.py 2020-08-25 10:22:36.000000000 +0200
+++ b/dill/_dill.py 2021-04-13 14:24:02.101956995 +0200
@@ -207,13 +207,13 @@

 FileType = get_file_type('rb', buffering=0)
 TextWrapperType = get_file_type('r', buffering=-1)
-BufferedRandomType = get_file_type('r+b', buffering=-1)
+BufferedRandomType = get_file_type('r+b', buffering=0)
 BufferedReaderType = get_file_type('rb', buffering=-1)
 BufferedWriterType = get_file_type('wb', buffering=-1)
 try:
     from _pyio import open as _open
     PyTextWrapperType = get_file_type('r', buffering=-1, open=_open)
-    PyBufferedRandomType = get_file_type('r+b', buffering=-1, open=_open)
+    PyBufferedRandomType = get_file_type('r+b', buffering=0, open=_open)
     PyBufferedReaderType = get_file_type('rb', buffering=-1, open=_open)
     PyBufferedWriterType = get_file_type('wb', buffering=-1, open=_open)
 except ImportError:

The changes to Andes mostly just involved commenting out some optional imports:

diff -Naur a/andes/core/model.py b/andes/core/model.py
--- a/andes/core/model.py   2021-03-21 16:00:35.000000000 +0100
+++ b/andes/core/model.py   2021-04-16 12:59:44.151276802 +0200
@@ -12,7 +12,6 @@
 #  Last modified: 8/16/20, 7:27 PM

 import logging
-import scipy as sp

 from collections import OrderedDict
 from typing import Iterable, Sized, Callable, Union
diff -Naur a/andes/core/solver.py b/andes/core/solver.py
--- a/andes/core/solver.py  2021-03-21 16:00:35.000000000 +0100
+++ b/andes/core/solver.py  2021-04-15 10:19:44.034012059 +0200
@@ -4,8 +4,8 @@

 import logging

-from scipy.sparse import csc_matrix
-from scipy.sparse.linalg import spsolve
+# from scipy.sparse import csc_matrix
+# from scipy.sparse.linalg import spsolve
 from andes.shared import np, matrix, umfpack, klu, cupy

 logger = logging.getLogger(__name__)
diff -Naur a/andes/main.py b/andes/main.py
--- a/andes/main.py 2021-03-21 16:00:35.000000000 +0100
+++ b/andes/main.py 2021-04-15 10:19:44.034012059 +0200
@@ -29,7 +29,7 @@
 from andes.routines import routine_cli
 from andes.utils.misc import elapsed, is_interactive
 from andes.utils.paths import get_config_path, tests_root, get_log_dir
-from andes.shared import coloredlogs, unittest
+# from andes.shared import coloredlogs, unittest
 from andes.shared import Pool, Process

 logger = logging.getLogger(__name__)
diff -Naur a/andes/routines/eig.py b/andes/routines/eig.py
--- a/andes/routines/eig.py 2021-03-21 16:00:35.000000000 +0100
+++ b/andes/routines/eig.py 2021-04-15 10:19:44.034012059 +0200
@@ -3,7 +3,7 @@
 """

 import logging
-import scipy.io
+# import scipy.io
 import numpy as np

 from math import ceil, pi
diff -Naur a/andes/shared.py b/andes/shared.py
--- a/andes/shared.py   2021-03-21 16:00:35.000000000 +0100
+++ b/andes/shared.py   2021-04-15 10:19:44.034012059 +0200
@@ -12,11 +12,11 @@

 import math
 import os
-import coloredlogs         # NOQA
+# import coloredlogs         # NOQA
 import numpy as np         # NOQA

 from andes.utils.lazyimport import LazyImport
-from distutils.spawn import find_executable
+# from distutils.spawn import find_executable

 # Library preference:
 # KVXOPT + ipadd > CVXOPT + ipadd > KXVOPT > CVXOPT (+ KLU or not)
diff -Naur a/andes/utils/paths.py b/andes/utils/paths.py
--- a/andes/utils/paths.py  2021-03-21 16:00:35.000000000 +0100
+++ b/andes/utils/paths.py  2021-04-15 11:10:31.134064356 +0200
@@ -2,7 +2,7 @@
 Utility functions for loading andes stock test cases
 """
 import os
-import platform
+# import platform
 import tempfile
 import pathlib
 import logging
@@ -219,13 +219,7 @@
     str
         The path to the temporary logging directory
     """
-    path = ''
-    if platform.system() in ('Linux', 'Darwin'):
-        path = tempfile.mkdtemp(prefix='andes-')
-
-    elif platform.system() == 'Windows':
-        appdata = os.getenv('APPDATA')
-        path = os.path.join(appdata, 'andes')
+    path = tempfile.mkdtemp(prefix='andes-')

     if not os.path.exists(path):
         os.makedirs(path)
diff -Naur a/setup.py b/setup.py
--- a/setup.py  2021-03-21 16:00:35.000000000 +0100
+++ b/setup.py  2021-04-15 10:19:44.034012059 +0200
@@ -56,7 +56,7 @@
             # 'path/to/data_file',
         ]
     },
-    install_requires=requirements,
+    # install_requires=requirements,
     license="GNU Public License v3",
     classifiers=[
         'Development Status :: 4 - Beta',

It would be interesting to see how FORTRAN to WASM compilation is coming along. Getting ARPACK working and performing SSSA in the browser would be a first!

[cuihantao]

Thank you very much for sharing the code. It still looks challenging, but let me try to build it first.

[rwl]

The code I wrote was mixed into another repository. I separated it out and pushed it here:

https://github.com/rwl/andesapp

I haven't tried it, but there it a Dockerfile that should build.

[sanurielf]

Hey @rwl and @cuihantao ,

I saw this interesting, and I'm always trying to improve KVXOPT. I will follow this if you need modifications in KVXOPT for better integration with this Pyodide interpreter.

[cuihantao]

Hello @sanurielf and @rwl ,

Sorry that I didn't follow up until now because I didn't even make the docker image to build. A lot internals have changed in Pyodide, and the documentations was being improved. I was later dragged away by some other works... I perhaps work on it again when the semester ends.