Closed certik closed 2 years ago
Here is how one can test it:
#!/bin/bash
set -e
find . -name "*.f" > files.txt
files=$(<files.txt)
set -x
for file in $files; do
lfortran --fixed-form fmt --no-color $file > $file.f90
gfortran -ffree-line-length-none -fallow-argument-mismatch -c $file.f90 -o $file.o
done
One has to manually comment out for now the three top issues with parsing, but otherwise everything parses to AST, the fmt emits f90 file, and GFortran can compile all the f90 files. So this is a big milestone!
I expect few more bugs as we try to link it and actually use it in SciPy, but with a bit of luck there is nothing major.
We can now parse data
correctly. So now we just need to implement the include
and LFortran can parse SciPy, unmodified. Update: done, we can now parse SciPy.
Now we need to hook LFortran or our wrapper script into SciPy's build system. I used SciPy's version 527f2ddcda98ebf4dae37abc38c85082218fcd6a and built it on macOS M1 Max using:
git submodule update --init
mamba env create -f environment.yml
conda activate scipy-dev
pip install .
Then I tested using:
import scipy
scipy.test()
And I got the following output:
Since some tests fail, probably a good way forward is to simply track how many fail / pass:
= 24 failed, 37647 passed, 2207 skipped, 12238 deselected, 135 xfailed, 13 xpassed in 471.13s (0:07:51) =
presumably if LFortran miscompiles something, more tests would fail.
@konradha here is how to get SciPy compiling with LFortran (any subset of files). First create:
$ cat wrapper.sh
#!/bin/bash
set -ex
dir="$HOME/repos/scipy"
A="$@"
Alast="${A##* }"
if [[ "${Alast: -2}" == ".f" ]]; then
echo $Alast >> $dir/log.txt
lfortran --fixed-form fmt --no-color $Alast > ${Alast}90
gfortran.actual -ffree-line-length-none ${A}90
else
gfortran.actual $@
fi
Then install it using:
$ mamba env create -f environment.yml
$ conda activate scipy-dev
$ mv $CONDA_PREFIX/bin/gfortran $CONDA_PREFIX/bin/gfortran.actual
$ cp wrapper.sh $CONDA_PREFIX/bin/gfortran
Then install as usual:
$ pip install -v .
This will create a log.txt
file with all the files that we are running via LFortran. One can easily customize this in any way we need. The above builds the following files:
Via the LFortran -> GFortran combo. Then when I run tests, I get:
So there are some failures, but I think things look pretty good so far.
To just compile minpack, do:
diff --git a/wrapper.sh b/wrapper.sh
index 7c32d024f..6a1ed6881 100755
--- a/wrapper.sh
+++ b/wrapper.sh
@@ -5,7 +5,7 @@ set -ex
dir="$HOME/repos/scipy"
A="$@"
Alast="${A##* }"
-if [[ "${Alast: -2}" == ".f" ]]; then
+if [[ "${Alast: -2}" == ".f" && "$Alast" == *"minpack"* ]]; then
echo $Alast >> $dir/log.txt
lfortran --fixed-form fmt --no-color $Alast > ${Alast}90
gfortran.actual -ffree-line-length-none ${A}90
Then it just compiles the following files:
../../scipy/optimize/minpack/chkder.f
../../scipy/optimize/minpack/dogleg.f
../../scipy/optimize/minpack/dpmpar.f
../../scipy/optimize/minpack/enorm.f
../../scipy/optimize/minpack/fdjac1.f
../../scipy/optimize/minpack/fdjac2.f
../../scipy/optimize/minpack/hybrd.f
../../scipy/optimize/minpack/hybrd1.f
../../scipy/optimize/minpack/hybrj.f
../../scipy/optimize/minpack/hybrj1.f
../../scipy/optimize/minpack/lmder.f
../../scipy/optimize/minpack/lmder1.f
../../scipy/optimize/minpack/lmdif.f
../../scipy/optimize/minpack/lmdif1.f
../../scipy/optimize/minpack/lmpar.f
../../scipy/optimize/minpack/lmstr.f
../../scipy/optimize/minpack/lmstr1.f
../../scipy/optimize/minpack/qform.f
../../scipy/optimize/minpack/qrfac.f
../../scipy/optimize/minpack/qrsolv.f
../../scipy/optimize/minpack/r1mpyq.f
../../scipy/optimize/minpack/r1updt.f
../../scipy/optimize/minpack/rwupdt.f
../../scipy/optimize/minpack2/dcsrch.f
../../scipy/optimize/minpack2/dcstep.f
And the test results are:
[...]
../optimize/tests/test_minpack.py ...................................... [ 42%]
........................... [ 42%]
[...]
= 24 failed, 37647 passed, 2207 skipped, 12238 deselected, 135 xfailed, 13 xpassed in 302.55s (0:05:02) =
Which is the same amount of failures as I was getting above, so I think LFortran is parsing all of minpack correctly to AST.
My output (Ubuntu 22.04) original
= 37692 passed, 2135 skipped, 12238 deselected, 160 xfailed, 13 xpassed, 26 errors in 473.30s (0:07:53) =
Haven't fully figured out the other steps to compile the generated .f90 files, but will get there later today :)
Thanks for the detailed plan.
I think this issue is fixed. See this answer for more details: https://fortran-lang.discourse.group/t/lfortran-can-now-parse-all-of-scipy-to-ast/4479/8.
The next step is to compile some well defined subset of SciPy fully, link with SciPy and all tests to pass. We are focusing on Minpack: #512.
The following
lfortran --show-ast --fixed-form $file > $file.ast
now works for every*.f
file in SciPy, except the following three issues:After resolving those three issues, here is a TODO list:
lfortran --fixed-form fmt --no-color $file > $file.f90
.f90
files generated in the previous step instead of the original.f
files..f
files) and ensure everything builds.f
files via LFortran -> GFortranThis would be a very solid test that we have parsed everything correctly to AST (and can format everything correctly); everything that is tested by SciPy tests at least.