search

ecmwf-ifs / loki

Freely programmable source-to-source translation for Fortran
https://sites.ecmwf.int/docs/loki/
Apache License 2.0
22 stars 11 forks source link

Add support for `FORALL` #117

Open quepas opened 1 year ago

quepas commented 1 year ago

When a FORALL statement is inside IF-ELSE construct Loki is not able to build the AST. I have prepared three Fortran examples, one not producing an error and two producing two different errors.

Tested versions:

No error

forall_no_error.f90.txt

subroutine forall_no_error_example()
  integer, parameter :: n = 10
  real :: array(n)
  integer :: i
  forall(i = 1:n) array(i) = 1.0
end subroutine

Error 1

forall_error_1.f90.txt

subroutine forall_error_1_example()
  integer, parameter :: n = 10
  real :: array(n)
  integer :: i
  if (n > 5) then
    forall(i = 1:n) array(i) = 1.0
  endif
end subroutine

Produces error:

>>> s1 = Sourcefile.from_file("forall_error_1.f90")
No specific handler for node type <class 'fparser.two.Fortran2003.Forall_Stmt'>
No specific handler for node type <class 'fparser.two.Fortran2003.Forall_Header'>
No specific handler for node type <class 'fparser.two.Fortran2003.Forall_Triplet_Spec_List'>
No specific handler for node type <class 'fparser.two.Fortran2003.Forall_Triplet_Spec'>
[Loki::Sourcefile] Constructed from forall_error_1.f90 in 0.05s
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/sourcefile.py", line 134, in from_file
    return cls.from_fparser(source, filepath, definitions=definitions)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/sourcefile.py", line 256, in from_fparser
    return cls._from_fparser_ast(path=filepath, ast=ast, definitions=definitions,
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/sourcefile.py", line 265, in _from_fparser_ast
    ir = parse_fparser_ast(ast, pp_info=pp_info, definitions=definitions, raw_source=raw_source)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 106, in parse_fparser_ast
    _ir = FParser2IR(raw_source=raw_source, definitions=definitions, pp_info=pp_info, scope=scope).visit(ast)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 301, in visit
    return super().visit(o, **kwargs)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/visitors/visitor.py", line 124, in visit
    return meth(o, *args, **kwargs)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 328, in visit_Specification_Part
    children = as_tuple(flatten(self.visit(c, **kwargs) for c in o.children))
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/tools/util.py", line 150, in flatten
    for el in l:
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 328, in <genexpr>
    children = as_tuple(flatten(self.visit(c, **kwargs) for c in o.children))
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 301, in visit
    return super().visit(o, **kwargs)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/visitors/visitor.py", line 124, in visit
    return meth(o, *args, **kwargs)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 1753, in visit_Subroutine_Subprogram
    body = self.visit(body_ast, **kwargs)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 301, in visit
    return super().visit(o, **kwargs)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/visitors/visitor.py", line 124, in visit
    return meth(o, *args, **kwargs)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 328, in visit_Specification_Part
    children = as_tuple(flatten(self.visit(c, **kwargs) for c in o.children))
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/tools/util.py", line 150, in flatten
    for el in l:
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 328, in <genexpr>
    children = as_tuple(flatten(self.visit(c, **kwargs) for c in o.children))
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 301, in visit
    return super().visit(o, **kwargs)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/visitors/visitor.py", line 124, in visit
    return meth(o, *args, **kwargs)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 2132, in visit_If_Construct
    node = ir.Conditional(condition=conditions[-1], body=body, else_body=as_tuple(else_body),
  File "pydantic/dataclasses.py", line 286, in pydantic.dataclasses._add_pydantic_validation_attributes.handle_extra_init
    def __init__(self, default, default_factory, init, repr, hash, compare,
  File "<string>", line 11, in __init__
  File "pydantic/dataclasses.py", line 305, in pydantic.dataclasses._add_pydantic_validation_attributes.new_post_init
    f'name={self.name!r},'
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/ir.py", line 597, in __post_init__
    assert all(isinstance(c, Node) for c in self.body)  # pylint: disable=not-an-iterable
AssertionError

Error 2

forall_error_2.f90.txt

subroutine forall_error_2_example()
  integer, parameter :: n = 10
  real :: array(n)
  integer :: i
  if (n > 5) then
  else
    forall(i = 1:n) array(i) = 1.0
  endif
end subroutine

Produces error:

>>> s1 = Sourcefile.from_file("forall_error_2.f90")
No specific handler for node type <class 'fparser.two.Fortran2003.Forall_Stmt'>
No specific handler for node type <class 'fparser.two.Fortran2003.Forall_Header'>
No specific handler for node type <class 'fparser.two.Fortran2003.Forall_Triplet_Spec_List'>
No specific handler for node type <class 'fparser.two.Fortran2003.Forall_Triplet_Spec'>
[Loki::Sourcefile] Constructed from forall_error_2.f90 in 0.05s
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/sourcefile.py", line 134, in from_file
    return cls.from_fparser(source, filepath, definitions=definitions)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/sourcefile.py", line 256, in from_fparser
    return cls._from_fparser_ast(path=filepath, ast=ast, definitions=definitions,
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/sourcefile.py", line 265, in _from_fparser_ast
    ir = parse_fparser_ast(ast, pp_info=pp_info, definitions=definitions, raw_source=raw_source)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 106, in parse_fparser_ast
    _ir = FParser2IR(raw_source=raw_source, definitions=definitions, pp_info=pp_info, scope=scope).visit(ast)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 301, in visit
    return super().visit(o, **kwargs)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/visitors/visitor.py", line 124, in visit
    return meth(o, *args, **kwargs)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 328, in visit_Specification_Part
    children = as_tuple(flatten(self.visit(c, **kwargs) for c in o.children))
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/tools/util.py", line 150, in flatten
    for el in l:
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 328, in <genexpr>
    children = as_tuple(flatten(self.visit(c, **kwargs) for c in o.children))
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 301, in visit
    return super().visit(o, **kwargs)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/visitors/visitor.py", line 124, in visit
    return meth(o, *args, **kwargs)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 1753, in visit_Subroutine_Subprogram
    body = self.visit(body_ast, **kwargs)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 301, in visit
    return super().visit(o, **kwargs)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/visitors/visitor.py", line 124, in visit
    return meth(o, *args, **kwargs)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 328, in visit_Specification_Part
    children = as_tuple(flatten(self.visit(c, **kwargs) for c in o.children))
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/tools/util.py", line 150, in flatten
    for el in l:
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 328, in <genexpr>
    children = as_tuple(flatten(self.visit(c, **kwargs) for c in o.children))
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 301, in visit
    return super().visit(o, **kwargs)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/visitors/visitor.py", line 124, in visit
    return meth(o, *args, **kwargs)
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/frontend/fparser.py", line 2132, in visit_If_Construct
    node = ir.Conditional(condition=conditions[-1], body=body, else_body=as_tuple(else_body),
  File "pydantic/dataclasses.py", line 286, in pydantic.dataclasses._add_pydantic_validation_attributes.handle_extra_init
    def __init__(self, default, default_factory, init, repr, hash, compare,
  File "<string>", line 11, in __init__
  File "pydantic/dataclasses.py", line 305, in pydantic.dataclasses._add_pydantic_validation_attributes.new_post_init
    f'name={self.name!r},'
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/ir.py", line 592, in __post_init__
    super().__post_init__()
  File "pydantic/dataclasses.py", line 305, in pydantic.dataclasses._add_pydantic_validation_attributes.new_post_init
    f'name={self.name!r},'
  File "/home/quepas/Research/AWACA/source2source/PaintingCode/venv/lib/python3.10/site-packages/loki/ir.py", line 236, in __post_init__
    super().__post_init__()
  File "pydantic/dataclasses.py", line 308, in pydantic.dataclasses._add_pydantic_validation_attributes.new_post_init
    f'default_factory={self.default_factory!r},'
  File "pydantic/dataclasses.py", line 425, in pydantic.dataclasses._dataclass_validate_values

pydantic.error_wrappers.ValidationError: 3 validation errors for Conditional
else_body -> 0
  instance of Node, tuple or dict expected (type=type_error.dataclass; class_name=Node)
else_body -> 1
  instance of Node, tuple or dict expected (type=type_error.dataclass; class_name=Node)
else_body -> 2
  instance of Node, tuple or dict expected (type=type_error.dataclass; class_name=Node)

btw. great project! :)

reuterbal commented 11 months ago

Hi,

apologies for the delay in acknowledging this and thank you very much for bringing this to our attention.

Loki currently doesn't have any support for forall. Even the "no error" example you provided produces a few warnings for me:

>>> source = Sourcefile.from_file('forall_no_error.f90.txt')
No specific handler for node type <class 'fparser.two.Fortran2003.Forall_Stmt'>
No specific handler for node type <class 'fparser.two.Fortran2003.Forall_Header'>
No specific handler for node type <class 'fparser.two.Fortran2003.Forall_Triplet_Spec_List'>
No specific handler for node type <class 'fparser.two.Fortran2003.Forall_Triplet_Spec'>
[Loki::Sourcefile] Constructed from forall_no_error.f90.txt in 0.05s

Although it succeeds in producing an internal representation, it's not the correct one if you inspect the Fortran that is generated from it:

>>> print(source.to_fortran())
SUBROUTINE forall_no_error_example ()
  INTEGER, PARAMETER :: n = 10
  REAL :: array(n)
  INTEGER :: i
i
1
n
  array(i) = 1.0
END SUBROUTINE forall_no_error_example

Note that we intentionally don't fail on unsupported node types by default to allow for a more graceful handling in bulk processing. However, a "strict" frontend mode can be enabled by setting LOKI_FRONTEND_STRICT_MODE=1:

$ LOKI_FRONTEND_STRICT_MODE=1 python -c "from loki import Sourcefile; Sourcefile.from_file('forall_no_error.f90.txt')"
No specific handler for node type <class 'fparser.two.Fortran2003.Forall_Stmt'>
[Loki::Sourcefile] Constructed from forall_no_error.f90.txt in 0.03s
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/sourcefile.py", line 134, in from_file
    return cls.from_fparser(source, filepath, definitions=definitions)
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/sourcefile.py", line 256, in from_fparser
    return cls._from_fparser_ast(path=filepath, ast=ast, definitions=definitions,
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/sourcefile.py", line 265, in _from_fparser_ast
    ir = parse_fparser_ast(ast, pp_info=pp_info, definitions=definitions, raw_source=raw_source)
  File "/usr/local/apps/python3/3.8.8-01/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/frontend/fparser.py", line 106, in parse_fparser_ast
    _ir = FParser2IR(raw_source=raw_source, definitions=definitions, pp_info=pp_info, scope=scope).visit(ast)
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/frontend/fparser.py", line 301, in visit
    return super().visit(o, **kwargs)
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/visitors/visitor.py", line 124, in visit
    return meth(o, *args, **kwargs)
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/frontend/fparser.py", line 328, in visit_Specification_Part
    children = as_tuple(flatten(self.visit(c, **kwargs) for c in o.children))
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/tools/util.py", line 150, in flatten
    for el in l:
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/frontend/fparser.py", line 328, in <genexpr>
    children = as_tuple(flatten(self.visit(c, **kwargs) for c in o.children))
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/frontend/fparser.py", line 301, in visit
    return super().visit(o, **kwargs)
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/visitors/visitor.py", line 124, in visit
    return meth(o, *args, **kwargs)
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/frontend/fparser.py", line 1753, in visit_Subroutine_Subprogram
    body = self.visit(body_ast, **kwargs)
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/frontend/fparser.py", line 301, in visit
    return super().visit(o, **kwargs)
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/visitors/visitor.py", line 124, in visit
    return meth(o, *args, **kwargs)
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/frontend/fparser.py", line 328, in visit_Specification_Part
    children = as_tuple(flatten(self.visit(c, **kwargs) for c in o.children))
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/tools/util.py", line 150, in flatten
    for el in l:
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/frontend/fparser.py", line 328, in <genexpr>
    children = as_tuple(flatten(self.visit(c, **kwargs) for c in o.children))
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/frontend/fparser.py", line 301, in visit
    return super().visit(o, **kwargs)
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/visitors/visitor.py", line 124, in visit
    return meth(o, *args, **kwargs)
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/frontend/fparser.py", line 2746, in visit_Base
    self.warn_or_fail(f'No specific handler for node type {o.__class__}')
  File "/etc/ecmwf/nfs/dh1_home_a/nabr/loki/main/loki/frontend/fparser.py", line 263, in warn_or_fail
    raise NotImplementedError
NotImplementedError

There is currently no timeline as to when forall is planned to be added, since we integrate newer Fortran features only on a best-effort basis when they are needed. However, we would much welcome the addition of forall to the Loki IR, which likely should be implemented as an extension of the Loop node in ir.py.

quepas commented 11 months ago

@reuterbal Thanks for your response! I am starting to work more and more with Loki, so it is quite possible that I will be able to contribute soon.

quepas commented 6 months ago

I attempted to implement the support of FORALL here https://github.com/ecmwf-ifs/loki/pull/210