OOM and RecursionError with large code base

[dev-zero]

I am trying to run FORD on https://github.com/cp2k/cp2k.

When setting graph: true I get (even when restricting to subdir, like src/dbcsr):

Generating graphs...
Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.5/ford", line 9, in <module>
    load_entry_point('FORD==5.0.6', 'console_scripts', 'ford')()
  File "/usr/lib64/python3.5/site-packages/ford/__init__.py", line 357, in run
    main(proj_data,proj_docs,md)
  File "/usr/lib64/python3.5/site-packages/ford/__init__.py", line 344, in main
    docs = ford.output.Documentation(proj_data,proj_docs_,project,page_tree)
  File "/usr/lib64/python3.5/site-packages/ford/output.py", line 77, in __init__
    self.graphs.register(item)
  File "/usr/lib64/python3.5/site-packages/ford/graphmanager.py", line 66, in register
    ford.graphs.FortranGraph.data.register(obj,type(obj))
  File "/usr/lib64/python3.5/site-packages/ford/graphs.py", line 121, in register
    if obj not in self.types: self.types[obj] = TypeNode(obj,self)
  File "/usr/lib64/python3.5/site-packages/ford/graphs.py", line 237, in __init__
    n = gd.get_node(var.proto[0],FortranType)
  File "/usr/lib64/python3.5/site-packages/ford/graphs.py", line 154, in get_node
    self.register(obj,cls,hist)
  File "/usr/lib64/python3.5/site-packages/ford/graphs.py", line 121, in register
    if obj not in self.types: self.types[obj] = TypeNode(obj,self)
  File "/usr/lib64/python3.5/site-packages/ford/graphs.py", line 237, in __init__
    n = gd.get_node(var.proto[0],FortranType)
  File "/usr/lib64/python3.5/site-packages/ford/graphs.py", line 154, in get_node
    self.register(obj,cls,hist)
  File "/usr/lib64/python3.5/site-packages/ford/graphs.py", line 121, in register
    if obj not in self.types: self.types[obj] = TypeNode(obj,self)
[...]
  File "/usr/lib64/python3.5/site-packages/ford/graphs.py", line 237, in __init__
    n = gd.get_node(var.proto[0],FortranType)
  File "/usr/lib64/python3.5/site-packages/ford/graphs.py", line 154, in get_node
    self.register(obj,cls,hist)
  File "/usr/lib64/python3.5/site-packages/ford/graphs.py", line 121, in register
    if obj not in self.types: self.types[obj] = TypeNode(obj,self)
  File "/usr/lib64/python3.5/site-packages/ford/graphs.py", line 222, in __init__
    super(TypeNode,self).__init__(obj)
  File "/usr/lib64/python3.5/site-packages/ford/graphs.py", line 177, in __init__
    self.ident = d + '~' + obj.ident
  File "/usr/lib64/python3.5/site-packages/ford/sourceform.py", line 238, in ident
    if ( type(self) in [FortranSubroutine,FortranFunction] and
RecursionError: maximum recursion depth exceeded while calling a Python object

When disabling graph generation, I get an Out-of-memory first when generating the search index (search: true). When disabling the search as well, the process gets killed in Generating HTML documentation (even on a machine with 16GB RAM).

[dev-zero]

The settings are as follows:

project: CP2K
src_dir: ./src
output_dir: ./doc
project_github: https://github.com/cp2k/cp2k
summary: CP2K -- a quantum chemistry and solid state physics software package
author: CP2K Developers Group
github: https://github.com/cp2k
website: https://cp2k.org
twitter: https://twitter.com/cp2kproject
display: public
source: true
graph: true
search: false
extensions: f90, f95, f03, f08, f15, F90, F95, F03, F08, F15, f, F
fixed_extensions: for, FOR

... the project uses .F by default but using a free-form standard (.f90 are partial fortran files to be included in others).

[zerothi]

@dev-zero please see #191 for a branch which may solve some of your problems.

[jmccreight]

+1 for this issue. FORD is over 14GB right now and still chugging. Since it's well under my memory on the machine, no big deal. But I want to run FORD for this project on laptops under docker, so I'd like to put it on a memory diet. I may eval #191 when I have the need. Thanks again for all of this work by everyone.

[cmacmackin]

The master branch on here might have fixed these issues. Could you please try running with it?

[cmacmackin]

I ran current version of FORD on the version of cp2k that existed when this issue was created, using the specified project file options. I was able to collect the graph data withing the recursion error. Generating graphs was still incredibly time-consuming for such a large project, even with the graph size limited. Memory requirements remained large (I suspect some of the slowness was due to using swap-space on the disc) but shouldn't be large enough to crash the program. When I ran with graphs turned off the memory usage did not go above 2GB. It remained quite slow due to the size of your project (over 1000 source files!). At some point it may be worth looking to implement some of the slower parts of FORD using Cython or a similar approach, although that is beyond the scope of this bug report.