Limit for document_build_from_file

[MarDiehl]

It seems that some resources are exhausted when calling fy_document_build_from_file.

The following code crashes for me after 1021 iterations

#ifdef HAVE_CONFIG_H
#include "config.h"
#endif

#include <stdlib.h>
#include <stdio.h>

#include <libfyaml.h>

int main(int argc, char *argv[])
{
    struct fy_document *fyd = NULL;

    int i=1;
    while(i<10000)
    {
        fyd = fy_document_build_from_file(NULL, "test.yaml");
        if (!fyd) {
            fprintf(stderr, "failed to build document");
            return 1;
        }
        fprintf(stdout,"%d\n",i);
        i++;
    }
    return 0;
}

The content of the file (test.yaml) seems to be irrelevant, I use the following small example.

---
test: 1

[pantoniou]

Of course you run out of resources since the returned document is not destroyed by calling fy_document_destroy()

I will leave the issue up until the documentation entry of fy_document_destroy() is updated to include all the cases where one should call it in order to dispose the document object.

[MarDiehl]

Many thanks. It was indeed not clear to me that there is a limit on open documents beyond any memory limitations of my system.

However, when I create a document with fy_document_build_from_string (see below) it does not have the limit. Could it be that a file handle is not closed?

#ifdef HAVE_CONFIG_H                                                                                
#include "config.h"                                                                                 
#endif                                                                                              

#include <stdlib.h>                                                                                 
#include <stdio.h>                                                                                  

#include <libfyaml.h>                                                                               

int main(int argc, char *argv[])                                                                    
{                                                                                                   
    struct fy_document *fyd = NULL;                                                                 

    static const char *yaml =                                                                       
        "invoice: 34843\n"                                                                          
        "date   : !!str 2001-01-23\n"                                                               
        "bill-to: &id001\n"                                                                         
        "    given  : Chris\n"                                                                      
        "    family : Dumars\n"                                                                     
        "    address:\n"                                                                            
        "        lines: |\n"                                                                        
        "            458 Walkman Dr.\n"                                                             
        "            Suite #292\n";                                                                 

    int i=1;                                                                                        
    while(i<10000)                                                                                  
    {                                                                                               
        fyd = fy_document_build_from_string(NULL, yaml, (size_t)-1);                                
        if (!fyd) {                                                                                 
            fprintf(stderr, "failed to build document");                                            
            return 1;                                                                               
        }                                                                                           
        fprintf(stdout,"%d\n",i);                                                                   
        i++;                                                                                        
    }                                                                                               
    return 0;                                                                                       
}

[pantoniou]

Indeed it is a case of open file handles. That's why the limit is 1021, which is 1021 file handles + stdin/stdout/stderr.

By default libfyaml is using mmap to map the file contents to the process's virtual address space so that it avoids copies into allocated memory from the heap.

You can suppress this optimization by using the FYPCF_DISABLE_MMAP_OPT flag (but you have to pass it via the parse configuration structure instead of NULL in the call to fy_document_build_from_string())

Obviously using the fy_document_build_from_string method involves no files, so the limit is only by the amount of memory lost by the dangling document object. For contemporary machines with gigabytes of memory this limit is very large indeed.

[MarDiehl]

thanks!

Honestly, these things are far beyond my noob C-knowledge. I've just open that many files because I want to use libfyaml as a backend for a YAML parser in Fortran and needed to make sure that I don't create any memory leaks when passing strings from C to Fortran.

Anyways, many thanks for the nice library.

[pantoniou]

No updates, I guess this is done with.