Add support for reading / writing compressed files

[mithro]

See https://github.com/mithro/duck2-gsoc/issues/16

[duck2]

I did an experiment with zstr streams:

void get_root_elements(const char *filename){
    pugi::xml_document doc;
    pugi::xml_parse_result result;

    std::string x(filename);
    if(x.rfind(".") != std::string::npos && x.substr(x.rfind(".")+1) == "gz"){
        std::ifstream F;
        F.open(x);
        zstr::istream Z(F);
        result = doc.load(Z);
    } else {
        result = doc.load_file(filename);
    }

    if(!result)
        throw std::runtime_error("Could not load XML file " + std::string(filename) + ".");
    for(pugi::xml_node node= doc.first_child(); node; node = node.next_sibling()){
        if(std::strcmp(node.name(), "rr_graph") == 0){
            count_rr_graph(node);
            alloc_arenas();
            load_rr_graph(node, &rr_graph);
        }
        else throw std::runtime_error("Invalid root-level element " + std::string(node.name()));
    }
}

Artix 7 rr_graph run with uncompressed file(922 MB)(without errno checking after strtol calls): 7.645 8.097 7.600 7.636 7.677

With gzip-compressed file: 11.34 11.15 11.10 11.29 11.13

[mithro]

Is that with or without a hot disk cache? Can you try flushing that?

[duck2]

It's with a hot disk cache. Without the file in the cache, the reading time can jump to 11 seconds or so.

[mithro]

@duck2 - How does the time between with gzip and without gzip compare without file in the disk cache?

[litghost]

Once SAX parsing support is complete (#3), a compressed one pass SAX parser may be a good compromise between CPU/disk/memory usage. Unclear if a two pass SAX + compression would have good numbers.