First draft for a buffered writing of files

[lesquoyb]

It's a work in progress, I'm currently testing it and not sure yet it's actually more performant. I tested it with this model (thanks @ptaillandier) and for now it seems to be way better, but it needs more work to be sure:

/**
* Name: Writingfile
* Based on the internal skeleton template. 
* Author: patricktaillandier
* Tags: 
*/

model Writingfile

global {
    int nb_writers <- 1000;
    int nb_cycle <- 100;
    string output_file <- "output.csv";

    init {
        create writer number: nb_writers;
        save "cycle,name,rnd(1.0),rnd(1.0)" to: output_file format: "text";

    }
}

species writer {
    reflex save_result {
        save string(cycle) + ","+name +"," + rnd(1.0) + "," + rnd(1.0) to: output_file format: "text" rewrite: false;

        save [cycle, name,rnd(1.0),rnd(1.0)] to: output_file format: "csv"  rewrite: false;
    }
}

experiment test_xp type: batch repeat: 1 keep_simulations: false until:cycle = nb_cycle  {
    float seed <- 1.0;
    float t;
    init {
         t <- gama.machine_time ;
    }
    reflex results {
        write "Simulation time (in s): " + ((gama.machine_time - t)/ 1000)with_precision 2;
    }
}

[lesquoyb]

I've finished implementing the file buffering in the csv and text delegates for the save statement. Currently users can pick a buffering strategy with the facet buffering of the save statement. Available strategies are:

• no_buffering, which is the old behaviour, content is directly written in the file when the save statement is reached
• per_cycle, the content is buffered and will be written in the file at the end of a cycle
• per_simulation, the content is buffered and will be written in the file when closing the simulation In the same simulation you can mix save calls with those three buffering strategies and they will be solved individually.

Here is an example model to showcase the different behaviors:

model buffering

global {

    reflex at_cycle {
        save "at cycle " + cycle to:"data.csv" header:false rewrite:false buffering:"per_cycle";
    }
    reflex at_cycle2 {
        save "at cycle2 " + cycle + " should appear after 'at cycle "+ cycle+"' as it's asked in that order"to:"data.csv" header:false rewrite:false buffering:"per_cycle";
    }
    reflex no_buffering {
        save "at cycle " + cycle + " too, should appear before all the other, as it's executed right when the code is reached" rewrite:false to:"data.csv" header:false buffering:"no_buffering";
    }
    reflex end_of_simulation {
        save "Run at cycle " + cycle + " but should be appended at the end of the file" to:"data.csv" header:false rewrite:false buffering:"per_simulation";
    }

}

experiment a type:batch until:cycle=10 autorun:true{

}

Here is what's left to do:

• [x] add a default buffering strategy in the settings
• [x] add the buffering strategies in the save of the GamaFile objects
• [x] handle the case of rewrite (seems to work rn but not fully tested)
• [ ] handle other formats than text and csv
• [ ] cache the file objects ? at least remove the file object opening from the save statement as it will be recreated in the WriteController. Must pay attention to the case of rewrite
• [ ] parallelize the flushing of files for one owner (must either wait them all before leaving the function or implement a lock because two simulations can write the same files)
• [x] add operators to flush manually a file or the whole simulation

[lesquoyb]

I have a functional first version, @AlexisDrogoul could you have a look and tell me if you see anything wrong ? I think WriteController.java is in the wrong package but I'm not sure where it should go.

And here is a model to test the functionalities:

/**
* Name: buffered
* Based on the internal empty template. 
* Author: baptiste
* Tags: 
*/

model buffering

global {

    reflex at_cycle {
        save "at cycle " + cycle to:"data.csv" header:false rewrite:false buffering:"per_cycle";
    }
    reflex at_cycle2 {
        save "at cycle2 " + cycle + " should appear after 'at cycle "+ cycle+"' as it's asked in that order"to:"data.csv" header:false rewrite:false buffering:"per_cycle";
    }
    reflex no_buffering {
        save "at cycle " + cycle + " too, should appear before all the other, as it's executed right when the code is reached" rewrite:false to:"data.csv" header:false buffering:"no_buffering";
    }
    reflex end_of_simulation {
        save "Run at cycle " + cycle + " but should be appended at the end of the file" to:"data.csv" header:false rewrite:false buffering:"per_simulation";
    }

    reflex combo_breaker when:cycle=5{
        let s <- flush_all_files(simulation);
    }

}

experiment a type:batch until:cycle=10 autorun:true{

}

[lesquoyb]

I'm merging this as it seems to be working and I opened separate issues for the remaining improvements I described above