orichters
commented
6 months ago -
code suggestion by @0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q to expand
ato the joint size ofaandb:somebody_name_this_function <- function(a, b, fill = 0) { r <- new.magpie(cells_and_regions = union(getRegions(a), getRegions(b)), years = sort(union(getYears(a, as.integer = TRUE), getYears(b, as.integer = TRUE))), names = union(getNames(a), getNames(b)), fill = fill, sets = names(dimnames(b))) r[getRegions(a),getYears(a),getNames(a)] <- a return(r) } B_ <- B[,t[-2],] %>% somebody_name_this_function(A) A + B_which returns:
An object of class "magpie" , , 1 t i y1995 y2005 y2015 AFR 1105.333 696.44 1821.22 CPA 2561.270 1429.53 3018.20
> system.time({
+ x <- vm_demFENonEnergySector <- readGDX(gdx, "vm_demFENonEnergySector",
+ field = "l", restore_zeros = TRUE,
+ react = "silent")[,t,]})
user system elapsed
12.134 4.380 16.553
> format(object.size(x), units = 'auto')
[1] "1.5 Gb"> system.time({
+ ref <- new.magpie(
+ cells_and_regions = readGDX(gdx, 'regi'),
+ years = readGDX(gdx, 't'),
+ names = quitte::cartesian(
+ readGDX(gdx, 'se2fe') %>%
+ mutate(x = paste(.data$all_enty, .data$all_enty1,
+ sep = '.')) %>%
+ pull('x'),
+ 'indst',
+ readGDX(gdx, 'all_emiMkt')),
+ fill = 0,
+ sets = c('t', 'regi', 'entySE', 'entyFE', 'sector', 'emiMkt'))
+
+
+
+ y <- vm_demFENonEnergySector <- readGDX(gdx, "vm_demFENonEnergySector",
+ field = "l", restore_zeros = FALSE,
+ react = "silent") %>%
+ somebody_name_this_function(ref)})
user system elapsed
0.133 0.000 0.129
> format(object.size(y), units = 'auto')
[1] "257.8 Kb"
0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q
fbenke-pik
mellamoSimon
ricardarosemann
cd /p/projects/remind/modeltests/remind/output; grep oom-kill testOne*/log_output.txtrestore_zeros = FALSEbecause we declare all variables over the general supersets (e.g.,tall,all_enty), but usually define them only over subsets (t,entyFE).restore_zeros = FALSEreads in only data that is not zero, which is usually what we want. But if some axis of the data (e.g. all values in 2010, or everythingfehos) happens to be entirely zero (usually that happens for specific gdxes only), that dimension of the resulting magpie object differs (e.g. c(2005, 2015, …) instead of c(2005, 2010, 2015, …)) and it is incompatible with all the rest of the objects we have. What would be needed is a function to expands a variable read in withrestore_zeros = FALSEto the actual sets it should have. But that is a moving target, as it depends on all the other variables/parameters used in conjunction with it.magpie_expandis not the solutionSome code to start with:
The question is how to be able to sum, mbind etc.
AandB.The problem is that the dimensions each variable is defined (not declared) over in the GAMS code (e.g. t instead of tall, se2fe(entySE,entyFE,te) instead of all all_enty/all_enty/all_te combinations) is not pre-defined, and the problem depends on which other variables the variables is combined with (in magpie operations) and what data those variables have.