search

femiguez / apsimx

R package for APSIM-X
https://femiguez.github.io/apsimx-docs/
45 stars 19 forks source link

inspect_apsimx example isnt working #159

Open peter-devoil opened 3 weeks ago

peter-devoil commented 3 weeks ago

When running the vignette for a recent wheat version (post CAMP), it shows a problem using grep to test strings when searching for nodes.

The error is 

> inspect_apsimx_replacement("wheatCultivarParmFromR.apsimx", src.dir = ".",
+                            node.string = "Wheat.Phenology.Phyllochron.BasePhyllochron", 
+                            display.available = T, verbose = T)
Replacements:  Wheat 
node: Wheat 
Level: node 
list Name: Wheat 
list length: 7 
list names: $type PlantType Name ResourceName Children Enabled ReadOnly 
Children: Yes 
Children length: 21 
Children Names: Introduction Arbitrator Phenology Structure Grain Root Leaf Spike Stem AboveGround AboveGroundLive AboveGroundDead BelowGround Total TotalLive TotalDead Ear StemPlusSpike Cultivars MortalityRate SeedMortalityRate 
node child: Phenology 
list Name: Phenology 
list length: 6 
list names: $type Name ResourceName Children Enabled ReadOnly 
Children: Yes 
Children length: 30 
Children Names: Memo ThermalTime Photoperiod HaunStage Phyllochron FinalLeafNumber HeadEmergencePpSensitivity HeadEmergenceLongDayBase Germinating Emerging Vernalising SpikeletDifferentiation StemElongation HeadEmergence EarlyFlowering GrainDevelopment GrainFilling Maturing Ripening ReadyForHarvesting Zadok TerminalSpikeletDAS FlagLeafDAS HeadingDAS FloweringDAS MaturityDAS EmergenceDAS PTQ PhyllochronPpSensitivity CAMP 
node subchild: Phyllochron 
Error in inspect_apsimx_replacement("wheatCultivarParmFromR.apsimx", src.dir = ".",  : 
  More than one subchild found. Make it unique (see regular expressions)

So the grep statement is picking up "PhyllochronPpSensitivity" as a duplicate of "Phyllochron". Setting grep.options (to fixed=T) isnt any help.

Perhaps only simple string equality is needed (without regex?)

femiguez commented 3 weeks ago

@peter-devoil Thanks for the report! I'll review the vignette and fix it. If you need it, this might work (using '$' to match 'Phyllochron' instead of multiple options:

inspect_apsimx_replacement("Wheat_phyllochron.apsimx",
                           node = "Wheat",
                           node.child = "Phenology",
                           node.subchild = "Phyllochron$",
                           node.subsubchild = "BasePhyllochron")
JongChao commented 3 weeks ago

Hi @femiguez I'm using the function to optimize parameters, and encounter some problems in the recent APSIM NG version . I add a new wheat cultivar (Newone) in .apsimx file. Here, I want to optimize RUE and two phenology parameters of the new cultivar. When I run the inspect_apsimx_replacement(), I can't find related parameters. How I should find its and construct the pp1, pp2, and pp3? I attach my .apsimx file and related error. Hope to get your help. 

## Finding RUE
inspect_apsimx_replacement("WheatPar.apsimx", src.dir = extd.dir,
+                            node.string = "Wheat.Leaf.Photosynthesis.RUE",
+                            parm = "FixedValue",verbose = FALSE)
These positions matched  Models.Core.Folder   1 3 4 
Error: Multiple root nodes found. Please provide a position
 ## Finding crop cultivar parameter
#Phyllochron.BasePhyllochron.FixedValue
inspect_apsimx_replacement("WheatPar.apsimx", src.dir = extd.dir,
+                            node.string = "Wheat.Newone.Phenology.Phyllochron.BasePhyllochron",
+                            parm = "FixedValue", verbose = FALSE)
These positions matched  Models.Core.Folder   1 3 4 
Error: Multiple root nodes found. Please provide a position
#CAMP.FLNparams.MinLN
inspect_apsimx_replacement("WheatPar.apsimx", src.dir = extd.dir,
+                             node.string = "Wheat.Newone.Phenology.CAMP.FLNparams",
+                            parm = "MinLN", verbose = FALSE)
These positions matched  Models.Core.Folder   1 3 4 
Error: Multiple root nodes found. Please provide a position

## Constructing the paths
pp1 <- "Wheat.Leaf.Photosynthesis.RUE.FixedValue"
pp2 <- "Wheat.Newone.Phenology.Phyllochron.BasePhyllochron.FixedValue"
pp3 <-"Wheat.Newone.Phenology.CAMP.FLNparams.MinLN"

WheatPar.zip

femiguez commented 3 weeks ago

Hi @JongChao The code below will generate the paths you need, but I think what you have will not work. In recent versions of APSIM they have made it harder to modify the model by changing replacements. What you need to do is use a version of models (as in the APSIM github repo) that matches the version of APSIM you are using. Only in this case, you can change replacements. I have a function in the model that makes it easy to insert replacements. It is called 'insert_replacment_node'. If you are not sure what this all means, you need to tell me which version of APSIM you are using and send me the file you are working with.

pp1 <- inspect_apsimx_replacement("WheatPar.apsimx", 
                                  node = "Wheat",
                                  root = list("Models.Core.Folder", 1),
                                  node.child = "Newone",
                                  parm = "FixedValue",
                                  print.path = TRUE)

pp2 <- inspect_apsimx_replacement("WheatPar.apsimx", 
                                  node = "Wheat",
                                  root = list("Models.Core.Folder", 1),
                                  node.child = "Newone",
                                  parm = "MinLN",
                                  print.path = TRUE)

pp3 <- inspect_apsimx_replacement("WheatPar.apsimx", 
                                  node = "Wheat",
                                  root = list("Models.Core.Folder", 1),
                                  node.child = "Newone",
                                  parm = "PpLN",
                                  print.path = TRUE)

Notice that I use '1' to pick the right 'Models.Core.Folder', this is what the error message above meant. Also, for 'parm' you only need a 'string' that makes the value in 'Command' unique.

peter-devoil commented 3 weeks ago

The background to the change in apsim that broke this method was/is being discussed here - the way chunks of json code (ie plant models) are copied/not copied from the installation's resources folder changed late last year, and the entire json code is no longer stored in the user's replacements folder - only a placeholder fragment with a pointer to the resources folder.

This makes it hard for a json inspector to find out what (let alone set the value of) a variable in that resources tree might be - as it's not there!

The other approach is to somehow use the replacement syntax that is used for apsim's batch / command line operation ( described here ) when evaluating the model. I'm not 100% sure this will work for the resource models that "aren't there".

Both approaches are still slow, as the links within a simulation have to be resolved before every run (can be ~50% of run time for single season simulations); the continuous methods (one of the many "server" methods) talk over a network to a live, running simulation trying to avoid this link resolution phase.

JongChao commented 3 weeks ago

Hi @femiguez I could find related crop parameters from the 'WheatPar.apsimx' file using your code. As you mentioned, I can't run optim_apsimx (), the specific errors are follow:

pp1 <- inspect_apsimx_replacement("WheatPar.apsimx", 
+                                   node = "Wheat",
+                                   root = list("Models.Core.Folder", 1),
+                                   node.child = "Newone",
+                                   parm = "FixedValue",
+                                   print.path = TRUE)
Replacements:  Wheat 
node: Wheat 
node child: Newone 
 : [Phenology].Phyllochron.BasePhyllochron.FixedValue =100 
Command 
Parm path:  .Simulations.Replacements.Wheat.Newone.FixedValue 
no node sub-children available or parm not equal to null 
> pp2 <- inspect_apsimx_replacement("WheatPar.apsimx", 
+                                   node = "Wheat",
+                                   root = list("Models.Core.Folder", 1),
+                                   node.child = "Newone",
+                                   parm = "MinLN",
+                                   print.path = TRUE)
Replacements:  Wheat 
node: Wheat 
node child: Newone 
 : [Phenology].CAMP.FLNparams.MinLN =9 
Command 
Parm path:  .Simulations.Replacements.Wheat.Newone.MinLN 
no node sub-children available or parm not equal to null 
> pp3 <- inspect_apsimx_replacement("WheatPar.apsimx", 
+                                   node = "Wheat",
+                                   root = list("Models.Core.Folder", 1),
+                                   node.child = "Newone",
+                                   parm = "PpLN",
+                                   print.path = TRUE)
Replacements:  Wheat 
node: Wheat 
node child: Newone 
 : [Phenology].CAMP.FLNparams.PpLN = 3.0 
Command 
Parm path:  .Simulations.Replacements.Wheat.Newone.PpLN 
no node sub-children available or parm not equal to null 
> pp3
[1] ".Simulations.Replacements.Wheat.Newone.PpLN"
> ###
> wop <- optim_apsimx("WheatPar.apsimx", 
+                     src.dir = extd.dir, 
+                     parm.paths = c(pp1, pp2),
+                     data = obsWheat, 
+                     weights = "mean",
+                     replacement = c(TRUE, TRUE),
+                     root = list("Models.Core.Folder", 1),
+                     initial.values = c(100, 9.0))
Error in stats::optim(par = rep(1, length(parm.paths)), fn = obj_fun,  : 
  function cannot be evaluated at initial parameters
In addition: Warning messages:
1: In auto_detect_apsimx() : Searching the Windows registry for APSIM-X
2: call dbDisconnect() when finished working with a connection 
3: In auto_detect_apsimx() : Searching the Windows registry for APSIM-X

I run 'insert_replacement_node ()' function. 

wheat <- get_apsimx_json(model = "Wheat", wrt.dir = ".")
insert_replacement_node("WheatPar174.apsimx", src.dir = ".", rep.node = wheat)

The version of 'wheat.json' is 174, but the version I am using APSIM NG is 171 (APSIM2024.3.7421.0). I don't know how to get the 171-version 'wheat.json'. Here, I try manually revising the version of 'WheatPar.apsimx' to 174 and save a new file-"WheatPar174.apsimx". After runing the above code (insert_replacement_node), I again revised the version of 'WheatPar174.apsimx' to 171 and save it as "WheatPar174_171.apsimx".

After then, I still get some errors from 'WheatPar174_171.apsimx' when constract pp1, pp2. This is because I used unmatched 'wheat.json' file?

> pp1 <- inspect_apsimx_replacement("WheatPar174_171.apsimx", 
+                                   node = "Wheat",
+                                   root = list("Models.Core.Folder", 1),
+                                   node.child = "Newone",
+                                   parm = "FixedValue",
+                                   print.path = TRUE)
Replacements:  Wheat Wheat 
node: Wheat 
Error in inspect_apsimx_replacement("WheatPar174_171.apsimx", node = "Wheat",  : 
  More than one node found. Make it unique (see regular expressions)
> pp1 <- inspect_apsimx_replacement("WheatPar174_171.apsimx", 
+                                   node = "Wheat",
+                                   root = list("Models.Core.Folder", 1),
+                                   node.child = "Newone",
+                                   parm = "FixedValue",
+                                   print.path = TRUE)
Replacements:  Wheat Wheat 
node: Wheat 
Error in inspect_apsimx_replacement("WheatPar174_171.apsimx", node = "Wheat",  : 
  More than one node found. Make it unique (see regular expressions)

I attach related data as 'WheatPar20240610.zip'. WheatPar20240610.zip

femiguez commented 3 weeks ago

@peter-devoil @JongChao It is possible to still use this method, but you need to match the Models (json files) to the APSIM version you are running. In this way, you can still edit replacements, add cultivars, etc. Your code is mostly not working because if you are running APSIM2024.03.7421.0 you need to use Model versions 171, otherwise it does not work. (The problem is that Models version 171 are not easily available - I keep copies however). The function 'insert_replacement_node' should give you a warning if there is a mismatch between the APSIM version and Models. I'm attaching an edited version of Wheat called 'Wheat_171.apsimx', but I got rid of the 'Newone' cultivar. You need to create new cultivars if you want to optimize them against observed data. Hope this helps! WheatPar_20240610_JC.zip