Update and clean 01_ scripts

[lidefi87]

• [x] Look at potentially merging the 01_ scripts. It should work with LMEs and FAOs, ideally just one script.
• [x] Remove area of grid cell calculation and use data provided by ISIMIP team
• [x] Update from raster to terra.
• [x] Divide big functions into shorter ones.

[lidefi87]

First two functions have been broken down into smaller ones that can be reused later in the script.

[lidefi87]

Functions were moved to supporting_functions.R and the application of these functions have been kept in: 01_getinputs_byFAO_ISIMIP3a.R.

Since data is available in NCI's gadi, the first obsclim and ctrlclim datasets are being processed there. Processed data can be found in: /g/data/vf71/fishmip_inputs/ISIMIP3a/processed_forcings/fao_inputs. The file naming system has been changed so it is more descriptive of the file content.

After the first data processing step is completed, I will continue cleaning the script.

[lidefi87]

Script 01_getinputs_byFAO_ISIMIP3a.R included the following comments:

4. check calculation below in terms of sphy and sphy and adopt same ----

variable names

function below: names(pp) <- c("lon", "lat", "t", "lphy", "sphy", "sbt", "sst") depth file: lat, lon, depth the 2 files are saved directly withing the function as RData
the function is applied to different protocols for CMIP63a protocols are observed and control at 0.25 deg and 1 deg resolution. However, you are only using observed at LME scale for calibration

• so run only this scenario.

However, the depth variable is not included in the FAO inputs data for obsclim: /rd/gem/private/fishmip_inputs/ISIMIP3a/fao_inputs/obsclim/025deg or for ctrlclim: /rd/gem/private/fishmip_inputs/ISIMIP3a/fao_inputs/ctrlclim/025deg.

For this reason, the processed files available in Gadi do not include the depth variable.

After effort data for the FAO was loaded in the same script, the following comment was included:

calculate climate inputs by Year as effort is by Year no - skip as Julia would like monthly inputs

I do not know what this refers to because the effort data loaded does not include months, it has yearly data.

The plot checking section of the catch and effort data selects three different FAO regions prior to making a plot that was saved in the Outputs folder called Effort_FAO1_check.pdf. If I run the code as originally written, that is selecting plot_df[[3]] to create the plot, the output I get does not match the saved PDF. But, if I use plot_df[[1]] to create the plot, then it matches the saved PDF exactly (see figure below). I am assuming that the code was not updated after the image was created and saved to the repo.

Step 7 in this same script has the comments shown below:

7. go to step 2 - batch create inputs.R ----

in batch create input, the code uses getgridin_ISIMIP3b to calculate intercept and slope the below are the steps you need from that function (checked with Julia - line 14 to 22)

I cannot find the batch create inputs.R script. It is not clear to me what this comment is referring to.

[juliablanchard]

Thanks Denisse, We do use depth elsewhere in the code -so need it in Gadi ( depth files should not be as big as the others as no ‘time’ variable)

———— Sent from my phone

---

From: Lidefi87 @.> Sent: Wednesday, July 10, 2024 8:27:15 AM To: Benthic-Pelagic-Size-Spectrum-Model/lme_scale_calibration_ISMIP3a @.> Cc: Subscribed @.***> Subject: Re: [Benthic-Pelagic-Size-Spectrum-Model/lme_scale_calibrationISMIP3a] Update and clean 01 scripts (Issue #4)

Script 01_getinputs_byFAO_ISIMIP3a.Rhttps://github.com/Benthic-Pelagic-Size-Spectrum-Model/lme_scale_calibration_ISMIP3a/blob/main/01_getinputs_byFAO_ISIMIP3a.R included the following comments:

4. check calculation below in terms of sphy and sphy and adopt same ----

variable names

function below: names(pp) <- c("lon", "lat", "t", "lphy", "sphy", "sbt", "sst") depth file: lat, lon, depth the 2 files are saved directly withing the function as RData the function is applied to different protocols for CMIP63a protocols are observed and control at 0.25 deg and 1 deg resolution. However, you are only using observed at LME scale for calibration

• so run only this scenario.

However, the depth variable is not included in the FAO inputs data for obsclim: /rd/gem/private/fishmip_inputs/ISIMIP3a/fao_inputs/obsclim/025deg or for ctrlclim: /rd/gem/private/fishmip_inputs/ISIMIP3a/fao_inputs/ctrlclim/025deg.

For this reason, the processed files available in Gadi do not include the depth variable.

— Reply to this email directly, view it on GitHubhttps://github.com/Benthic-Pelagic-Size-Spectrum-Model/lme_scale_calibration_ISMIP3a/issues/4#issuecomment-2219152714, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABGD7OXHDEPL5MTY6JC2LUTZLR5WHAVCNFSM6AAAAABKGLU3L2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJZGE2TENZRGQ. You are receiving this because you are subscribed to this thread.Message ID: @.***>

This email is confidential, and is for the intended recipient only. Access, disclosure, copying, distribution, or reliance on any of it by anyone outside the intended recipient organisation is prohibited and may be a criminal offence. Please delete if obtained in error and email confirmation to the sender. The views expressed in this email are not necessarily the views of the University of Tasmania, unless clearly intended otherwise.

[lidefi87]

Hi @juliablanchard, please do not feel like you have to reply on your time off. I am just keeping a record of the things I find in the code as I go so I can talk about them when you are back in Hobart. You will probably receive quite a few notifications, but please ignore them until you are back and we can talk about them. Even if I get stuck with this, I am also working on a couple more things (updating regional app and processing data for the new submissions), so I can move to that other task until you are back.

[juliablanchard]

Ok :) thanks Denisse! Catch up when I'm back

———— Sent from my phone

---

From: Lidefi87 @.> Sent: Wednesday, July 10, 2024 8:52:11 AM To: Benthic-Pelagic-Size-Spectrum-Model/lme_scale_calibration_ISMIP3a @.> Cc: Julia Blanchard @.>; Mention @.> Subject: Re: [Benthic-Pelagic-Size-Spectrum-Model/lme_scale_calibrationISMIP3a] Update and clean 01 scripts (Issue #4)

Hi @juliablanchardhttps://github.com/juliablanchard, please do not feel like you have to reply on your time off. I am just keeping a record of the things I find in the code as I go so I can talk about them when you are back in Hobart. You will probably receive quite a few notifications, but please ignore them until you are back and we can talk about them. Even if I get stuck with this, I am also working on a couple more things (updating regional app and processing data for the new submissions), so even I can move to that other task until you are back.

— Reply to this email directly, view it on GitHubhttps://github.com/Benthic-Pelagic-Size-Spectrum-Model/lme_scale_calibration_ISMIP3a/issues/4#issuecomment-2219234689, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABGD7ORXCANJMDJGOYAFY23ZLSATXAVCNFSM6AAAAABKGLU3L2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJZGIZTINRYHE. You are receiving this because you were mentioned.Message ID: @.***>

This email is confidential, and is for the intended recipient only. Access, disclosure, copying, distribution, or reliance on any of it by anyone outside the intended recipient organisation is prohibited and may be a criminal offence. Please delete if obtained in error and email confirmation to the sender. The views expressed in this email are not necessarily the views of the University of Tasmania, unless clearly intended otherwise.

[lidefi87]

The 01_getinputs_byFAO_ISIMIP3a.R script has been significantly reduced in size for three reasons:

1. Functions were moved to a new script (supporting_functions.R)
2. Data were incrementally changed and a number of variables were created and kept in memory with data that was only marginally different. These variables were often only used once to create a new variable with small changes, so all processing was done and assigned to one variable. This will also likely reduce the amount of memory needed to run the script.
3. Comments added when checking the workflow have been removed as I assume the script has been quality controlled and these are not needed anymore. These can be accessed through git logs.

Work has been completed up to line 914: ##### END LME scale -----. I could run the script up to this point in about 3 hours using a Large compute size in Gadi.

Functions defined after this point appear to be a repetition of first batch of functions, with only small changes. I will generalise original functions to avoiding repeating them.

[lidefi87]

LME inputs at 1 degree resolution were renamed so instead of onedeg, the resolution in the name followed CMIP file naming convention: 60arcmin. These files can be found under folders: /g/data/vf71/fishmip_inputs/ISIMIP3a/lme_inputs/ctrlclim/1deg and /g/data/vf71/fishmip_inputs/ISIMIP3a/lme_inputs/obsclim/1deg.

Functions have been generalised in (supporting_functions.R) script.

The 01_getinputs_byFAO_ISIMIP3a.R script has been cleaned up to line 815. Gridded data at 0.25 degree horizontal resolution being processed now, the 1 degree data is already available in Gadi for obsclim: /g/data/vf71/fishmip_inputs/ISIMIP3a/processed_forcings/fao_inputs_gridcell/obsclim/1deg and ctrlclim: /g/data/vf71/fishmip_inputs/ISIMIP3a/processed_forcings/fao_inputs_gridcell/ctrlclim/1deg.

[lidefi87]

Code updated and working up to line 1074 from original script. Files have been processed for 0.25 degree and 1 degree and are available in Gadi.

[lidefi87]

Finished updating 01_getinputs_byFAO_ISIMIP3a.R script. All inputs have been calculated using this version of the code and are now available in Gadi.

Moving onto merging this script with 01_getinputsISIMIP3a.R.

[lidefi87]

Functions in the supporting_functions.R script have been adapted to work with LME data.

These functions have been tested and they can do the work up to line 840 from 01_getinputsISIMIP3a.R.

Note that plot saved in line 547 is not available in the Output folder, so it cannot be compared, but the plot produced with the updated code is available in the Outputs folder and it is called Effort_LME1_check.pdf.

[lidefi87]

Functions in the supporting_functions.R script have been tested up to line 1053 in the 01_getinputsISIMIP3a.R script. Script has been run with 1 degree and 0.25 degree outputs.

Checked results against data included as comments from line 1058 in the same script and the results for all columns, except intercept are the same. The calculations are done exactly the same as included in the latest version of the code, so it is not obvious why there is a difference.

I calculated the intercept with the gridded data using the formula in line 1008: intercept = GetPPIntSlope(sphy,lphy,mmin=10^-14.25,mmid=10^-10.184,mmax=10^-5.25,depth,output="intercept") and I still get the different results than what is included in 01_getinputsISIMIP3a.R.

I also copied and pasted the results seen from line 1062 in the original script and recalculated the intercept and I still get the a different result.

Finally, I applied copied the code for the GetPPIntSlope function included in the dbpm_model_functions.R script (original script is on the left, which is exactly the same as the edited version where I added space vertically and reduced horizontal scrolling). Even then I got a different result for the intercept column.

I then checked the file referenced in 1058 (which can be found here: /rd/gem/private/fishmip_inputs/ISIMIP3a/processed_forcings/lme_inputs_gridcell/obsclim/1deg), and values in this file matched the outputs produced in the new code (i.e., they were different from the comments from line 1058). In fact, the intercept column in all the examples included in the 01_getinputsISIMIP3a.R script are not correct and they do not match the data available in GEM.

From line 1190 in the 01_getinputsISIMIP3a.R script, it is not necessary to run the code. This is because this function processes gridded data from GFDL-MOM6-COBALT2 without adding data at LME or FAO level. The same data is used for both LME and FAO.

[lidefi87]

All checks have been finalised and data has been processed for the 1 degree and 0.25 degree horizontal resolution.

The 01_getinputsISIMIP3a.R script has been removed as it is no longer relevant.

The 01_getinputs_byFAO_ISIMIP3a.R script has been renamed 01_getinputs_ISIMIP3a.R. This new script processes data for FAO and LME regions.