Clean up scripts 02 and 03

[lidefi87]

@juliablanchard to clean up comments.

[lidefi87]

I edited these files to decrease horizontal scrolling needed to read them. I also added some vertical spaces to make it easier to read. This means that the code will appear longer than the original version, but I did not change the code itself.

File paths will need to be updated in both scripts, and we also need to update 03_rungridbyLME.R to use terra instead of raster.

I almost done with #4 and I plan to work on these scripts from Monday, August 29. @juliablanchard, does that work for you or would you prefer that I leave this for later?

[lidefi87]

The get_lme_inputs from the LME_calibration.R script and used in 02_runLMEcalibration.R is being updated because paths to the data used were from GEM.

I have also simplified it significantly and it can now be used with either LMEs or FAO regions. Tested up to line 285 and it produces the same outputs as original version. The code will be updated in the repository once testing is completed for this function.

The run_model function (lines 290-363) has also been updated and tested.

Work to be continued on getError function from line 367 tomorrow.

[lidefi87]

Script tested and updated up to line 397. Encountered issues with the pbapply function used in line 444.

[lidefi87]

Encountered issues with the pbapply function used in line 444.

The issue is that pbapply extracts rows in a data frame as a named numeric vector. This vector has a single column with no names and rows with names that match the original data frame columns. For example, the first row of the data frame below:

         f.u       f.v  f.minu   f.minv search.vol
1 0.01558051 0.1735269 1.95498 0.658593  0.7518921

will become:

                 [,1]        
f.u         0.1558051
f.v         1.7352690
f.minu     19.5498000
f.minv      6.5859300
search.vol  7.5189210 

The run_model function has now been adapted to work with this structure. Information about data structure expected by this function has been added to the script.

[lidefi87]

Functions: get_lme_inputs, run_model, getError, and LHSsearch from the LME_calibration.R script have been updated to work with updated outputs from 01 script and streamlined. These functions are used in 02_runLMEcalibration.R.

They have been tested against original script and data stored in GEM and outputs are the same. Testing their use with pbsapply function in 02 script. I initially set up the connection for two hours and it was not enough time. I set the connection for five hours and it may be a bit short, I may need to redo this tomorrow, so I can get an estimate of the time it would take to process all LMEs.

[lidefi87]

Since calculations for all LMEs (66 in total) take a long time, I ran out of computing time to process all LMEs. This is an issue because we need all LMEs to be processed before we can save the results. My work around this was to save results for the LHSsearch function (from the LME_calibration.R script) every time an LME is processed. Saving can be done by providing a folder where results will be stored, but this is an optional parameter. This parameter can be especially useful in the "refine estimates" section that iterates 1000 times instead of 100 times per LME to estimates the best values for the fishing parameters.

The saved results will have a single row in a csv file, which can be read all together in a single data frame. The advantage of doing this is that we can process only a subset of LMEs when parallelising work. The disadvantage is that when saving results, processing takes longer.

The individual LME results now include a region column (shown below) to identify the LME these results belong to. Additionally, the file name also has the LME number (e.g., best-fishing-parameters_LME_2_searchvol_estimated_numb-iter_100.csv).

        f.u       f.v   f.minu   f.minv search.vol      rmse region
1 0.7556098 0.8719562 1.038566 1.333471  0.3523104 0.3063338      2

Note that individual LME files can be removed after all results are merged.

[lidefi87]

Script 02_runLMEcalibration.R has been tidied up, but there are still some comments that I do not know if we should keep or not. The first section of this script (up to line 34) is almost done running.

The rungridbyLME function in script 03_rungridbyLME.R has been tidied up too. But there are still lots of comments, which I do not know if they should be kept or recorded somewhere.

[lidefi87]

All plots in sections two of 02_runLMEcalibration.R have been created. This section and the one below have been parallelised. Creating plots for all LMEs take about 2-3 minutes now.

Tuning the fishing parameters values takes approx. 1.5 hours per LME (1000 iterations per LME) and there are 26 LMEs in total. Results are saved for each LME so we can pick up from any LME.

[lidefi87]

The last section of 02_runLMEcalibration.R, which tunes finishing parameters (iter = 1000) is still being processed (7 out of 26 LME sectors are now complete). I will leave it running overnight.

I divided the rungridbyLME function in script 03_rungridbyLME.R into smaller tasks. This way we can save intermediate outputs more often and can reuse them to run specific tasks (e.g., save input forcings vs getting input forcings + calculating gridded parameters + running gridded model).

[lidefi87]

The optimisation section of script 02_runLMEcalibration.R is still being completed. We have 14 out 26 LME sectors processed, and I am planning to complete this by tomorrow afternoon.

Work on script 03_rungridbyLME.R is ongoing, but testing is scheduled to start on Thursday 15/08.

@juliablanchard script 02_runLMEcalibration.R is to ready for you to review and decide if comments should be kept or moved somewhere else (e.g., README file). I removed comments that were testing different ways of parallelising work as they did not work as well as the option left uncommented.

[lidefi87]

Script 02_runLMEcalibration.R has successfully ran for all LME sectors ahead of time.

Testing script 03_rungridbyLME.R with LME 4 because this is the smallest region.

[lidefi87]

Julia confirmed all comments can be removed from scripts 02 and 03.