Regarding general code development, the CPT had the following requests:
1-year projection for calculating Tier 3 or 4 OFLs
specify catchability as a fixed or estimated parameter or use the analytic calculation for the MLE
specify priors (e.g., gamma) using mean and variance/standard deviation for all parameters to
ease specifying priors
include an option to calculate dynamic BMSY
add the ability to “jitter” initial parameter values
add the ability to conduct retrospective analyses
add ability to estimate bycatch fishing mortality rates when observer data are missing but effort
data is available
allow different phases for “rec_ini”, “rec_dev” estimation
D’Arcy also reported on development of a Gmacs-based assessment model for SMBKC and comparison
with the 2015 assessment model. The (original) Gmacs version incorporated four seasons: 1) start July 1,
duration = 0, surveys occur; 2) duration = 0.44 yr, fisheries occur; 3) duration = 0.185 yr, MMB
calculated at end of season; 4) duration = 0.375 yr, molting/growth and recruitment calculated at end of
season. To compare with the assessment model, Gmacs parameters were selected to closely match the
size transition matrix used in the assessment model code (which André noted was different from that
reported in the assessment document). Four scenarios were evaluated using the Gmacs version, including
a model (“base”) intended to match the 2015 assessment model as closely as possible, one which
estimated selectivity on the first two size classes (“selex”) and M in 1998, one based on “selex” which
also estimated additional error CV’s on pot survey data (“CV”), and one based on “selex” but which fixed
M = 0.18 yr-1 for all years (“M”).
Results for the base Gmacs model did not match the 2015 assessment model as closely as D’Arcy or the
CPT felt they should. The CPT requested D’Arcy and Jim make a run fixing initial conditions and recruits
to the same values as in the 2015 assessment model and that the units for input data and seasonal timing
be checked to make sure all were consistent between the Gmacs and 2015 assessment models. In addition,
D’Arcy reported that he models that estimated selectivity (“selex”, “CV”, “M”) all resulted in selectivities
for the NMFS trawl and ADFG pot surveys > 1 for size class 2, which the CPT found questionable. The
CPT suggested fixing selectivity in both stages 2 and 3 to 1 for the NMFS trawl survey (only).
Following the first presentation on Monday, Jim Ianelli on Wednesday reported that the implementation
of the robust multinomial likelihood in the 2015 assessment model included an additive factor of 2*π
which was not included in Gmacs, so likelihoods from the two models wouldn’t match but should lead to
identical results. Additionally, the 2015 model had time-varying weights. Jim re-ran the 2015 model with
constant weights to compare with the Gmacs version, resulting in better but not perfect agreement for
MMB after 1986.
D’Arcy reported on some additional checks, including that one parameter not used in the Gmacs model
had been turned “on” inadvertently, but turning it off made no difference to the results. He also
implemented time-varying weights-by-stage, time-varying season lengths, and redefined seasons used in
the Gmacs version to model fishing mortality as a pulse, rather than as continuous, to match the
assessment model. After these changes, running the Gmacs model with fixed initial conditions, average
recruitment and rec devs, M in 1998/99 set to the value estimated in the assessment model led to
substantially different results between the two models, indicating a fundamental disconnect still existed.
All data units and CV’s were then checked and it was found that the units for bycatch in the pot fishery
were incorrect. Once these were corrected, the Gmacs model achieved much closer agreement with the
assessment model.
Jim, D’Arcy and Andre continued to review Gmacs and the assessment model for further dissimilarities.
On Thursday, André reported he had found an error in the assessment model code associated with
applying the growth transition matrix to the numbers-in-stage vector, that values for the transition matrix
reported in the 2015 assessment report and in the code were different, and that the timing of fisheries
between Gmacs and the assessment model was slightly different. He also reported that he had duplicated
the 2015 assessment model code in R, including the error in the growth transition matrix multiplication,
and had obtained the same numbers-in-stage as the assessment model. One effect of the matrix
multiplication error was to effectively introduce an additional 10% mortality on stage 3 crab. The CPT
noted that identifying this type of error in the current assessment model code would have been impossible
without comparing to results from completely independent code such as Gmacs.
The CPT recommends that the model using GMACs, corrected for the growth transition matrix
multiplication error, be presented to the CPT in September for use in stock status and OFL
determination for SMBKC. The CPT understands that the corrected version will be presented to the
SSC in June for their recommendations moving forward. Pending the outcome of the SSC meeting in
June, the CPT requests that some evaluation should also be included in the September report to the CPT
which compares against the previous assessment model corrected for the error.
Gmacs/SMBKC Crab Plan Team Report May 2016
Regarding general code development, the CPT had the following requests: 1-year projection for calculating Tier 3 or 4 OFLs specify catchability as a fixed or estimated parameter or use the analytic calculation for the MLE specify priors (e.g., gamma) using mean and variance/standard deviation for all parameters to ease specifying priors include an option to calculate dynamic BMSY add the ability to “jitter” initial parameter values add the ability to conduct retrospective analyses add ability to estimate bycatch fishing mortality rates when observer data are missing but effort data is available
allow different phases for “rec_ini”, “rec_dev” estimation
D’Arcy also reported on development of a Gmacs-based assessment model for SMBKC and comparison with the 2015 assessment model. The (original) Gmacs version incorporated four seasons: 1) start July 1, duration = 0, surveys occur; 2) duration = 0.44 yr, fisheries occur; 3) duration = 0.185 yr, MMB calculated at end of season; 4) duration = 0.375 yr, molting/growth and recruitment calculated at end of season. To compare with the assessment model, Gmacs parameters were selected to closely match the size transition matrix used in the assessment model code (which André noted was different from that reported in the assessment document). Four scenarios were evaluated using the Gmacs version, including a model (“base”) intended to match the 2015 assessment model as closely as possible, one which estimated selectivity on the first two size classes (“selex”) and M in 1998, one based on “selex” which also estimated additional error CV’s on pot survey data (“CV”), and one based on “selex” but which fixed M = 0.18 yr-1 for all years (“M”). Results for the base Gmacs model did not match the 2015 assessment model as closely as D’Arcy or the CPT felt they should. The CPT requested D’Arcy and Jim make a run fixing initial conditions and recruits to the same values as in the 2015 assessment model and that the units for input data and seasonal timing be checked to make sure all were consistent between the Gmacs and 2015 assessment models. In addition, D’Arcy reported that he models that estimated selectivity (“selex”, “CV”, “M”) all resulted in selectivities for the NMFS trawl and ADFG pot surveys > 1 for size class 2, which the CPT found questionable. The CPT suggested fixing selectivity in both stages 2 and 3 to 1 for the NMFS trawl survey (only). Following the first presentation on Monday, Jim Ianelli on Wednesday reported that the implementation of the robust multinomial likelihood in the 2015 assessment model included an additive factor of 2*π which was not included in Gmacs, so likelihoods from the two models wouldn’t match but should lead to identical results. Additionally, the 2015 model had time-varying weights. Jim re-ran the 2015 model with constant weights to compare with the Gmacs version, resulting in better but not perfect agreement for MMB after 1986. D’Arcy reported on some additional checks, including that one parameter not used in the Gmacs model had been turned “on” inadvertently, but turning it off made no difference to the results. He also implemented time-varying weights-by-stage, time-varying season lengths, and redefined seasons used in the Gmacs version to model fishing mortality as a pulse, rather than as continuous, to match the assessment model. After these changes, running the Gmacs model with fixed initial conditions, average recruitment and rec devs, M in 1998/99 set to the value estimated in the assessment model led to substantially different results between the two models, indicating a fundamental disconnect still existed. All data units and CV’s were then checked and it was found that the units for bycatch in the pot fishery were incorrect. Once these were corrected, the Gmacs model achieved much closer agreement with the assessment model. Jim, D’Arcy and Andre continued to review Gmacs and the assessment model for further dissimilarities. On Thursday, André reported he had found an error in the assessment model code associated with applying the growth transition matrix to the numbers-in-stage vector, that values for the transition matrix reported in the 2015 assessment report and in the code were different, and that the timing of fisheries between Gmacs and the assessment model was slightly different. He also reported that he had duplicated the 2015 assessment model code in R, including the error in the growth transition matrix multiplication, and had obtained the same numbers-in-stage as the assessment model. One effect of the matrix multiplication error was to effectively introduce an additional 10% mortality on stage 3 crab. The CPT noted that identifying this type of error in the current assessment model code would have been impossible without comparing to results from completely independent code such as Gmacs. The CPT recommends that the model using GMACs, corrected for the growth transition matrix multiplication error, be presented to the CPT in September for use in stock status and OFL determination for SMBKC. The CPT understands that the corrected version will be presented to the SSC in June for their recommendations moving forward. Pending the outcome of the SSC meeting in June, the CPT requests that some evaluation should also be included in the September report to the CPT which compares against the previous assessment model corrected for the error.