ncx-gitbot
commented
1 year ago NCX response: Our business as usual model is a hierarchical statistical model that predicts one-year harvest risk and intensity based on FIA training data and a suite of covariates that include geographic, biological, economic, and sociological factors. Partial pooling across forest types ensures that the model is able to leverage the similarity and ubiquity of covariate relationships across the forests of the continental U.S. while still allowing for regionally specific differences. Predicting behavior of any type, which is the basis for any forest carbon program, is not straightforward, and depends on models whose performance can be measured. Our revised methodology requires the propagation of model uncertainty through to calcation of final credits, as well as reporting of benchmarking for all models.
Commenter Organization: CarbonPlan
Commenter: Danny Cullenward
2021 Deferred Harvest Methodology Section: No Section Indicated
Comment: It is striking that we aren’t able to review the method or see any comprehensive description of its accuracy or comparison against validation benchmarks. As far as we are aware, there is no complete and publicly reviewable description of NCX’s baseline modeling approach — neither in the Proposed Methodology nor in any other forum. 7 Under these conditions, it is not possible to tell what exactly the baseline model is, let alone if it adequately mitigates ton-year accounting’s unique additionality risks. To be clear, we do not wish to prejudge the accuracy of NCX’s models (or any other models that might be developed for use under the Proposed Methodology). But without more detail, it is impossible to evaluate the rigor of NCX’s modeling approach. As a result, we are unable to test the Proposed Methodology’s technical accuracy in projecting baseline scenarios nor its vulnerability to additionality gaming — including the novel risks introduced with 1-year crediting periods. 8 The opacity of what is actually being proposed is more than a theoretical concern. For example, Appendix A of the Proposed Methodology indicates that the model developed by NCX to predict common practice harvest patterns would be an acceptable approach. We are told that this model is trained on FIA data. 9 However, predicting harvest patterns from FIA data is a very difficult problem, especially for the small, family landowners NCX targets in its marketing efforts. 10 As described by a paper NCX itself cites in Appendix A: “[W]e found little predictive information either from the FIA, census, or NWOS data to explain harvest behavior within the private woodland owner-class … [T]heir reactive harvest behavior due to external stimuli or unplanned financial need[] confounds the ability to predict future conditions in a consistent way[.]” 11 We recognize that the Proposed Methodology requires baseline models like NCX’s to be reviewed by an expert panel and approved by Verra. 12 However, since the process does not specify formal criteria or model validation benchmarks, it does not alleviate our concern about the opacity of the baseline modeling approach and inscrutability of the Proposed Methodology’s additionality protection.
Proposed Change: Verra should require full public disclosure of all baseline scenario models approved for use with this methodology