cannot extract values csvs using the most recent version of the master branch

[manorman5]

hey @brews and @jrising , sorry to bother you both again. I ran into an issue extracting value csvs on the master branch. When I go back to commits pre the results.sum_into_data function, I do not have this issue. I wrote up a couple fixes on this branch. I made a slight modification to quantiles.py, so it could receive a years variable from results.sum_into_data and two minor tweaks to the results.sum_into_data function. I doubt these will be the long term solutions to the two bugs I ran into, but I tried 🙃 .

[brews]

+1 to @mayaanorman for working out a solution on your own (even a temporary one)!

Real quick:

• Is it throwing an error for you or what console output do you get (when it fails)?
• It worked pre 47b1c2712c3b284aad6cb42500a4eec81d443596?
• Can I please get the exact bash and run config yml you're running under? (cut-n-paste or drag-n-drop here)

[manorman5]

hey @brews! My configs/quantiles call for extracting point estimate (vs variance) valuescsvs worked at this commit. Sorry I know that is awhile back. I had trouble deciphering which commit I could go back to that wasn't contaminated with delta method updates. I went back really far to convince myself that my inputs were probably not causing the problem :).

So there were three bugs I "fixed":

• output_format is referenced in results.sum_into_data(), but it is undefined. Here is the error message:

  /shares/gcp/outputs/energy/impacts-blueghost/median_OTHERIND_electricity_TINV_clim_income_spline_GMFD/median/rcp45/surrogate_CanESM2_89/high/SSP3
  Failed to read /shares/gcp/outputs/energy/impacts-blueghost/median_OTHERIND_electricity_TINV_clim_income_spline_GMFD/median/rcp45/surrogate_CanESM2_89/high/SSP3/FD_FGLS_inter_climGMFD_Exclude_all-issues_break2_semi-parametric_poly2_OTHERIND_electricity_TINV_clim_income_spline.nc4
  Traceback (most recent call last):
  File "/home/manorman/repos/prospectus-tools/gcp/extract/lib/results.py", line 162, in sum_into_data
  if 'region' in config.get('file-organize', []) and 'year' not in config.get('file-organize', []) and output_format == 'valuescsv':
  NameError: global name 'output_format' is not defined

• years is referenced in line 116 of quantiles.py, but it is undefined
• deltamethod_variance() if referenced in line 69, but it is undefined

Errors were thrown for all of these issues as I worked through them. It will take me a little bit of time to regenerate all of them, but if that would be helpful I can definitely do that. Sorry I should have copied them down as they were received.

To run into the first two bugs here are the commands and configs I used--

bash command:

python -u quantiles.py <path to config>/energy-extract-impactpc-median_OTHERIND_electricity.yml  --region=ARE.3 --suffix=_ARE.3_impactpc_median_fulladapt FD_FGLS_inter_climGMFD_Exclude_all-issues_break2_semi-parametric_poly2_OTHERIND_electricity_TINV_clim_income_spline -FD_FGLS_inter_climGMFD_Exclude_all-issues_break2_semi-parametric_poly2_OTHERIND_electricity_TINV_clim_income_spline-histclim

config:

results-root: /shares/gcp/outputs/energy/impacts-blueghost/median_OTHERIND_electricity_TINV_clim_income_spline_GMFD
output-dir: /shares/gcp/social/parameters/energy/extraction/multi-models/rationalized_code/break2_Exclude_all-issues_semi-parametric/TINV_clim_income_spline_GMFD/median_OTHERIND_electricity_TINV_clim_income_spline_GMFD/impactpc
file-organize: [ssp, region]
do-montecarlo: no
only-rcp: null
only-models: all
only-iam: null
column: rebased
output-format: valuescsv

To run into the last bug (after I got through the first two) here are the commands and configs I used -- bash command:

python -u quantiles.py <path to config>/energy-extract-impactpc-median_OTHERIND_electricity_dm.yml  --region=ARE.3 --suffix=_ARE.3_impactpc_median_fulladapt_dm FD_FGLS_inter_climGMFD_Exclude_all-issues_break2_semi-parametric_poly2_OTHERIND_electricity_TINV_clim_income_spline -FD_FGLS_inter_climGMFD_Exclude_all-issues_break2_semi-parametric_poly2_OTHERIND_electricity_TINV_clim_income_spline-histclim

config:

results-root: /shares/gcp/outputs/energy/impacts-blueghost/median_OTHERIND_electricity_TINV_clim_income_spline_GMFD_dm
output-dir: /shares/gcp/social/parameters/energy/extraction/multi-models/rationalized_code/break2_Exclude_all-issues_semi-parametric/TINV_clim_income_spline_GMFD/median_OTHERIND_electricity_TINV_clim_income_spline_GMFD/impactpc
file-organize: [ssp, region]
do-montecarlo: no
only-rcp: null
only-models: all
only-iam: null
deltamethod: true
column: rebased
output-format: valuescsv

Let me know if you have questions!

PS: sorry for submitting this so many times... im not used to the key bindings happening rn 🙃

[brews]

@mayaanorman This is excellent! I may not be able to get to this immediately but this makes it much easier for me to replicate and dig into this. Thanks so much!

[brews]

Hey @mayaanorman, I'm checking our issue backlog. Are you still having this issue?

[manorman5]

hey @brews! @jrising and I wrapped this up issue up with this pull request. Sorry i should have closed this issue!