Error when attempting to generate CHN allocations template

[ZekeMarshall]

Hi Matt,

I'm receiving the following error when I attempt to generate_fu_allocation_template("CHN")

> generate_fu_allocation_template("CHN")
Error: Can't rename columns that don't exist.
x Column `0` doesn't exist.
Run `rlang::last_error()` to see where the error occurred.
In addition: Warning messages:
1: In max(E.dot) : no non-missing arguments to max; returning -Inf
2: In max(`E.dot [%]`) :

I successfully generated GHA, so I'm unsure why China is not working!

Do you receive the same error?

Thanks,

Zeke

[MatthewHeun]

Did you add "CHN" to the countries vector? You need to run the drake workflow with "CHN" included before trying to generate_fu_allocation_template("CHN")?

[ZekeMarshall]

I added "CHN" to the countries vector in the plan at the same time as GHA, and then ran a drake workflow. I did receive the following error when i ran the drake workflow. But as GHA generated successfully, I didn't think it was relevant to the process.

\
Attaching package: 'dplyr'

The following objects are masked from 'package:stats':

    filter, lag

The following objects are masked from 'package:base':

    intersect, setdiff, setequal, union

> dynamic PSUT_final
> subtarget PSUT_final_49bab2b2
x fail PSUT_final_49bab2b2
Error : target PSUT_final_49bab2b2 failed.
diagnose(PSUT_final_49bab2b2)error$message:
  Problem with `mutate()` input `matnames`.
x LHS of case 2 (`.data[["Flow"]] %>% starts_with_any_of(c(production, resources))`) must be a logical vector, not a list.
i Input `matnames` is `dplyr::case_when(...)`.
diagnose(PSUT_final_49bab2b2)error$calls:
   1. \-global::make_psut(Specified, countries)
   2.   \-filter(SpecifiedIEAData, Country %in% countries) %>% prep_psut()
   3.     +-base::withVisible(eval(quote(`_fseq`(`_lhs`)), env, env))
   4.     \-base::eval(quote(`_fseq`(`_lhs`)), env, env)
   5.       \-base::eval(quote(`_fseq`(`_lhs`)), env, env)
   6.         \-`_fseq`(`_lhs`)
   7.           \-magrittr::freduce(value, `_function_list`)
   8.             +-base::withVisible(function_list[[k]](value))
   9.             \-function_list[[k]](value)
  10.               \-IEATools::prep_psut(.)
  11.                 \-`%>%`(...)
  12.                   +-base::withVisible(eval(quote(`_fseq`(`_lhs`)), env, env))
  13.                   \-base::eval(quote(`_fseq`(`_lhs`)), env, env)
  14.                     \-base::eval(quote(`_fseq`(`_lhs`)), env, env)
  15.                       \-IEATools:::`_fseq`(`_lhs`)
  16.                         \-magrittr::freduce(value, `_function_list`)
  17.                           \-function_list[[i]](value)
  18.                             \-IEATools::add_psut_matnames(...)
  19.                               \-`%>%`(...)
  20.                                 +-base::withVisible(eval(quote(`_fseq`(`_lhs`)), env, env))
  21.                                 \-base::eval(quote(`_fseq`(`_lhs`)), env, env)
  22.                                   \-base::eval(quote(`_fseq`(`_lhs`)), env, env)
  23.                                     \-IEATools:::`_fseq`(`_lhs`)
  24.                                       \-magrittr::freduce(value, `_function_list`)
+-base::withVisible(function_list[[k]](value))
  26.                                         \-function_list[[k]](value)
  27.                                           +-dplyr::mutate(...)
  28.                                           \-dplyr:::mutate.data.frame(...)
  29.                                             \-dplyr:::mutate_cols(.data, ...)
  30.                                               \-base::tryCatch(...)
  31.                                                 \-base:::tryCatchList(expr, classes, parentenv, handlers)
  32.                                                   \-base:::tryCatchOne(expr, names, parentenv, handlers[[1L]])
  33.                                                     \-value[[3L]](cond)
  34.                                                       \-dplyr:::stop_dplyr(...)
  35.                                                         \-rlang::abort(...)
  36.                                                           \-rlang:::signal_abort(cnd)
In addition: Warning message:
The `add` argument of `group_by()` is deprecated as of dplyr 1.0.0.
Please use the `.add` argument instead.

[MatthewHeun]

I get a different error.

> generate_fu_allocation_template("CHN")
Error: Can't rename columns that don't exist.
x Column `0` doesn't exist.
Run `rlang::last_error()` to see where the error occurred.
In addition: Warning messages:
1: In max(E.dot) : no non-missing arguments to max; returning -Inf
2: In max(`E.dot [%]`) :
 Error: Can't rename columns that don't exist.
x Column `0` doesn't exist.
Run `rlang::last_error()` to see where the error occurred. 
> rlang::last_error()
<error/vctrs_error_subscript_oob>
Can't rename columns that don't exist.
x Column `0` doesn't exist.
Backtrace:
  1. global::generate_fu_allocation_template("CHN")
 38. vctrs:::stop_subscript_oob(...)
 39. vctrs:::stop_subscript(...)
Run `rlang::last_trace()` to see the full context.
> rlang::last_trace()
<error/vctrs_error_subscript_oob>
Can't rename columns that don't exist.
x Column `0` doesn't exist.
Backtrace:
     █
  1. ├─global::generate_fu_allocation_template("CHN")
  2. │ └─`%>%`(...) R/functions/template_functions.R:27:2
  3. │   ├─base::withVisible(eval(quote(`_fseq`(`_lhs`)), env, env))
  4. │   └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
  5. │     └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
  6. │       └─`_fseq`(`_lhs`)
  7. │         └─magrittr::freduce(value, `_function_list`)
  8. │           └─function_list[[i]](value)
  9. │             └─IEATools::fu_allocation_template(.)
 10. │               └─`%>%`(...) /Users/mkh2/github/IEATools/R/templates.R:153:2
 11. │                 ├─base::withVisible(eval(quote(`_fseq`(`_lhs`)), env, env))
 12. │                 └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
 13. │                   └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
 14. │                     └─IEATools:::`_fseq`(`_lhs`)
 15. │                       └─magrittr::freduce(value, `_function_list`)
 16. │                         └─function_list[[i]](value)
 17. │                           ├─dplyr::rename(...)
 18. │                           └─dplyr:::rename.data.frame(...)
 19. │                             └─tidyselect::eval_rename(expr(c(...)), .data)
 20. │                               └─tidyselect:::rename_impl(...)
 21. │                                 └─tidyselect:::eval_select_impl(...)
 22. │                                   ├─tidyselect:::with_subscript_errors(...)
 23. │                                   │ ├─base::tryCatch(...)
 24. │                                   │ │ └─base:::tryCatchList(expr, classes, parentenv, handlers)
 25. │                                   │ │   └─base:::tryCatchOne(expr, names, parentenv, handlers[[1L]])
 26. │                                   │ │     └─base:::doTryCatch(return(expr), name, parentenv, handler)
 27. │                                   │ └─tidyselect:::instrument_base_errors(expr)
 28. │                                   │   └─base::withCallingHandlers(...)
 29. │                                   └─tidyselect:::vars_select_eval(...)
 30. │                                     └─tidyselect:::walk_data_tree(expr, data_mask, context_mask)
 31. │                                       └─tidyselect:::eval_c(expr, data_mask, context_mask)
 32. │                                         └─tidyselect:::reduce_sels(node, data_mask, context_mask, init = init)
 33. │                                           └─tidyselect:::walk_data_tree(new, data_mask, context_mask)
 34. │                                             └─tidyselect:::as_indices_sel_impl(...)
 35. │                                               └─tidyselect:::as_indices_impl(x, vars, strict = strict)
 36. │                                                 └─tidyselect:::chr_as_locations(x, vars)
 37. │                                                   └─vctrs::vec_as_location(x, n = length(vars), names = vars)
 38. └─vctrs:::stop_subscript_oob(...)
 39.   └─vctrs:::stop_subscript(...)

Since I can usually solve only errors I can reproduce, I'll work on this one first.

[MatthewHeun]

The problem appears to be in the following section of the IEATools code:

    dplyr::rename(
      # Rename the year 0 column
      !!as.name(maximum_values) := !!as.name(year_for_maximum_values), 
      # Rename the product column: it is really a final energy product
      !!as.name(ef_product) := !!as.name(product)
    ) %>% 

In it, I use year "0" as the year for maximum values. But that column appears to be missing.

[MatthewHeun]

Inside generate_fu_template(), there is no "CHN" data. Hmmmm.

When I run the drake workflow, I get

Error: callr subprocess failed: target PSUT_final_49bab2b2 failed.
diagnose(PSUT_final_49bab2b2)error$message:
  Problem with `mutate()` input `matnames`.
x LHS of case 2 (`.data[["Flow"]] %>% starts_with_any_of(c(production, resources))`) must be a logical vector, not a list.

which is the same error that Zeke reported. So now I can replicate the error!

[MatthewHeun]

In IEATools::add_psut_matnames(), seems like IEATools::starts_with_any_of() is returning a list, not a vector. The particular line is at the end of this section of code:

  .tidy_iea_df %>%
    dplyr::mutate(
      "{matnames}" := dplyr::case_when(
        # All Consumption items belong in the final demand (Y) matrix.
        .data[[ledger_side]] == consumption ~ Y,
        # All production items belong in the resources (R) matrix.
        .data[[flow]] %>% starts_with_any_of(c(production, resources)) ~ R,

I note that all tests for IEATools::starts_with_any_of() show that a vector is returned, so this error remains mysterious. I need to do additional diagnostics.

[MatthewHeun]

At the top of add_psut_matnames(), I put the line

print(nrow(.tidy_iea_df))

The answer is 0. So generating the FU Template might be failing, because we have no rows of China data. Still digging!

[MatthewHeun]

"CHN" is not a country in the AllIEAData data frame. We have, instead, "China (P.R. of China and Hong Kong, China)". Thus, when we ask for an analysis of "CHN", we get back no rows. Now to figure out why we don't get "CHN". Getting closer to solving this one!

[MatthewHeun]

This problem occurs because the IEA extended energy balance data have the country "China (P.R. of China and Hong Kong, China)", but the country name database in the countrycode package has separate entries for China CHN and Hong Kong SAR China HKG. What to do?

[MatthewHeun]

For the moment, I have coded "China (P.R. of China and Hong Kong, China)" as "CHN". Another option would be to code it as "CHNHKG".

[MatthewHeun]

And this looks to have solved the problem with China not completing its part of the drake workflow. I can't vouch for the accuracy of the results, only that the drake workflow finishes without error now where it didn't before.

Note that I have also added some code to emit a warning whenever a country results in 0 rows of data. That will be a signal to us to investigate what is happening.

I need to release a new version of IEATools again. When that is released, I will close this issue.

[MatthewHeun]

I released a new version of IEATools that should fix the problem with "CHN". Grab a new version of IEATools with `install_github("MatthewHeun/IEATools").

[MatthewHeun]

I should have also noted that generate_fu_allocation_template("CHN") now works for me.

[ZekeMarshall]

Hi Matt, thanks for looking into this. I have updated IEATools however I am now receiving this error message:

Error: Can't rename columns that don't exist.
x Column `0` doesn't exist.
Run `rlang::last_error()` to see where the error occurred.
In addition: Warning messages:
1: The `add` argument of `group_by()` is deprecated as of dplyr 1.0.0.
Please use the `.add` argument instead.
This warning is displayed once every 8 hours.
Call `lifecycle::last_warnings()` to see where this warning was generated. 
2: In max(E.dot) : no non-missing arguments to max; returning -Inf
3: In max(`E.dot [%]`) :
 Error: Can't rename columns that don't exist.
x Column `0` doesn't exist.
Run `rlang::last_error()` to see where the error occurred. 

As you have successfully generated the template, is this a problem on my end?

Cheers,

Zeke

[MatthewHeun]

Here's what I did:

drake::clean()
Addins|drake|Run a drake workflow
generate_fu_allocation_template("CHN")

Everything works perfectly. You can see the result at Country-level exergy accounting data/CHN/CHN FU Allocations Template.xlsx.

[ZekeMarshall]

Hi Matt, i restarted RStudio this morning and generate_fu_allocation_template("CHN") is now working, thanks!