r-lib / vctrs

Generic programming with typed R vectors
https://vctrs.r-lib.org
Other
287 stars 66 forks source link

Error in `vctrs::vec_locate_matches()`: #1856

Closed abishek07bimali closed 1 year ago

abishek07bimali commented 1 year ago

Match procedure results in an allocation larger than 2^31-1 elements. Attempted allocation size was 3067698393. ℹ In file match.c at line 2644. ℹ This is an internal error that was detected in the vctrs package.

Backtrace: ▆

  1. ├─... %>% as_tibble()
  2. ├─tibble::as_tibble(.)
  3. ├─stats::na.omit(.)
  4. ├─dplyr::select(., shortPostcode, Year, CrimeType, )
  5. ├─dplyr::group_by(., shortPostcode, CrimeType, Year)
  6. ├─dplyr::mutate(., Year = str_trim(substring(Year, 1, 4)))
  7. ├─dplyr::left_join(., LsoaToPostcode, by = "LSOA.code")
  8. ├─dplyr:::left_join.data.frame(., LsoaToPostcode, by = "LSOA.code")
  9. │ └─dplyr:::join_mutate(...)
    1. │ └─dplyr:::join_rows(...)
    2. │ └─dplyr:::dplyr_locate_matches(...)
    3. │ ├─base::withCallingHandlers(...)
    4. │ └─vctrs::vec_locate_matches(...)
    5. └─rlang:::stop_internal_c_lib(...)
    6. └─rlang::abort(message, call = call, .internal = TRUE, .frame = frame)
DavisVaughan commented 1 year ago

your join is attempting to result in over 3 billion rows. you are likely missing a by key variable. if you really are expecting that many rows, there is unfortunately not much we can do because R doesn't have great support for >2 billion rows right now