Subscript out of bounds error while deduping dataset

[wbakerrobinson]

Hello I am getting a subscript out of bounds error while running fastLink for de-duplication. I have updated and reinstalled my packages and this did not fix my issues. Below is a reproducible example of the error that I am encountering.

Error: task 1 failed - "error in evaluating the argument 'x' in selecting a method for function 'which': subscript out of bounds"

Traceback: 6: stop(simpleError(msg, call = expr)) 5: e$fun(obj, substitute(ex), parent.frame(), e$data) 4: foreach(i = 1:length(matches.2)) %oper% { ht1 <- which(matrix.1 == matches.2[[i]][[1]]) ht2 <- which(matrix.2 == matches.2[[i]][[2]]) list(ht1, ht2) } 3: gammaNUMCK2par(dfA[, varnames[i]], dfB[, varnames[i]], cut.a = cut.a.num, n.cores = n.cores) 2: fastLink(df, df, varnames = c("Sex", "Age", "zip_code", "Hours_Since"), numeric.match = c("Hours_Since"), cut.a.num = 0.01, cut.p.num = 1.5, n.cores = 1) at #2 1: dedupe_fx(synthetic_df)

Reproducible example:

# Load required package
library(fastLink)

# Create a synthetic data frame
set.seed(123)
synthetic_df <- data.frame(
  Sex = sample(c("Male", "Female"), 100, replace = TRUE),
  Age = sample(18:90, 100, replace = TRUE),
  zip_code = sample(sprintf("%05d", sample(10000:99999, 100, replace = TRUE)), 100, replace = TRUE),
  Hours_Since = sample(1:100, 100, replace = TRUE)
)

# Make hours_since numeric
synthetic_df[["Hours_Since"]] <- as.numeric(synthetic_df[["Hours_Since"]])

# Add duplicate rows
synthetic_df <- rbind(synthetic_df, synthetic_df[1:3,])

# Display the structure of the synthetic data frame
str(synthetic_df)

# Function to dedupe using fastLink
dedupe_fx <- function(df) {
  result <- fastLink(df, df,
                     varnames = c("Sex", "Age", "zip_code", "Hours_Since"),
                     numeric.match = c("Hours_Since"),
                     cut.a.num = 0.01,
                     cut.p.num = 1.5,
                     n.cores = 1) # Set n.cores to 1 for simplicity
  return(result)
}

# Run the deduplication function on the synthetic data
dedupe_result <- dedupe_fx(synthetic_df)

Notes: fastLink is wrapped in a function because I want to iterate through a split dataframe in order to dedupe using blocking.

Session Information: R version 4.4.0 (2024-04-24 ucrt) Platform: x86_64-w64-mingw32/x64 Running under: Windows 10 x64 (build 19045)

Matrix products: default

locale: [1] LC_COLLATE=English_United States.utf8 [2] LC_CTYPE=English_United States.utf8
[3] LC_MONETARY=English_United States.utf8 [4] LC_NUMERIC=C
[5] LC_TIME=English_United States.utf8

time zone: America/Los_Angeles tzcode source: internal

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] Matrix_1.7-0 Rcpp_1.0.12 fastLink_0.6.1

loaded via a namespace (and not attached): [1] vctrs_0.6.5 doParallel_1.0.17 cli_3.6.2 rlang_1.1.4
[5] stringi_1.8.4 generics_0.1.3 data.table_1.15.4 gtools_3.9.5
[9] glue_1.7.0 adagio_0.9.2 fansi_1.0.6 lpSolve_5.6.20
[13] grid_4.4.0 tibble_3.2.1 foreach_1.5.2 lifecycle_1.0.4
[17] stringr_1.5.1 compiler_4.4.0 dplyr_1.1.4 codetools_0.2-20 [21] pkgconfig_2.0.3 rstudioapi_0.16.0 lattice_0.22-6 R6_2.5.1
[25] tidyselect_1.2.1 utf8_1.2.4 pillar_1.9.0 parallel_4.4.0
[29] stringdist_0.9.12 magrittr_2.0.3 tools_4.4.0 plotrix_3.8-4
[33] iterators_1.0.14

[tedenamorado]

Hi,

What about trying this instead:

dedupe_fx <- function(df) {
  result <- fastLink(df, df,
                     varnames = c("Sex", "Age", "zip_code", "Hours_Since"),
                     numeric.match = c("Hours_Since"),
                     partial.match = c("Hours_Since"),
                     cut.a.num = 0.01,
                     cut.p.num = 1.5,
                     n.cores = 1) # Set n.cores to 1 for simplicity
  return(result)
}

As you can see I just added one line. Note that in fastLink, you need to specify which variables will be compared using three agreement levels. If not, specifying different two cutpoints does not work as intended.

Hope this helps!

Ted