Open etiennebacher opened 10 months ago
Hi there, thanks for flagging this. I was worried about the same problem when I was writing the benchmarks for the package and thought I was measuring both R + Rust memory allocations, but I will look at this more closely given the example you provide.
In my benchmarks, the memory usage does increase linearly in the size of the input, but the graphs make it look like it uses almost no memory at all as the other package has quadratic memory scaling. The linear scaling helped convince me that the benchmarks were accurate, but I will look into this over the weekend to make sure that the memory usage I see isn't just the data being copied / subsetted in R before it's sent to Rust.
If the memory results aren't accurate, I'll let you know if there I find workaround, or I'll remove them if I can't.
Best, Ben
Indeed, the memory usage increases linearly, thanks for the clarification. I'll leave this issue open if you want to investigate more.
library(zoomerjoin)
library(tidyverse)
library(profmem)
# Sample million rows from DIME dataset
data_1 <- as.data.frame(sample_n(dime_data, 10^6))
names(data_1) <- c("id_1", "name")
data_2 <- as.data.frame(sample_n(dime_data, 10^6))
names(data_2) <- c("id_2", "name")
benches <- bench::press(
n = seq(500, 4000, 250),
bench::mark(
jaccard_inner_join(data_1[1:n, ], data_2[1:n, ],
by = "name", band_width = 11,
n_bands = 350, threshold = .7,
n_gram_width = 4
)
)
)
benches |>
select(n, mem_alloc) |>
ggplot(aes(n, mem_alloc)) +
geom_point() +
geom_line()
Hi, I just found this package, it looks cool and super useful!
One thing that I noted in the benchmarks is how low the memory usage is. I know that using Rust is more efficient in speed and memory usage, but I also think the numbers reported about memory might be inaccurate. From
?profmem
:I suspect that a lot of memory allocations are not done in R but in Rust and that the memory usage is actually higher than reported. I run into the same thing when I benchmark
polars
andtidypolars
, so I'm interested if you find a workaround 😉Just to give you an example: in
polars
, when I take the mean of a column with 100_000, 1_000_000, 10_000_000, or 100_000_000 rows, R reports the same (tiny) memory usage but I clearly see a peak in Windows task manager:I was told there's a linux tool to make more accurate benchmarks when calling other languages from R but I don't remember the name, I'll update this post if I find it.