search

pachadotdev / fixest2

Fixed-effects estimations
https://lrberge.github.io/fixest/
8 stars 3 forks source link

Integers vs doubles matrix breaks fixef() #52

Closed pachadotdev closed 1 year ago

pachadotdev commented 1 year ago

Why fixef() fails?

The issue is that the C++ function cpp_get_fe_gnl_() doesn't like the matrix obsCluster, which is a matrix of integers. If we cast it as integer in R, the R session crashes with a 'memory not mapped' message. If we pass it as a double with no cast, the function does nothing and returns an error of 'double vs integer'.

The problematic functions in my code are

  1. https://github.com/pachadotdev/fixest2/blob/cpp11_wip/R/Methods.R#L882
  2. https://github.com/pachadotdev/fixest2/blob/cpp11_wip/src/05_01_misc_helpers.cpp#L540

R output

I included a line by line example here https://github.com/pachadotdev/fixest2/blob/cpp11_wip/dev/gdb-debug-4.R to generate the error.

Valgrind output

To dig this a bit I used valgrind by following these steps https://github.com/pachadotdev/fixest2/tree/cpp11_wip#debugging.

fixest2 (cpp11_wip) $ debugr dev/gdb-debug-4.R

The result is

fixest2 (cpp11_wip) $ debugr dev/gdb-debug-4.R 
Loading required package: utils
Tracing function "install.packages" in package "utils"
Loading required package: usethis
==25743== Memcheck, a memory error detector
==25743== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==25743== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==25743== Command: /usr/lib/R/bin/exec/R -f dev/gdb-debug-4.R
==25743== 

R version 4.3.1 (2023-06-16) -- "Beagle Scouts"
Copyright (C) 2023 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

Loading required package: utils
Tracing function "install.packages" in package "utils"
Loading required package: usethis
> # library(fixest2)
> devtools::load_all()
ℹ Loading fixest2
> 
> # REPRODUCIBLE ERROR ---
> 
> # gravity_pois = fepois(Euros ~ log(dist_km) | Origin + Destination + Product + Year, trade)
> # fixedEffects = fixef(gravity_pois)
> # Error: Invalid input type, expected 'integer' actual 'double'
> 
> # NOW WE SEE IT LINE BY LINE ----
> 
> # the problem is in fixef.fixest function L882 Methods.R
> 
> # this is the same as to run fixef.fixest line by line
> # lines 924-1079 that do not apply for this case
> 
> gravity_pois = fepois(Euros ~ log(dist_km) | Origin + Destination + Product + Year, trade)
> 
> object <- gravity_pois
> notes <- getFixest_notes()
> sorted <- TRUE
> nthreads <- getFixest_nthreads()
> fixef.tol <- 1e-5
> fixef.iter <- 10000
> S <- object$sumFE
> family <- object$family
> fixef_names <- object$fixef_vars
> fixef_id <- object$fixef_id
> Q <- length(fixef_id)
> N <- length(S)
> id_dummies_vect <- list()
> for (i in 1:Q) id_dummies_vect[[i]] <- as.vector(fixef_id[[i]])
> is_ref_approx <- FALSE
> isSlope <- FALSE
> dumMat <- matrix(unlist(id_dummies_vect), N, Q) - 1
> orderCluster <- matrix(unlist(lapply(id_dummies_vect, order)), N, Q) - 1
> nbCluster <- sapply(fixef_id, max)
> 
> # check the data types
> print(paste("Q", class(Q)))
[1] "Q integer"
> print(paste("N", class(N)))
[1] "N integer"
> print(paste("S", class(S)))
[1] "S numeric"
> print(paste("dumMat", class(dumMat)))
[1] "dumMat matrix" "dumMat array" 
> print(paste("nbCluster", class(nbCluster)))
[1] "nbCluster integer"
> print(paste("orderCluster", class(orderCluster)))
[1] "orderCluster matrix" "orderCluster array" 
> 
> # ERROR 1 ----
> 
> # THIS IS THE LINE THAT BREAKS FIXEF()
> fixef_values <- cpp_get_fe_gnl(Q, N, S, dumMat, nbCluster, orderCluster)
OK L570OK L5954
0
OK L612OK L6321
==25743== Invalid read of size 4
==25743==    at 0x17E21D51: cpp11::writable::r_vector<int>::proxy::operator int() const (integers.hpp:76)
==25743==    by 0x17E1F297: cpp_get_fe_gnl_(int, int, cpp11::writable::r_vector<double>, cpp11::matrix<cpp11::writable::r_vector<int>, cpp11::writable::r_vector<int>::proxy, cpp11::by_column>, cpp11::writable::r_vector<int>, cpp11::matrix<cpp11::writable::r_vector<int>, cpp11::writable::r_vector<int>::proxy, cpp11::by_column>) (05_01_misc_helpers.cpp:613)
==25743==    by 0x17E4B575: _fixest2_cpp_get_fe_gnl_ (cpp11.cpp:292)
==25743==    by 0x4956239: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x495684C: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49AE33F: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49B1377: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49AE0FF: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49AFD85: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49B0BB4: Rf_applyClosure (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49ADE1B: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49B2891: ??? (in /usr/lib/R/lib/libR.so)
==25743==  Address 0x20fad780 is 0 bytes after a block of size 613,248 alloc'd
==25743==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x49F06C4: Rf_allocVector3 (in /usr/lib/R/lib/libR.so)
==25743==    by 0x495E8DB: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x495F32A: Rf_shallow_duplicate (in /usr/lib/R/lib/libR.so)
==25743==    by 0x17E37B87: decltype (((declval<SEXPREC* (*&)(SEXPREC*)>)())((declval<SEXPREC* const&>)())) cpp11::detail::apply<SEXPREC* (*&)(SEXPREC*), SEXPREC* const&, 0ul>(SEXPREC* (*&)(SEXPREC*), std::tuple<SEXPREC* const&>&&, cpp11::detail::index_sequence<0ul> const&) (protect.hpp:183)
==25743==    by 0x17E372DB: decltype (((declval<SEXPREC* (*&)(SEXPREC*)>)())((declval<SEXPREC* const&>)())) cpp11::detail::apply<SEXPREC* (*&)(SEXPREC*), SEXPREC* const&>(SEXPREC* (*&)(SEXPREC*), std::tuple<SEXPREC* const&>&&) (protect.hpp:189)
==25743==    by 0x17E366BF: cpp11::detail::closure<SEXPREC* (SEXPREC*), SEXPREC* const&>::operator()() && (protect.hpp:202)
==25743==    by 0x17E34EC7: cpp11::unwind_protect<cpp11::detail::closure<SEXPREC* (SEXPREC*), SEXPREC* const&>, void>(cpp11::detail::closure<SEXPREC* (SEXPREC*), SEXPREC* const&>&&)::{lambda(void*)#2}::operator()(void*) const (protect.hpp:107)
==25743==    by 0x17E34EEA: cpp11::unwind_protect<cpp11::detail::closure<SEXPREC* (SEXPREC*), SEXPREC* const&>, void>(cpp11::detail::closure<SEXPREC* (SEXPREC*), SEXPREC* const&>&&)::{lambda(void*)#2}::_FUN(void*) (protect.hpp:105)
==25743==    by 0x493D768: R_UnwindProtect (in /usr/lib/R/lib/libR.so)
==25743==    by 0x17E3512D: SEXPREC* cpp11::unwind_protect<cpp11::detail::closure<SEXPREC* (SEXPREC*), SEXPREC* const&>, void>(cpp11::detail::closure<SEXPREC* (SEXPREC*), SEXPREC* const&>&&) (protect.hpp:104)
==25743==    by 0x17E33E4D: decltype (((declval<SEXPREC* (*)(SEXPREC*)>)())((declval<SEXPREC* const&>)())) cpp11::protect::function<SEXPREC* (SEXPREC*)>::operator()<SEXPREC* const&>(SEXPREC* const&) const (protect.hpp:216)
==25743== 

 *** caught segfault ***
address 0x17dd9ea8, cause 'invalid permissions'

Traceback:
 1: .Call(`_fixest2_cpp_get_fe_gnl_`, as.integer(Q), as.integer(N),     sumFE, as.integer(dumMat), as.integer(cluster_sizes), as.integer(obsCluster))
 2: cpp_get_fe_gnl(Q, N, S, dumMat, nbCluster, orderCluster)
An irrecoverable exception occurred. R is aborting now ...
==25743== 
==25743== Process terminating with default action of signal 11 (SIGSEGV)
==25743==    at 0x4D55A7C: __pthread_kill_implementation (pthread_kill.c:44)
==25743==    by 0x4D55A7C: __pthread_kill_internal (pthread_kill.c:78)
==25743==    by 0x4D55A7C: pthread_kill@@GLIBC_2.34 (pthread_kill.c:89)
==25743==    by 0x4D01475: raise (raise.c:26)
==25743==    by 0x4D0151F: ??? (in /usr/lib/x86_64-linux-gnu/libc.so.6)
==25743==    by 0x17E08489: cpp11::writable::r_vector<int>::proxy::operator=(int const&) (integers.hpp:65)
==25743== 
==25743== HEAP SUMMARY:
==25743==     in use at exit: 233,524,814 bytes in 36,657 blocks
==25743==   total heap usage: 244,218 allocs, 207,561 frees, 790,091,076 bytes allocated
==25743== 
==25743== 22 bytes in 1 blocks are possibly lost in loss record 45 of 2,301
==25743==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x4D0403F: __add_to_environ (setenv.c:217)
==25743==    by 0x48543FF: setenv (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x4A805FD: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x4993254: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49ADCEF: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49AFD85: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49B0BB4: Rf_applyClosure (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49ADE1B: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743==    by 0x491F55E: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x4993254: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49ADCEF: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743== 
==25743== 24 bytes in 1 blocks are possibly lost in loss record 66 of 2,301
==25743==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x4DDF14E: tsearch (tsearch.c:337)
==25743==    by 0x4D0406A: __add_to_environ (setenv.c:233)
==25743==    by 0x48543FF: setenv (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x48C994F: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x48C9C33: process_site_Renviron (in /usr/lib/R/lib/libR.so)
==25743==    by 0x4AE939C: Rf_initialize_R (in /usr/lib/R/lib/libR.so)
==25743==    by 0x109099: main (in /usr/lib/R/bin/exec/R)
==25743== 
==25743== 24 bytes in 1 blocks are possibly lost in loss record 67 of 2,301
==25743==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x4DDF14E: tsearch (tsearch.c:337)
==25743==    by 0x4D0406A: __add_to_environ (setenv.c:233)
==25743==    by 0x48543FF: setenv (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x4A805FD: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x4993254: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49ADCEF: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49AFD85: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49B0BB4: Rf_applyClosure (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49ADE1B: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49E1E13: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49E23DB: ??? (in /usr/lib/R/lib/libR.so)
==25743== 
==25743== 24 bytes in 1 blocks are possibly lost in loss record 68 of 2,301
==25743==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x4DDF14E: tsearch (tsearch.c:337)
==25743==    by 0x4D0406A: __add_to_environ (setenv.c:233)
==25743==    by 0x48543FF: setenv (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x4A805FD: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x4993254: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49ADCEF: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49AFD85: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49B0BB4: Rf_applyClosure (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49ADE1B: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743==    by 0x491F55E: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x4993254: ??? (in /usr/lib/R/lib/libR.so)
==25743== 
==25743== 45 bytes in 1 blocks are possibly lost in loss record 96 of 2,301
==25743==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x4D0403F: __add_to_environ (setenv.c:217)
==25743==    by 0x48543FF: setenv (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x48C994F: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x48C9A8E: process_system_Renviron (in /usr/lib/R/lib/libR.so)
==25743==    by 0x4AE8C33: Rf_initialize_R (in /usr/lib/R/lib/libR.so)
==25743==    by 0x109099: main (in /usr/lib/R/bin/exec/R)
==25743== 
==25743== 95 bytes in 1 blocks are possibly lost in loss record 138 of 2,301
==25743==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x4D0403F: __add_to_environ (setenv.c:217)
==25743==    by 0x48543FF: setenv (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x48C994F: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x48C9C33: process_site_Renviron (in /usr/lib/R/lib/libR.so)
==25743==    by 0x4AE939C: Rf_initialize_R (in /usr/lib/R/lib/libR.so)
==25743==    by 0x109099: main (in /usr/lib/R/bin/exec/R)
==25743== 
==25743== 96 bytes in 4 blocks are possibly lost in loss record 145 of 2,301
==25743==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x4DDF14E: tsearch (tsearch.c:337)
==25743==    by 0x4D0406A: __add_to_environ (setenv.c:233)
==25743==    by 0x48543FF: setenv (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x48C994F: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x48C9A8E: process_system_Renviron (in /usr/lib/R/lib/libR.so)
==25743==    by 0x4AE8C33: Rf_initialize_R (in /usr/lib/R/lib/libR.so)
==25743==    by 0x109099: main (in /usr/lib/R/bin/exec/R)
==25743== 
==25743== 168 bytes in 7 blocks are possibly lost in loss record 171 of 2,301
==25743==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x4DDF14E: tsearch (tsearch.c:337)
==25743==    by 0x4D0406A: __add_to_environ (setenv.c:233)
==25743==    by 0x48543FF: setenv (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x48C994F: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x4AE93A1: Rf_initialize_R (in /usr/lib/R/lib/libR.so)
==25743==    by 0x109099: main (in /usr/lib/R/bin/exec/R)
==25743== 
==25743== 336 bytes in 1 blocks are possibly lost in loss record 277 of 2,301
==25743==    at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x40147D9: calloc (rtld-malloc.h:44)
==25743==    by 0x40147D9: allocate_dtv (dl-tls.c:375)
==25743==    by 0x40147D9: _dl_allocate_tls (dl-tls.c:634)
==25743==    by 0x4D54834: allocate_stack (allocatestack.c:430)
==25743==    by 0x4D54834: pthread_create@@GLIBC_2.34 (pthread_create.c:647)
==25743==    by 0x1194668E: ??? (in /usr/lib/R/site-library/cli/libs/cli.so)
==25743==    by 0x119466D7: ??? (in /usr/lib/R/site-library/cli/libs/cli.so)
==25743==    by 0x495629D: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x4999262: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49ADCEF: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49AFD85: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49B0BB4: Rf_applyClosure (in /usr/lib/R/lib/libR.so)
==25743==    by 0x499ECEC: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49ADCEF: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743== 
==25743== 672 bytes in 2 blocks are possibly lost in loss record 340 of 2,301
==25743==    at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x40147D9: calloc (rtld-malloc.h:44)
==25743==    by 0x40147D9: allocate_dtv (dl-tls.c:375)
==25743==    by 0x40147D9: _dl_allocate_tls (dl-tls.c:634)
==25743==    by 0x4D54834: allocate_stack (allocatestack.c:430)
==25743==    by 0x4D54834: pthread_create@@GLIBC_2.34 (pthread_create.c:647)
==25743==    by 0x57331EF: ??? (in /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0)
==25743==    by 0x5729A10: GOMP_parallel (in /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0)
==25743==    by 0x17E15C52: cpp_sparse_products_(cpp11::matrix<cpp11::r_vector<double>, double, cpp11::by_column>, cpp11::r_vector<double>, SEXPREC*, bool, int) (04_03_linear_model_mmult.cpp:207)
==25743==    by 0x17E485FA: _fixest2_cpp_sparse_products_ (cpp11.cpp:166)
==25743==    by 0x495625D: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x495684C: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49AE33F: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49B1377: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49AE0FF: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743== 
==25743== 736 bytes in 37 blocks are possibly lost in loss record 343 of 2,301
==25743==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x4D0403F: __add_to_environ (setenv.c:217)
==25743==    by 0x48543FF: setenv (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x4A805FD: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x4993254: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49ADCEF: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49AFD85: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49B0BB4: Rf_applyClosure (in /usr/lib/R/lib/libR.so)
==25743==    by 0x499ECEC: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49ADCEF: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49AFD85: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49B0BB4: Rf_applyClosure (in /usr/lib/R/lib/libR.so)
==25743== 
==25743== 888 bytes in 37 blocks are possibly lost in loss record 364 of 2,301
==25743==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x4DDF14E: tsearch (tsearch.c:337)
==25743==    by 0x4D0406A: __add_to_environ (setenv.c:233)
==25743==    by 0x48543FF: setenv (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x4A805FD: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x4993254: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49ADCEF: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49AFD85: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49B0BB4: Rf_applyClosure (in /usr/lib/R/lib/libR.so)
==25743==    by 0x499ECEC: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49ADCEF: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49AFD85: ??? (in /usr/lib/R/lib/libR.so)
==25743== 
==25743== 1,008 bytes in 3 blocks are possibly lost in loss record 370 of 2,301
==25743==    at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x40147D9: calloc (rtld-malloc.h:44)
==25743==    by 0x40147D9: allocate_dtv (dl-tls.c:375)
==25743==    by 0x40147D9: _dl_allocate_tls (dl-tls.c:634)
==25743==    by 0x4D54834: allocate_stack (allocatestack.c:430)
==25743==    by 0x4D54834: pthread_create@@GLIBC_2.34 (pthread_create.c:647)
==25743==    by 0x57331EF: ??? (in /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0)
==25743==    by 0x5729A10: GOMP_parallel (in /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0)
==25743==    by 0x17E395DB: cpppar_which_na_inf_mat_(cpp11::matrix<cpp11::r_vector<double>, double, cpp11::by_column>, int) (07_01_parallel_helpers.cpp:112)
==25743==    by 0x17E4E665: _fixest2_cpppar_which_na_inf_mat_ (cpp11.cpp:418)
==25743==    by 0x49562B9: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x495684C: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49AE33F: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49B1377: ??? (in /usr/lib/R/lib/libR.so)
==25743==    by 0x49AE0FF: Rf_eval (in /usr/lib/R/lib/libR.so)
==25743== 
==25743== 2,304 bytes in 1 blocks are possibly lost in loss record 443 of 2,301
==25743==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25743==    by 0x4005D97: malloc (rtld-malloc.h:56)
==25743==    by 0x4005D97: _dlfo_mappings_segment_allocate (dl-find_object.c:217)
==25743==    by 0x4005D97: _dl_find_object_update_1 (dl-find_object.c:671)
==25743==    by 0x4005D97: _dl_find_object_update (dl-find_object.c:804)
==25743==    by 0x400ECCF: dl_open_worker_begin (dl-open.c:735)
==25743==    by 0x4E33C27: _dl_catch_exception (dl-error-skeleton.c:208)
==25743==    by 0x400DF99: dl_open_worker (dl-open.c:782)
==25743==    by 0x4E33C27: _dl_catch_exception (dl-error-skeleton.c:208)
==25743==    by 0x400E34D: _dl_open (dl-open.c:883)
==25743==    by 0x4E33F90: do_dlopen (dl-libc.c:95)
==25743==    by 0x4E33C27: _dl_catch_exception (dl-error-skeleton.c:208)
==25743==    by 0x4E33CF2: _dl_catch_error (dl-error-skeleton.c:227)
==25743==    by 0x4E340C6: dlerror_run (dl-libc.c:45)
==25743==    by 0x4E340C6: __libc_dlopen_mode (dl-libc.c:162)
==25743==    by 0x4CF4EB2: __gconv_find_shlib (gconv_dl.c:114)
==25743== 
==25743== LEAK SUMMARY:
==25743==    definitely lost: 0 bytes in 0 blocks
==25743==    indirectly lost: 0 bytes in 0 blocks
==25743==      possibly lost: 6,442 bytes in 98 blocks
==25743==    still reachable: 233,518,372 bytes in 36,559 blocks
==25743==                       of which reachable via heuristic:
==25743==                         newarray           : 4,264 bytes in 1 blocks
==25743==         suppressed: 0 bytes in 0 blocks
==25743== Reachable blocks (those to which a pointer was found) are not shown.
==25743== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==25743== 
==25743== For lists of detected and suppressed errors, rerun with: -s
==25743== ERROR SUMMARY: 21 errors from 15 contexts (suppressed: 0 from 0)
Segmentation fault (core dumped)
pachadotdev commented 1 year ago

@s3alfisc this is the kind of issue that I find when writing C++ Over the last year I have made decent success porting fixest to a more "neutral" and "modern" C++. It would be great to have the same engine for both Python and R, like Arrow does. Will you be interested in making it happen? pybind11 seems to be the same as cpp11 (R package), so the integration is not so bad.

also, I made progress at identifying my issues because @lrberge also answered one of my questions on SO

s3alfisc commented 1 year ago

Hi @pachadotdev, of course, I would be more than happy to!

pachadotdev commented 1 year ago

great! I'll add u to this repository then is it ok to email you later?

pachadotdev commented 1 year ago

@s3alfisc I updated the bug description

this is some progress, because I have moved the C++ to a more "unsweetened" C++ style, which we can then use with Python