rgiordan / zaminfluence

Tools in R for computing and using Z-estimator approximate influence functions.
Apache License 2.0
94 stars 10 forks source link

testthat.R fails #6

Closed jmcastagnetto closed 3 years ago

jmcastagnetto commented 3 years ago

OS: Ubuntu 20.04.1 LTS R: 4.0.3 Python: 3.8.5

Everything installed as per the README.md

$ ./testthat.R 
Loading required package: reticulate
Loading required package: dplyr

Attaching package: ‘dplyr’

The following object is masked from ‘package:testthat’:

    matches

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

Loading required package: ggplot2
Loading required package: latex2exp
Loading required package: stringr
Loading required package: broom
── Skip (test_end_to_end.R:16:1): python runs ────────────────────────────────────────────────────────────────────────────────────────────────────
Reason: empty test

── Error (test_end_to_end.R:72:3): regression works ──────────────────────────────────────────────────────────────────────────────────────────────
Error: ValueError: The estimating equation is not zero at the proposed  values.  ||ee|| = 602841342157458.6 > 1e-08 = solution_tol

Detailed traceback: 
  File "<string>", line 3, in <module>
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/regsens_rgiordandev-0.1-py3.8.egg/regsens_rgiordandev/regression_sensitivity_lib.py", line 143, in get_regression_w_grads
    obs_w_sens = vittles.HyperparameterSensitivityLinearApproximation(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 365, in __init__
    EstimatingEquationLinearApproximation.__init__(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 186, in __init__
    self.set_location(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 215, in set_location
    raise ValueError(err_msg)

Backtrace:
    █
 1. └─zaminfluence:::TestRegressionConfiguration(...) test_end_to_end.R:72:2
 2.   └─zaminfluence:::TestConfiguration(lm_result, se_group = df[["se_group"]]) test_end_to_end.R:69:4
 3.     └─zaminfluence::ComputeModelInfluence(model_fit, se_group) test_end_to_end.R:32:4
 4.       └─zaminfluence::ComputeRegressionInfluence(model_fit, se_group)
 5.         └─reticulate::py_run_string("\nbetahat = regsens_rgiordandev.reg(y, x, w=w0)\nse, betahat_grad, se_grad = regsens_rgiordandev.get_regression_w_grads(\n    betahat, y, x, w0, se_group=se_group)\n")
 6.           └─reticulate:::py_run_string_impl(code, local, convert)

── Error (test_end_to_end.R:94:3): plotting works ────────────────────────────────────────────────────────────────────────────────────────────────
Error: ValueError: The estimating equation is not zero at the proposed  values.  ||ee|| = 1.3508418382381986e+16 > 1e-08 = solution_tol

Detailed traceback: 
  File "<string>", line 3, in <module>
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/regsens_rgiordandev-0.1-py3.8.egg/regsens_rgiordandev/regression_sensitivity_lib.py", line 143, in get_regression_w_grads
    obs_w_sens = vittles.HyperparameterSensitivityLinearApproximation(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 365, in __init__
    EstimatingEquationLinearApproximation.__init__(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 186, in __init__
    self.set_location(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 215, in set_location
    raise ValueError(err_msg)

Backtrace:
    █
 1. ├─`%>%`(...) test_end_to_end.R:94:2
 2. ├─zaminfluence::SortAndAccumulate(.)
 3. ├─zaminfluence::GetTargetRegressorGrads(., "x1")
 4. │ └─base::which(reg_infl$regressor_names == target_regressor)
 5. └─zaminfluence::ComputeRegressionInfluence(lm_result)
 6.   └─reticulate::py_run_string("\nbetahat = regsens_rgiordandev.reg(y, x, w=w0)\nse, betahat_grad, se_grad = regsens_rgiordandev.get_regression_w_grads(\n    betahat, y, x, w0, se_group=se_group)\n")
 7.     └─reticulate:::py_run_string_impl(code, local, convert)

── Error (test_end_to_end.R:178:3): rerun works ──────────────────────────────────────────────────────────────────────────────────────────────────
Error: ValueError: The estimating equation is not zero at the proposed  values.  ||ee|| = 2225790403806.557 > 1e-08 = solution_tol

Detailed traceback: 
  File "<string>", line 3, in <module>
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/regsens_rgiordandev-0.1-py3.8.egg/regsens_rgiordandev/regression_sensitivity_lib.py", line 143, in get_regression_w_grads
    obs_w_sens = vittles.HyperparameterSensitivityLinearApproximation(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 365, in __init__
    EstimatingEquationLinearApproximation.__init__(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 186, in __init__
    self.set_location(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 215, in set_location
    raise ValueError(err_msg)

Backtrace:
    █
 1. ├─zaminfluence:::test_rerun(lm_result, paired = FALSE) test_end_to_end.R:178:2
 2. │ ├─`%>%`(...) test_end_to_end.R:123:6
 3. │ └─zaminfluence::ComputeModelInfluence(model_fit)
 4. │   └─zaminfluence::ComputeRegressionInfluence(model_fit, se_group)
 5. │     └─reticulate::py_run_string("\nbetahat = regsens_rgiordandev.reg(y, x, w=w0)\nse, betahat_grad, se_grad = regsens_rgiordandev.get_regression_w_grads(\n    betahat, y, x, w0, se_group=se_group)\n")
 6. │       └─reticulate:::py_run_string_impl(code, local, convert)
 7. ├─zaminfluence::SortAndAccumulate(.)
 8. └─zaminfluence::GetTargetRegressorGrads(., "x1")
 9.   └─base::which(reg_infl$regressor_names == target_regressor)

── Error (test_end_to_end.R:202:3): pairing works ────────────────────────────────────────────────────────────────────────────────────────────────
Error: ValueError: The estimating equation is not zero at the proposed  values.  ||ee|| = 186209921265830.16 > 1e-08 = solution_tol

Detailed traceback: 
  File "<string>", line 3, in <module>
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/regsens_rgiordandev-0.1-py3.8.egg/regsens_rgiordandev/regression_sensitivity_lib.py", line 143, in get_regression_w_grads
    obs_w_sens = vittles.HyperparameterSensitivityLinearApproximation(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 365, in __init__
    EstimatingEquationLinearApproximation.__init__(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 186, in __init__
    self.set_location(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 215, in set_location
    raise ValueError(err_msg)

Backtrace:
    █
 1. ├─ComputeRegressionInfluence(lm_result) %>% GetTargetRegressorGrads("x1") test_end_to_end.R:202:2
 2. ├─zaminfluence::GetTargetRegressorGrads(., "x1")
 3. │ └─base::which(reg_infl$regressor_names == target_regressor)
 4. └─zaminfluence::ComputeRegressionInfluence(lm_result)
 5.   └─reticulate::py_run_string("\nbetahat = regsens_rgiordandev.reg(y, x, w=w0)\nse, betahat_grad, se_grad = regsens_rgiordandev.get_regression_w_grads(\n    betahat, y, x, w0, se_group=se_group)\n")
 6.     └─reticulate:::py_run_string_impl(code, local, convert)

── Error (test_end_to_end.R:241:3): aggregated pairing works ─────────────────────────────────────────────────────────────────────────────────────
Error: ValueError: The estimating equation is not zero at the proposed  values.  ||ee|| = 186209921265830.16 > 1e-08 = solution_tol

Detailed traceback: 
  File "<string>", line 3, in <module>
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/regsens_rgiordandev-0.1-py3.8.egg/regsens_rgiordandev/regression_sensitivity_lib.py", line 143, in get_regression_w_grads
    obs_w_sens = vittles.HyperparameterSensitivityLinearApproximation(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 365, in __init__
    EstimatingEquationLinearApproximation.__init__(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 186, in __init__
    self.set_location(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 215, in set_location
    raise ValueError(err_msg)

Backtrace:
    █
 1. ├─ComputeRegressionInfluence(lm_result) %>% GetTargetRegressorGrads("x1") test_end_to_end.R:241:2
 2. ├─zaminfluence::GetTargetRegressorGrads(., "x1")
 3. │ └─base::which(reg_infl$regressor_names == target_regressor)
 4. └─zaminfluence::ComputeRegressionInfluence(lm_result)
 5.   └─reticulate::py_run_string("\nbetahat = regsens_rgiordandev.reg(y, x, w=w0)\nse, betahat_grad, se_grad = regsens_rgiordandev.get_regression_w_grads(\n    betahat, y, x, w0, se_group=se_group)\n")
 6.     └─reticulate:::py_run_string_impl(code, local, convert)

── Error (test_end_to_end.R:294:3): regression moment conditions works ───────────────────────────────────────────────────────────────────────────
Error: ValueError: The estimating equation is not zero at the proposed  values.  ||ee|| = 114081245724969.67 > 1e-08 = solution_tol

Detailed traceback: 
  File "<string>", line 3, in <module>
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/regsens_rgiordandev-0.1-py3.8.egg/regsens_rgiordandev/regression_sensitivity_lib.py", line 172, in get_regression_offset_grads
    offset_sens = vittles.HyperparameterSensitivityLinearApproximation(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 365, in __init__
    EstimatingEquationLinearApproximation.__init__(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 186, in __init__
    self.set_location(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 215, in set_location
    raise ValueError(err_msg)

Backtrace:
    █
 1. └─zaminfluence::ComputeRegressionMomentSensitivity(lm_result) test_end_to_end.R:294:2
 2.   └─reticulate::py_run_string("\nbetahat = regsens_rgiordandev.reg(y, x, w=w0)\nse, betahat_grad, se_grad = regsens_rgiordandev.get_regression_offset_grads(\n    betahat, y, x, offset0, se_group=se_group)\n")
 3.     └─reticulate:::py_run_string_impl(code, local, convert)

── Skipped tests  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
● empty test (1)

══ testthat results  ═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
Error (test_end_to_end.R:72:3): regression works
Error (test_end_to_end.R:94:3): plotting works
Error (test_end_to_end.R:178:3): rerun works
Error (test_end_to_end.R:202:3): pairing works
Error (test_end_to_end.R:241:3): aggregated pairing works
Error (test_end_to_end.R:294:3): regression moment conditions works

[ FAIL 6 | WARN 0 | SKIP 1 | PASS 0 ]
Error: Test failures
Execution halted

Also, when testing the Python bits, they seem to work, but give a deprecation warning:

$ python -m pytest
============================================================== test session starts ===============================================================
platform linux -- Python 3.8.5, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/jesus/devel/others/zaminfluence/regression_sensitivity
collected 2 items                                                                                                                                

regsens_rgiordandev/tests/test_basics.py ..                                                                                                [100%]

================================================================ warnings summary ================================================================
../venv/lib/python3.8/site-packages/autograd-1.3-py3.8.egg/autograd/numpy/numpy_wrapper.py:20
../venv/lib/python3.8/site-packages/autograd-1.3-py3.8.egg/autograd/numpy/numpy_wrapper.py:20
../venv/lib/python3.8/site-packages/autograd-1.3-py3.8.egg/autograd/numpy/numpy_wrapper.py:20
../venv/lib/python3.8/site-packages/autograd-1.3-py3.8.egg/autograd/numpy/numpy_wrapper.py:20
  /home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/autograd-1.3-py3.8.egg/autograd/numpy/numpy_wrapper.py:20: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. Use `int` by itself, which is identical in behavior, to silence this warning. If you specifically wanted the numpy scalar type, use `np.int_` here.
    int_types = {_np.int, _np.int8, _np.int16, _np.int32, _np.int64, _np.integer}

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================= 2 passed, 4 warnings in 0.87s ==========================================================

Finally, editing examples/simple_examples.R to get the correct env var, and running it gives a similar not zero error as the test scripts:

$ Rscript simple_examples.R 

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

Attaching package: ‘gridExtra’

The following object is masked from ‘package:dplyr’:

    combine

Loading required package: reticulate
Loading required package: latex2exp
Loading required package: stringr
Loading required package: broom
Loading required package: car
Loading required package: carData

Attaching package: ‘car’

The following object is masked from ‘package:dplyr’:

    recode

Loading required package: lmtest
Loading required package: zoo

Attaching package: ‘zoo’

The following objects are masked from ‘package:base’:

    as.Date, as.Date.numeric

Loading required package: survival
Error in py_run_string_impl(code, local, convert) : 
  ValueError: The estimating equation is not zero at the proposed  values.  ||ee|| = 178.39173099675423 > 1e-08 = solution_tol

Detailed traceback: 
  File "<string>", line 3, in <module>
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/regsens_rgiordandev-0.1-py3.8.egg/regsens_rgiordandev/regression_sensitivity_lib.py", line 143, in get_regression_w_grads
    obs_w_sens = vittles.HyperparameterSensitivityLinearApproximation(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 365, in __init__
    EstimatingEquationLinearApproximation.__init__(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 186, in __init__
    self.set_location(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 215, in set_
Calls: ComputeModelInfluence ... ComputeRegressionInfluence -> <Anonymous> -> py_run_string_impl
Execution halted
rgiordan commented 3 years ago

Just getting to this now, thank you for your patience! This is a bit strange, but, then, that's why we have unit tests.

Could you please run this script and tell me if you get zeros printed at the end?

# Set this to your own zaminfluence directory.
base_dir  <- "/home/rgiordan/Documents/git_repos/zaminfluence"
setwd(base_dir)

library(devtools)
library(tidyverse)
library(zaminfluence)

devtools::load_all("zaminfluence")  # Load even non-exported zaminfluence functions.
py_main <- InitializePython(file.path(base_dir, "venv/bin/python"))

# Test utility for generating data
source(file.path(base_dir, "zaminfluence/tests/testthat/utils.R"))

set.seed(42)

# Generate data and a regression model.
x_dim <- 3
beta_true <- runif(x_dim)
df <- GenerateRegressionData(10, beta_true, num_groups=NULL)

# Fit a regression model.
x_names <- sprintf("x%d", 1:x_dim)
reg_form <- formula(sprintf("y ~ %s - 1", paste(x_names, collapse=" + ")))
reg_fit <- lm(data = df, formula = reg_form, x=TRUE, y=TRUE)

py_main_set <- SetPythonRegressionVariables(reg_fit)

# If we are setting the python variables correctly, these differences should all be zero.
CheckPyMain <- function(py_main_check) {
    print(py_main_check$x - df[, x_names] %>% as.matrix())
    print(py_main_check$y - df$y %>% as.numeric())
    print(py_main_check$beta - reg_fit$coefficients)
    print(py_main_check$w0 - rep(1, nrow(df)))
}

CheckPyMain(py_main_set)
CheckPyMain(py_main)

For example, when I run this on a working installation, I get

> CheckPyMain(py_main_set)
      x1 x2 x3
 [1,]  0  0  0
 [2,]  0  0  0
 [3,]  0  0  0
 [4,]  0  0  0
 [5,]  0  0  0
 [6,]  0  0  0
 [7,]  0  0  0
 [8,]  0  0  0
 [9,]  0  0  0
[10,]  0  0  0
 [1] 0 0 0 0 0 0 0 0 0 0
[1] 0 0 0
 [1] 0 0 0 0 0 0 0 0 0 0
> CheckPyMain(py_main)
      x1 x2 x3
 [1,]  0  0  0
 [2,]  0  0  0
 [3,]  0  0  0
 [4,]  0  0  0
 [5,]  0  0  0
 [6,]  0  0  0
 [7,]  0  0  0
 [8,]  0  0  0
 [9,]  0  0  0
[10,]  0  0  0
 [1] 0 0 0 0 0 0 0 0 0 0
[1] 0 0 0
 [1] 0 0 0 0 0 0 0 0 0 0
rgiordan commented 3 years ago

For some context, the error you're getting is because the Python part (which computes the derivatives) is being asked to evaluate the sensitivity at values which are not the OLS solution. The most likely culprit is that the values are not being copied correctly from Python to R, so that's what we're checking here.

jmcastagnetto commented 3 years ago

Sorry I just had time to run your script, and it works w/o a problem:

$ Rscript  test.R 
Loading required package: usethis
── Attaching packages ───────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse 1.3.0 ──
✔ ggplot2 3.3.2     ✔ purrr   0.3.4
✔ tibble  3.0.4     ✔ dplyr   1.0.2
✔ tidyr   1.1.2     ✔ stringr 1.4.0
✔ readr   1.4.0     ✔ forcats 0.5.0
── Conflicts ──────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
Loading required package: reticulate
Loading required package: latex2exp
Loading required package: broom
Loading zaminfluence

Attaching package: ‘testthat’

The following object is masked from ‘package:dplyr’:

    matches

The following object is masked from ‘package:purrr’:

    is_null

The following object is masked from ‘package:tidyr’:

    matches

The following object is masked from ‘package:devtools’:

    test_file

Attaching package: ‘gridExtra’

The following object is masked from ‘package:dplyr’:

    combine

      x1 x2 x3
 [1,]  0  0  0
 [2,]  0  0  0
 [3,]  0  0  0
 [4,]  0  0  0
 [5,]  0  0  0
 [6,]  0  0  0
 [7,]  0  0  0
 [8,]  0  0  0
 [9,]  0  0  0
[10,]  0  0  0
 [1] 0 0 0 0 0 0 0 0 0 0
[1] 0 0 0
 [1] 0 0 0 0 0 0 0 0 0 0
      x1 x2 x3
 [1,]  0  0  0
 [2,]  0  0  0
 [3,]  0  0  0
 [4,]  0  0  0
 [5,]  0  0  0
 [6,]  0  0  0
 [7,]  0  0  0
 [8,]  0  0  0
 [9,]  0  0  0
[10,]  0  0  0
 [1] 0 0 0 0 0 0 0 0 0 0
[1] 0 0 0
 [1] 0 0 0 0 0 0 0 0 0 0

Also, tried doing a git pull origin, and see that there are updates in the README.md and zaminfluence/DESCRIPTION files.

Just in case, did a reinstall of both the python and the R parts, and got the same errors as before.

rgiordan commented 3 years ago

Thanks! Next, could I ask you to append the following code to the end of the previous debugging script and share what you get? Here, I'm just running the constituent parts of ComputeRegressionInfluence, leading up to the function itself, to see what is going wrong. If this works, then I guess the problem is in the unit testing setup.


# Manually run the Python code in ComputeRegressionInfluence
reticulate::py_run_string("
reg_obj = regsens_rgiordandev.reg_obj
get_standard_error_matrix = regsens_rgiordandev.get_standard_error_matrix
sens_reg_obj = lambda beta, w: reg_obj(beta, y=y, x=x, w=w)
obs_w_sens = vittles.HyperparameterSensitivityLinearApproximation(
    objective_fun=sens_reg_obj,
    opt_par_value=beta,
    hyper_par_value=w0,
    validate_optimum=True,
    grad_tol=1e-08)

get_betahat = obs_w_sens.get_opt_par_function()

def get_se(w):
    betahat = get_betahat(w)
    se_cov = get_standard_error_matrix(betahat, y, x, w=w, se_group=se_group)
    return np.sqrt(np.diag(se_cov))

se = get_se(w0)
betahat_grad = obs_w_sens.get_dopt_dhyper()
se_grad = autograd.jacobian(get_se)(w0)
")

print(py_main$betahat_grad)
print(py_main$se_grad)

# Manually run the R code in ComputeRegressionInfluence
py_main <- SetPythonRegressionVariables(reg_fit, se_group=NULL)
reg <- broom::tidy(reg_fit)
reticulate::py_run_string("
betahat = regsens_rgiordandev.reg(y, x, w=w0)
se, betahat_grad, se_grad = regsens_rgiordandev.get_regression_w_grads(
    betahat, y, x, w0, se_group=se_group)
")

print(py_main$betahat_grad)
print(py_main$se_grad)

# Just run ComputeRegressionInfluence
ComputeRegressionInfluence(reg_fit, se_group=NULL)
jmcastagnetto commented 3 years ago

OK, done, the final script is

# Set this to your own zaminfluence directory.
base_dir  <- "/home/jesus/devel/others/zaminfluence"

setwd(base_dir)

library(devtools)
library(tidyverse)
library(zaminfluence)

devtools::load_all("zaminfluence")  # Load even non-exported zaminfluence functions.
py_main <- InitializePython(file.path(base_dir, "venv/bin/python"))

# Test utility for generating data
source(file.path(base_dir, "zaminfluence/tests/testthat/utils.R"))

set.seed(42)

# Generate data and a regression model.
x_dim <- 3
beta_true <- runif(x_dim)
df <- GenerateRegressionData(10, beta_true, num_groups=NULL)

# Fit a regression model.
x_names <- sprintf("x%d", 1:x_dim)
reg_form <- formula(sprintf("y ~ %s - 1", paste(x_names, collapse=" + ")))
reg_fit <- lm(data = df, formula = reg_form, x=TRUE, y=TRUE)

py_main_set <- SetPythonRegressionVariables(reg_fit)

# If we are setting the python variables correctly, these differences should all be zero.
CheckPyMain <- function(py_main_check) {
    print(py_main_check$x - df[, x_names] %>% as.matrix())
    print(py_main_check$y - df$y %>% as.numeric())
    print(py_main_check$beta - reg_fit$coefficients)
    print(py_main_check$w0 - rep(1, nrow(df)))
}

CheckPyMain(py_main_set)
CheckPyMain(py_main)

# new code below

# Manually run the Python code in ComputeRegressionInfluence
reticulate::py_run_string("
reg_obj = regsens_rgiordandev.reg_obj
get_standard_error_matrix = regsens_rgiordandev.get_standard_error_matrix
sens_reg_obj = lambda beta, w: reg_obj(beta, y=y, x=x, w=w)
obs_w_sens = vittles.HyperparameterSensitivityLinearApproximation(
    objective_fun=sens_reg_obj,
    opt_par_value=beta,
    hyper_par_value=w0,
    validate_optimum=True,
    grad_tol=1e-08)

get_betahat = obs_w_sens.get_opt_par_function()

def get_se(w):
    betahat = get_betahat(w)
    se_cov = get_standard_error_matrix(betahat, y, x, w=w, se_group=se_group)
    return np.sqrt(np.diag(se_cov))

se = get_se(w0)
betahat_grad = obs_w_sens.get_dopt_dhyper()
se_grad = autograd.jacobian(get_se)(w0)
")

print(py_main$betahat_grad)
print(py_main$se_grad)

# Manually run the R code in ComputeRegressionInfluence
py_main <- SetPythonRegressionVariables(reg_fit, se_group=NULL)
reg <- broom::tidy(reg_fit)
reticulate::py_run_string("
betahat = regsens_rgiordandev.reg(y, x, w=w0)
se, betahat_grad, se_grad = regsens_rgiordandev.get_regression_w_grads(
    betahat, y, x, w0, se_group=se_group)
")

print(py_main$betahat_grad)
print(py_main$se_grad)

# Just run ComputeRegressionInfluence
ComputeRegressionInfluence(reg_fit, se_group=NULL)

And I see that the last call to reticulate::py_run_string() seems to be the point where it breaks, and the first call generates some warnings because some values are NaN

Sorry I can't give more insight because I have not use reticulate in my calculations.

$ Rscript test.R 
Loading required package: usethis
── Attaching packages ───────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse 1.3.0 ──
✔ ggplot2 3.3.2     ✔ purrr   0.3.4
✔ tibble  3.0.4     ✔ dplyr   1.0.2
✔ tidyr   1.1.2     ✔ stringr 1.4.0
✔ readr   1.4.0     ✔ forcats 0.5.0
── Conflicts ──────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
Loading required package: reticulate
Loading required package: latex2exp
Loading required package: broom
Loading zaminfluence

Attaching package: ‘testthat’

The following object is masked from ‘package:dplyr’:

    matches

The following object is masked from ‘package:purrr’:

    is_null

The following object is masked from ‘package:tidyr’:

    matches

The following object is masked from ‘package:devtools’:

    test_file

Attaching package: ‘gridExtra’

The following object is masked from ‘package:dplyr’:

    combine

      x1 x2 x3
 [1,]  0  0  0
 [2,]  0  0  0
 [3,]  0  0  0
 [4,]  0  0  0
 [5,]  0  0  0
 [6,]  0  0  0
 [7,]  0  0  0
 [8,]  0  0  0
 [9,]  0  0  0
[10,]  0  0  0
 [1] 0 0 0 0 0 0 0 0 0 0
[1] 0 0 0
 [1] 0 0 0 0 0 0 0 0 0 0
      x1 x2 x3
 [1,]  0  0  0
 [2,]  0  0  0
 [3,]  0  0  0
 [4,]  0  0  0
 [5,]  0  0  0
 [6,]  0  0  0
 [7,]  0  0  0
 [8,]  0  0  0
 [9,]  0  0  0
[10,]  0  0  0
 [1] 0 0 0 0 0 0 0 0 0 0
[1] 0 0 0
 [1] 0 0 0 0 0 0 0 0 0 0
/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/autograd-1.3-py3.8.egg/autograd/tracer.py:48: RuntimeWarning: invalid value encountered in sqrt
  return f_raw(*args, **kwargs)
/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/autograd-1.3-py3.8.egg/autograd/numpy/numpy_vjps.py:99: RuntimeWarning: invalid value encountered in power
  defvjp(anp.sqrt,    lambda ans, x : lambda g: g * 0.5 * x**-0.5)
            [,1]        [,2]         [,3]        [,4]         [,5]        [,6]
[1,]  0.08487399 -0.07229308  0.148011153  0.03189895  0.016717574  0.05025406
[2,] -0.02707678 -0.25624932 -0.143256362  0.15593617  0.003747749 -0.41389411
[3,]  0.05874361 -0.54259092 -0.008895226 -0.05026861 -0.001068631 -0.61169193
          [,7]       [,8]        [,9]       [,10]
[1,] 0.4767400 -0.4690735 -0.06759143 -0.19953767
[2,] 0.1889611  0.5178116  0.06956875 -0.09554881
[3,] 0.7807653  0.3860833 -0.01931623  0.00823938
     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,]  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN   NaN
[2,]  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN   NaN
[3,]  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN   NaN
Error in py_run_string_impl(code, local, convert) : 
  ValueError: The estimating equation is not zero at the proposed  values.  ||ee|| = 0.21276610243399807 > 1e-08 = solution_tol

Detailed traceback: 
  File "<string>", line 3, in <module>
  File "/home/jesus/devel/others/zaminfluence/regression_sensitivity/regsens_rgiordandev/regression_sensitivity_lib.py", line 143, in get_regression_w_grads
    obs_w_sens = vittles.HyperparameterSensitivityLinearApproximation(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 365, in __init__
    EstimatingEquationLinearApproximation.__init__(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 186, in __init__
    self.set_location(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 215, in set_location
    raise ValueError(err_msg)
Calls: <Anonymous> -> py_run_string_impl
Execution halted
rgiordan commented 3 years ago

Sorry, just to avoid ambiguity, could you please provide something that shows what command is being run, like the output from an interactive terminal, or

Rscript -e 'source("test.R", echo=TRUE)'

In case you're curious, reticulate is just running a Python interpreter in the background, passing values back and forth with R through the list-like py_main object. The zaminfluence package computes derivatives by setting variables in the Python interpreter, computing derivatives using autograd and vittles, and then passing the results back to R.

rgiordan commented 3 years ago

By the way, I assume the Python unit tests are passing, right?

source venv/bin/activate
cd $REPO/regression_sensitivity
python3 -m pytest
jmcastagnetto commented 3 years ago

In the output above I put the command line used (vide supra)

image

Idem for the results of the python unit tests

image

Cheers

rgiordan commented 3 years ago

Yes, I understand. Could I please ask you to run

Rscript -e 'source("test.R", echo=TRUE)'

instead of Rscript test.R? That way I can see all the output.

jmcastagnetto commented 3 years ago

Sure thing, here it is:

$ Rscript -e 'source("test.R", echo=TRUE)' 

> base_dir <- "/home/jesus/devel/others/zaminfluence"

> setwd(base_dir)

> library(devtools)
Loading required package: usethis

> library(tidyverse)
── Attaching packages ───────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse 1.3.0 ──
✔ ggplot2 3.3.2     ✔ purrr   0.3.4
✔ tibble  3.0.4     ✔ dplyr   1.0.2
✔ tidyr   1.1.2     ✔ stringr 1.4.0
✔ readr   1.4.0     ✔ forcats 0.5.0
── Conflicts ──────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()

> library(zaminfluence)
Loading required package: reticulate
Loading required package: latex2exp
Loading required package: broom

> devtools::load_all("zaminfluence")
Loading zaminfluence

Attaching package: ‘testthat’

The following object is masked from ‘package:dplyr’:

    matches

The following object is masked from ‘package:purrr’:

    is_null

The following object is masked from ‘package:tidyr’:

    matches

The following object is masked from ‘package:devtools’:

    test_file

> py_main <- InitializePython(file.path(base_dir, "venv/bin/python"))

> source(file.path(base_dir, "zaminfluence/tests/testthat/utils.R"))

Attaching package: ‘gridExtra’

The following object is masked from ‘package:dplyr’:

    combine

> set.seed(42)

> x_dim <- 3

> beta_true <- runif(x_dim)

> df <- GenerateRegressionData(10, beta_true, num_groups = NULL)

> x_names <- sprintf("x%d", 1:x_dim)

> reg_form <- formula(sprintf("y ~ %s - 1", paste(x_names, 
+     collapse = " + ")))

> reg_fit <- lm(data = df, formula = reg_form, x = TRUE, 
+     y = TRUE)

> py_main_set <- SetPythonRegressionVariables(reg_fit)

> CheckPyMain <- function(py_main_check) {
+     print(py_main_check$x - df[, x_names] %>% as.matrix())
+     print(py_main_check$y - df$y %>% as.nume .... [TRUNCATED] 

> CheckPyMain(py_main_set)
      x1 x2 x3
 [1,]  0  0  0
 [2,]  0  0  0
 [3,]  0  0  0
 [4,]  0  0  0
 [5,]  0  0  0
 [6,]  0  0  0
 [7,]  0  0  0
 [8,]  0  0  0
 [9,]  0  0  0
[10,]  0  0  0
 [1] 0 0 0 0 0 0 0 0 0 0
[1] 0 0 0
 [1] 0 0 0 0 0 0 0 0 0 0

> CheckPyMain(py_main)
      x1 x2 x3
 [1,]  0  0  0
 [2,]  0  0  0
 [3,]  0  0  0
 [4,]  0  0  0
 [5,]  0  0  0
 [6,]  0  0  0
 [7,]  0  0  0
 [8,]  0  0  0
 [9,]  0  0  0
[10,]  0  0  0
 [1] 0 0 0 0 0 0 0 0 0 0
[1] 0 0 0
 [1] 0 0 0 0 0 0 0 0 0 0

> reticulate::py_run_string("\nreg_obj = regsens_rgiordandev.reg_obj\nget_standard_error_matrix = regsens_rgiordandev.get_standard_error_matrix\nsens_ ..." ... [TRUNCATED] 
/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/autograd-1.3-py3.8.egg/autograd/tracer.py:48: RuntimeWarning: invalid value encountered in sqrt
  return f_raw(*args, **kwargs)
/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/autograd-1.3-py3.8.egg/autograd/numpy/numpy_vjps.py:99: RuntimeWarning: invalid value encountered in power
  defvjp(anp.sqrt,    lambda ans, x : lambda g: g * 0.5 * x**-0.5)

> print(py_main$betahat_grad)
            [,1]        [,2]         [,3]        [,4]         [,5]        [,6]
[1,]  0.08487399 -0.07229308  0.148011153  0.03189895  0.016717574  0.05025406
[2,] -0.02707678 -0.25624932 -0.143256362  0.15593617  0.003747749 -0.41389411
[3,]  0.05874361 -0.54259092 -0.008895226 -0.05026861 -0.001068631 -0.61169193
          [,7]       [,8]        [,9]       [,10]
[1,] 0.4767400 -0.4690735 -0.06759143 -0.19953767
[2,] 0.1889611  0.5178116  0.06956875 -0.09554881
[3,] 0.7807653  0.3860833 -0.01931623  0.00823938

> print(py_main$se_grad)
     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,]  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN   NaN
[2,]  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN   NaN
[3,]  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN   NaN

> py_main <- SetPythonRegressionVariables(reg_fit, se_group = NULL)

> reg <- broom::tidy(reg_fit)

> reticulate::py_run_string("\nbetahat = regsens_rgiordandev.reg(y, x, w=w0)\nse, betahat_grad, se_grad = regsens_rgiordandev.get_regression_w_grads(\ ..." ... [TRUNCATED] 
Error in py_run_string_impl(code, local, convert) : 
  ValueError: The estimating equation is not zero at the proposed  values.  ||ee|| = 0.21276610243399807 > 1e-08 = solution_tol

Detailed traceback: 
  File "<string>", line 3, in <module>
  File "/home/jesus/devel/others/zaminfluence/regression_sensitivity/regsens_rgiordandev/regression_sensitivity_lib.py", line 143, in get_regression_w_grads
    obs_w_sens = vittles.HyperparameterSensitivityLinearApproximation(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 365, in __init__
    EstimatingEquationLinearApproximation.__init__(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 186, in __init__
    self.set_location(
  File "/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/vittles-0.173-py3.8.egg/vittles/sensitivity_lib.py", line 215, in set_location
    raise ValueError(err_msg)
Calls: source ... withVisible -> eval -> eval -> <Anonymous> -> py_run_string_impl
Execution halted

Confirms that the last reticulate call is where it breaks.

rgiordan commented 3 years ago

Thanks for your patience! Before digging into the Python line by line, let's make sure that reticulate is picking up the right virtual environment. Could you please run (using Rscript -e 'source("test.R", echo=TRUE)'):

# Set this to your own zaminfluence directory.
base_dir  <- "/home/rgiordan/Documents/git_repos/zaminfluence"
setwd(base_dir)

library(devtools)
library(tidyverse)
library(zaminfluence)

devtools::load_all("zaminfluence")  # Load even non-exported zaminfluence functions.
py_main <- InitializePython(file.path(base_dir, "venv/bin/python"))

# Manually run the Python code in ComputeRegressionInfluence.  This is showing
# some problems.
PyPrint <- function(py_expr) {
    reticulate::py_run_string(sprintf("foo = %s", py_expr))
    print(py_main$foo)
}

reticulate::py_run_string("import pkg_resources")
reticulate::py_run_string("import sys")
PyPrint("sys.version")
PyPrint("vittles.__version__")
PyPrint("paragami.__version__")
PyPrint("pkg_resources.get_distribution(\"autograd\")")
rgiordan commented 3 years ago

Actually, please run the below script, which contains the version checks. Hopefully this will give some clues as to where things are going wrong.

# Set this to your own zaminfluence directory.
base_dir  <- "/home/rgiordan/Documents/git_repos/zaminfluence"
setwd(base_dir)

library(devtools)
library(tidyverse)
library(zaminfluence)

devtools::load_all("zaminfluence")  # Load even non-exported zaminfluence functions.
py_main <- InitializePython(file.path(base_dir, "venv/bin/python"))

# Manually run the Python code in ComputeRegressionInfluence.  This is showing
# some problems.
PyPrint <- function(py_expr) {
    reticulate::py_run_string(sprintf("foo = %s", py_expr))
    print(py_main$foo)
}

reticulate::py_run_string("import pkg_resources")
reticulate::py_run_string("import sys")
PyPrint("sys.version")
PyPrint("vittles.__version__")
PyPrint("paragami.__version__")
PyPrint("pkg_resources.get_distribution(\"autograd\")")

#####################

# Test utility for generating data
source(file.path(base_dir, "zaminfluence/tests/testthat/utils.R"))

set.seed(42)

# Generate data and a regression model.
x_dim <- 3
beta_true <- runif(x_dim)
df <- GenerateRegressionData(10, beta_true, num_groups=NULL)

# Fit a regression model.
x_names <- sprintf("x%d", 1:x_dim)
reg_form <- formula(sprintf("y ~ %s - 1", paste(x_names, collapse=" + ")))
reg_fit <- lm(data = df, formula = reg_form, x=TRUE, y=TRUE)

py_main <- SetPythonRegressionVariables(reg_fit)

# If we are setting the python variables correctly, these differences should all be zero.
CheckPyMain <- function(py_main_check) {
    print(py_main_check$x - df[, x_names] %>% as.matrix())
    print(py_main_check$y - df$y %>% as.numeric())
    print(py_main_check$beta - reg_fit$coefficients)
    print(py_main_check$w0 - rep(1, nrow(df)))
}

CheckPyMain(py_main)

reticulate::py_run_string("
reg_obj = regsens_rgiordandev.reg_obj
get_standard_error_matrix = regsens_rgiordandev.get_standard_error_matrix
sens_reg_obj = lambda beta, w: reg_obj(beta, y=y, x=x, w=w)
obs_w_sens = vittles.HyperparameterSensitivityLinearApproximation(
    objective_fun=sens_reg_obj,
    opt_par_value=beta,
    hyper_par_value=w0,
    validate_optimum=True,
    grad_tol=1e-08)
get_betahat = obs_w_sens.get_opt_par_function()
def get_se(w):
    betahat = get_betahat(w)
    print('get_se computed betahat = {}'.format(betahat))
    se_cov = get_standard_error_matrix(betahat, y, x, w=w, se_group=se_group)
    return np.sqrt(np.diag(se_cov))
")

PyPrint("get_betahat(w0)")
PyPrint("obs_w_sens._input_val0")
PyPrint("obs_w_sens._hyper_val0")

PyPrint("get_betahat(w0)")
PyPrint("obs_w_sens._input_val0")
PyPrint("obs_w_sens._hyper_val0")

PyPrint("get_se(w0)")
PyPrint("obs_w_sens._input_val0")
PyPrint("obs_w_sens._hyper_val0")

PyPrint("get_se(w0)")
PyPrint("obs_w_sens._input_val0")
PyPrint("obs_w_sens._hyper_val0")

PyPrint("obs_w_sens.get_dopt_dhyper()")
PyPrint("obs_w_sens._input_val0")
PyPrint("obs_w_sens._hyper_val0")

PyPrint("autograd.jacobian(get_se)(w0)")
PyPrint("obs_w_sens._input_val0")
PyPrint("obs_w_sens._hyper_val0")
jmcastagnetto commented 3 years ago

OK, just finished some morning meetings, so I was able to run the script:

$ Rscript -e 'source("test2.R", echo=TRUE)' 

> base_dir <- "/home/jesus/devel/others/zaminfluence"

> setwd(base_dir)

> library(devtools)
Loading required package: usethis

> library(tidyverse)
── Attaching packages ─────────────────────────────────────── tidyverse 1.3.0 ──
✔ ggplot2 3.3.2     ✔ purrr   0.3.4
✔ tibble  3.0.4     ✔ dplyr   1.0.2
✔ tidyr   1.1.2     ✔ stringr 1.4.0
✔ readr   1.4.0     ✔ forcats 0.5.0
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()

> library(zaminfluence)
Loading required package: reticulate
Loading required package: latex2exp
Loading required package: broom

> devtools::load_all("zaminfluence")
Loading zaminfluence

Attaching package: ‘testthat’

The following object is masked from ‘package:dplyr’:

    matches

The following object is masked from ‘package:purrr’:

    is_null

The following object is masked from ‘package:tidyr’:

    matches

The following object is masked from ‘package:devtools’:

    test_file

> py_main <- InitializePython(file.path(base_dir, "venv/bin/python"))

> PyPrint <- function(py_expr) {
+     reticulate::py_run_string(sprintf("foo = %s", py_expr))
+     print(py_main$foo)
+ }

> reticulate::py_run_string("import pkg_resources")

> reticulate::py_run_string("import sys")

> PyPrint("sys.version")
[1] "3.8.5 (default, Jul 28 2020, 12:59:40) \n[GCC 9.3.0]"

> PyPrint("vittles.__version__")
[1] "0.173"

> PyPrint("paragami.__version__")
[1] "0.42"

> PyPrint("pkg_resources.get_distribution(\"autograd\")")
autograd 1.3

> source(file.path(base_dir, "zaminfluence/tests/testthat/utils.R"))

Attaching package: ‘gridExtra’

The following object is masked from ‘package:dplyr’:

    combine

> set.seed(42)

> x_dim <- 3

> beta_true <- runif(x_dim)

> df <- GenerateRegressionData(10, beta_true, num_groups = NULL)

> x_names <- sprintf("x%d", 1:x_dim)

> reg_form <- formula(sprintf("y ~ %s - 1", paste(x_names, 
+     collapse = " + ")))

> reg_fit <- lm(data = df, formula = reg_form, x = TRUE, 
+     y = TRUE)

> py_main <- SetPythonRegressionVariables(reg_fit)

> CheckPyMain <- function(py_main_check) {
+     print(py_main_check$x - df[, x_names] %>% as.matrix())
+     print(py_main_check$y - df$y %>% as.nume .... [TRUNCATED] 

> CheckPyMain(py_main)
      x1 x2 x3
 [1,]  0  0  0
 [2,]  0  0  0
 [3,]  0  0  0
 [4,]  0  0  0
 [5,]  0  0  0
 [6,]  0  0  0
 [7,]  0  0  0
 [8,]  0  0  0
 [9,]  0  0  0
[10,]  0  0  0
 [1] 0 0 0 0 0 0 0 0 0 0
[1] 0 0 0
 [1] 0 0 0 0 0 0 0 0 0 0

> reticulate::py_run_string("\nreg_obj = regsens_rgiordandev.reg_obj\nget_standard_error_matrix = regsens_rgiordandev.get_standard_error_matrix\nsens_ ..." ... [TRUNCATED] 

> PyPrint("get_betahat(w0)")
[1]  1.9036205 -0.1935714 -0.3037477

> PyPrint("obs_w_sens._input_val0")
[1]  1.9036205 -0.1935714 -0.3037477

> PyPrint("obs_w_sens._hyper_val0")
 [1] 1 1 1 1 1 1 1 1 1 1

> PyPrint("get_betahat(w0)")
[1]  1.9036205 -0.1935714 -0.3037477

> PyPrint("obs_w_sens._input_val0")
[1]  1.9036205 -0.1935714 -0.3037477

> PyPrint("obs_w_sens._hyper_val0")
 [1] 1 1 1 1 1 1 1 1 1 1

> PyPrint("get_se(w0)")
/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/autograd-1.3-py3.8.egg/autograd/tracer.py:48: RuntimeWarning: invalid value encountered in sqrt
  return f_raw(*args, **kwargs)
get_se computed betahat = [ 1.90362055 -0.19357143 -0.30374769]
[1] 0.9887754       NaN       NaN

> PyPrint("obs_w_sens._input_val0")
[1]  1.9036205 -0.1935714 -0.3037477

> PyPrint("obs_w_sens._hyper_val0")
 [1] 1 1 1 1 1 1 1 1 1 1

> PyPrint("get_se(w0)")
get_se computed betahat = [ 1.90362055 -0.19357143 -0.30374769]
[1] 0.9887754       NaN       NaN

> PyPrint("obs_w_sens._input_val0")
[1]  1.9036205 -0.1935714 -0.3037477

> PyPrint("obs_w_sens._hyper_val0")
 [1] 1 1 1 1 1 1 1 1 1 1

> PyPrint("obs_w_sens.get_dopt_dhyper()")
            [,1]        [,2]         [,3]        [,4]         [,5]        [,6]
[1,]  0.08487399 -0.07229308  0.148011153  0.03189895  0.016717574  0.05025406
[2,] -0.02707678 -0.25624932 -0.143256362  0.15593617  0.003747749 -0.41389411
[3,]  0.05874361 -0.54259092 -0.008895226 -0.05026861 -0.001068631 -0.61169193
          [,7]       [,8]        [,9]       [,10]
[1,] 0.4767400 -0.4690735 -0.06759143 -0.19953767
[2,] 0.1889611  0.5178116  0.06956875 -0.09554881
[3,] 0.7807653  0.3860833 -0.01931623  0.00823938

> PyPrint("obs_w_sens._input_val0")
[1]  1.9036205 -0.1935714 -0.3037477

> PyPrint("obs_w_sens._hyper_val0")
 [1] 1 1 1 1 1 1 1 1 1 1

> PyPrint("autograd.jacobian(get_se)(w0)")
/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/autograd-1.3-py3.8.egg/autograd/numpy/numpy_vjps.py:99: RuntimeWarning: invalid value encountered in power
  defvjp(anp.sqrt,    lambda ans, x : lambda g: g * 0.5 * x**-0.5)
get_se computed betahat = Autograd ArrayBox with value [ 1.90362055 -0.19357143 -0.30374769]
     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,]  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN   NaN
[2,]  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN   NaN
[3,]  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN   NaN

> PyPrint("obs_w_sens._input_val0")
[1]  1.9036205 -0.1935714 -0.3037477

> PyPrint("obs_w_sens._hyper_val0")
 [1] 1 1 1 1 1 1 1 1 1 1

HTH

rgiordan commented 3 years ago

Ok, we're zooming in. Perhaps the regression data is getting corrupted. Could you please run this one?

# Set this to your own zaminfluence directory.
base_dir  <- "/home/rgiordan/Documents/git_repos/zaminfluence"
setwd(base_dir)

library(devtools)
library(tidyverse)
library(zaminfluence)

devtools::load_all("zaminfluence")  # Load even non-exported zaminfluence functions.
py_main <- InitializePython(file.path(base_dir, "venv/bin/python"))

# Manually run the Python code in ComputeRegressionInfluence.  This is showing
# some problems.
PyPrint <- function(py_expr) {
    reticulate::py_run_string(sprintf("foo = %s", py_expr))
    print(py_main$foo)
}

reticulate::py_run_string("import pkg_resources")
reticulate::py_run_string("import sys")
PyPrint("sys.version")
PyPrint("vittles.__version__")
PyPrint("paragami.__version__")
PyPrint("pkg_resources.get_distribution(\"autograd\")")

#####################

# Test utility for generating data
source(file.path(base_dir, "zaminfluence/tests/testthat/utils.R"))

set.seed(42)

# Generate data and a regression model.
x_dim <- 3
beta_true <- runif(x_dim)
df <- GenerateRegressionData(10, beta_true, num_groups=NULL)

# Fit a regression model.
x_names <- sprintf("x%d", 1:x_dim)
reg_form <- formula(sprintf("y ~ %s - 1", paste(x_names, collapse=" + ")))
reg_fit <- lm(data = df, formula = reg_form, x=TRUE, y=TRUE)

py_main <- SetPythonRegressionVariables(reg_fit)

# If we are setting the python variables correctly, these differences should all be zero.
CheckPyMain <- function(py_main_check) {
    print(py_main_check$x - df[, x_names] %>% as.matrix())
    print(py_main_check$y - df$y %>% as.numeric())
    print(py_main_check$beta - reg_fit$coefficients)
    print(py_main_check$w0 - rep(1, nrow(df)))
}

CheckPyMain(py_main)

# Make deep copies of the correct values
reticulate::py_run_string("
from copy import deepcopy
x_copy = deepcopy(x)
y_copy = deepcopy(y)
w0_copy = deepcopy(w0)
beta_copy = deepcopy(beta)
")

reticulate::py_run_string("
reg_obj = regsens_rgiordandev.reg_obj
get_standard_error_matrix = regsens_rgiordandev.get_standard_error_matrix
sens_reg_obj = lambda beta, w: reg_obj(beta, y=y, x=x, w=w)
obs_w_sens = vittles.HyperparameterSensitivityLinearApproximation(
    objective_fun=sens_reg_obj,
    opt_par_value=beta,
    hyper_par_value=w0,
    validate_optimum=True,
    grad_tol=1e-08)

get_betahat = obs_w_sens.get_opt_par_function()

def get_se(w):
    betahat = get_betahat(w)
    print('get_se computed betahat = {}'.format(betahat))
    print('get_se x error = {}'.format(x_copy - x))
    print('get_se y error = {}'.format(y_copy - y))
    print('get_se w difference = {}'.format(w0_copy - w))
    print('get_se se_group = {}'.format(se_group))
    se_cov = get_standard_error_matrix(betahat, y, x, w=w, se_group=se_group)
    print('get_se se_cov = {}'.format(se_cov))
    return np.sqrt(np.diag(se_cov))
")

# Run it twice just to make sure the values are the same each time.
PyPrint("get_se(w0)")
PyPrint("get_se(w0)")
jmcastagnetto commented 3 years ago

Here is the output of running the code above.

$ Rscript -e 'source("test3.R", echo=TRUE)' 

> base_dir <- "/home/jesus/devel/others/zaminfluence"

> setwd(base_dir)

> library(devtools)
Loading required package: usethis

> library(tidyverse)
── Attaching packages ───────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse 1.3.0 ──
✔ ggplot2 3.3.2     ✔ purrr   0.3.4
✔ tibble  3.0.4     ✔ dplyr   1.0.2
✔ tidyr   1.1.2     ✔ stringr 1.4.0
✔ readr   1.4.0     ✔ forcats 0.5.0
── Conflicts ──────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()

> library(zaminfluence)
Loading required package: reticulate
Loading required package: latex2exp
Loading required package: broom

> devtools::load_all("zaminfluence")
Loading zaminfluence

Attaching package: ‘testthat’

The following object is masked from ‘package:dplyr’:

    matches

The following object is masked from ‘package:purrr’:

    is_null

The following object is masked from ‘package:tidyr’:

    matches

The following object is masked from ‘package:devtools’:

    test_file

> py_main <- InitializePython(file.path(base_dir, "venv/bin/python"))

> PyPrint <- function(py_expr) {
+     reticulate::py_run_string(sprintf("foo = %s", py_expr))
+     print(py_main$foo)
+ }

> reticulate::py_run_string("import pkg_resources")

> reticulate::py_run_string("import sys")

> PyPrint("sys.version")
[1] "3.8.5 (default, Jul 28 2020, 12:59:40) \n[GCC 9.3.0]"

> PyPrint("vittles.__version__")
[1] "0.173"

> PyPrint("paragami.__version__")
[1] "0.42"

> PyPrint("pkg_resources.get_distribution(\"autograd\")")
autograd 1.3

> source(file.path(base_dir, "zaminfluence/tests/testthat/utils.R"))

Attaching package: ‘gridExtra’

The following object is masked from ‘package:dplyr’:

    combine

> set.seed(42)

> x_dim <- 3

> beta_true <- runif(x_dim)

> df <- GenerateRegressionData(10, beta_true, num_groups = NULL)

> x_names <- sprintf("x%d", 1:x_dim)

> reg_form <- formula(sprintf("y ~ %s - 1", paste(x_names, 
+     collapse = " + ")))

> reg_fit <- lm(data = df, formula = reg_form, x = TRUE, 
+     y = TRUE)

> py_main <- SetPythonRegressionVariables(reg_fit)

> CheckPyMain <- function(py_main_check) {
+     print(py_main_check$x - df[, x_names] %>% as.matrix())
+     print(py_main_check$y - df$y %>% as.nume .... [TRUNCATED] 

> CheckPyMain(py_main)
      x1 x2 x3
 [1,]  0  0  0
 [2,]  0  0  0
 [3,]  0  0  0
 [4,]  0  0  0
 [5,]  0  0  0
 [6,]  0  0  0
 [7,]  0  0  0
 [8,]  0  0  0
 [9,]  0  0  0
[10,]  0  0  0
 [1] 0 0 0 0 0 0 0 0 0 0
[1] 0 0 0
 [1] 0 0 0 0 0 0 0 0 0 0

> reticulate::py_run_string("\nfrom copy import deepcopy\nx_copy = deepcopy(x)\ny_copy = deepcopy(y)\nw0_copy = deepcopy(w0)\nbeta_copy = deepcopy(bet ..." ... [TRUNCATED] 

> reticulate::py_run_string("\nreg_obj = regsens_rgiordandev.reg_obj\nget_standard_error_matrix = regsens_rgiordandev.get_standard_error_matrix\nsens_ ..." ... [TRUNCATED] 

> PyPrint("get_se(w0)")
get_se computed betahat = [ 1.90362055 -0.19357143 -0.30374769]
get_se x error = [[0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]]
get_se y error = [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
get_se w difference = [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
get_se se_group = None
get_se se_cov = [[ 0.97767675 -0.46919679  2.4775263 ]
 [ 1.00699179 -0.31824127 -0.00303038]
 [ 0.06094292 -2.56488699 -0.04013961]]
/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/autograd-1.3-py3.8.egg/autograd/tracer.py:48: RuntimeWarning: invalid value encountered in sqrt
  return f_raw(*args, **kwargs)
[1] 0.9887754       NaN       NaN

> PyPrint("get_se(w0)")
get_se computed betahat = [ 1.90362055 -0.19357143 -0.30374769]
get_se x error = [[0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]]
get_se y error = [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
get_se w difference = [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
get_se se_group = None
get_se se_cov = [[ 0.97767675 -0.46919679  2.4775263 ]
 [ 1.00699179 -0.31824127 -0.00303038]
 [ 0.06094292 -2.56488699 -0.04013961]]
[1] 0.9887754       NaN       NaN

And don't worry if you need me to run more stuff, I've been there trying to debug what was happening with code I wrote in other people's config, so I will try to help with what I can.

rgiordan commented 3 years ago

Thanks again for your patience. I do wish I could reproduce your error on my side so I could iterate more quickly. Unfortunately my guesses about where things are going wrong are all coming up blank.

Starting after CheckPyMain(py_main) in the previous script I sent, could you run the attached code? In a sense I'm hoping this will fail in the same way so I don't have to go and have you debug the Python library.

############# Starting after CheckPyMain(py_main):

# Make deep copies of the correct values
reticulate::py_run_string("
from copy import deepcopy
x_copy = deepcopy(x)
y_copy = deepcopy(y)
w0_copy = deepcopy(w0)
beta_copy = deepcopy(beta)
")

reticulate::py_run_string("
reg_obj = regsens_rgiordandev.reg_obj
get_standard_error_matrix = regsens_rgiordandev.get_standard_error_matrix
sens_reg_obj = lambda beta, w: reg_obj(beta, y=y, x=x, w=w)
obs_w_sens = vittles.HyperparameterSensitivityLinearApproximation(
    objective_fun=sens_reg_obj,
    opt_par_value=beta,
    hyper_par_value=w0,
    validate_optimum=True,
    grad_tol=1e-08)

get_betahat = obs_w_sens.get_opt_par_function()

def get_se(w):
    betahat = get_betahat(w)
    print('get_se computed betahat = {}'.format(betahat))
    print('get_se x error = {}'.format(x_copy - x))
    print('get_se y error = {}'.format(y_copy - y))
    print('get_se w difference = {}'.format(w0_copy - w))
    print('get_se se_group = {}'.format(se_group))
    se_cov = get_standard_error_matrix(betahat, y, x, w=w, se_group=se_group)
    print('get_se se_cov = {}'.format(se_cov))
    return np.sqrt(np.diag(se_cov))

# Copied from regsens_rgiordandev.get_standard_error_matrix with debugging
def get_standard_error_matrix_local(betahat, y, x, w, se_group=None):
    assert(se_group is None)

    resid = y - x @ betahat
    num_obs = len(y)
    xtx_bar = np.einsum('ni,nj,n->ij', x, x, w) / num_obs
    sigma2hat = np.sum(w * (resid ** 2)) / (num_obs - len(betahat))
    xtx_inv = np.linalg.inv(xtx_bar)
    se2 = sigma2hat * xtx_inv / num_obs

    print('get_se_local betahat = {}'.format(betahat))
    print('get_se_local y = {}'.format(y))
    print('get_se_local x = {}'.format(x))
    print('get_se_local w = {}'.format(w))
    print('get_se_local resid = {}'.format(resid))
    print('get_se_local xtx_bar = {}'.format(xtx_bar))
    print('get_se_local sigma2hat = {}'.format(sigma2hat))

    return se2
")

PyPrint("get_se(w0)")
reticulate::py_run_string("betahat = get_betahat(w0)")
PyPrint("get_standard_error_matrix_local(betahat, y, x, w=w0, se_group=se_group)")
jmcastagnetto commented 3 years ago

OK, replaced the lines you suggested with the new code, and here is the result:

$ Rscript -e 'source("test4.R", echo=TRUE)'

> base_dir <- "/home/jesus/devel/others/zaminfluence"

> setwd(base_dir)

> library(devtools)
Loading required package: usethis

> library(tidyverse)
── Attaching packages ───────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse 1.3.0 ──
✔ ggplot2 3.3.2     ✔ purrr   0.3.4
✔ tibble  3.0.4     ✔ dplyr   1.0.2
✔ tidyr   1.1.2     ✔ stringr 1.4.0
✔ readr   1.4.0     ✔ forcats 0.5.0
── Conflicts ──────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()

> library(zaminfluence)
Loading required package: reticulate
Loading required package: latex2exp
Loading required package: broom

> devtools::load_all("zaminfluence")
Loading zaminfluence

Attaching package: ‘testthat’

The following object is masked from ‘package:dplyr’:

    matches

The following object is masked from ‘package:purrr’:

    is_null

The following object is masked from ‘package:tidyr’:

    matches

The following object is masked from ‘package:devtools’:

    test_file

> py_main <- InitializePython(file.path(base_dir, "venv/bin/python"))

> PyPrint <- function(py_expr) {
+     reticulate::py_run_string(sprintf("foo = %s", py_expr))
+     print(py_main$foo)
+ }

> reticulate::py_run_string("import pkg_resources")

> reticulate::py_run_string("import sys")

> PyPrint("sys.version")
[1] "3.8.5 (default, Jul 28 2020, 12:59:40) \n[GCC 9.3.0]"

> PyPrint("vittles.__version__")
[1] "0.173"

> PyPrint("paragami.__version__")
[1] "0.42"

> PyPrint("pkg_resources.get_distribution(\"autograd\")")
autograd 1.3

> source(file.path(base_dir, "zaminfluence/tests/testthat/utils.R"))

Attaching package: ‘gridExtra’

The following object is masked from ‘package:dplyr’:

    combine

> set.seed(42)

> x_dim <- 3

> beta_true <- runif(x_dim)

> df <- GenerateRegressionData(10, beta_true, num_groups = NULL)

> x_names <- sprintf("x%d", 1:x_dim)

> reg_form <- formula(sprintf("y ~ %s - 1", paste(x_names, 
+     collapse = " + ")))

> reg_fit <- lm(data = df, formula = reg_form, x = TRUE, 
+     y = TRUE)

> py_main <- SetPythonRegressionVariables(reg_fit)

> CheckPyMain <- function(py_main_check) {
+     print(py_main_check$x - df[, x_names] %>% as.matrix())
+     print(py_main_check$y - df$y %>% as.nume .... [TRUNCATED] 

> CheckPyMain(py_main)
      x1 x2 x3
 [1,]  0  0  0
 [2,]  0  0  0
 [3,]  0  0  0
 [4,]  0  0  0
 [5,]  0  0  0
 [6,]  0  0  0
 [7,]  0  0  0
 [8,]  0  0  0
 [9,]  0  0  0
[10,]  0  0  0
 [1] 0 0 0 0 0 0 0 0 0 0
[1] 0 0 0
 [1] 0 0 0 0 0 0 0 0 0 0

> cat("***** 2020-12-15 CODE *****\n\n")
***** 2020-12-15 CODE *****

> reticulate::py_run_string("\nfrom copy import deepcopy\nx_copy = deepcopy(x)\ny_copy = deepcopy(y)\nw0_copy = deepcopy(w0)\nbeta_copy = deepcopy(bet ..." ... [TRUNCATED] 

> reticulate::py_run_string("\nreg_obj = regsens_rgiordandev.reg_obj\nget_standard_error_matrix = regsens_rgiordandev.get_standard_error_matrix\nsens_ ..." ... [TRUNCATED] 

> PyPrint("get_se(w0)")
get_se computed betahat = [ 1.90362055 -0.19357143 -0.30374769]
get_se x error = [[0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]]
get_se y error = [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
get_se w difference = [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
get_se se_group = None
get_se se_cov = [[ 0.97767675 -0.46919679  2.4775263 ]
 [ 1.00699179 -0.31824127 -0.00303038]
 [ 0.06094292 -2.56488699 -0.04013961]]
/home/jesus/devel/others/zaminfluence/venv/lib/python3.8/site-packages/autograd-1.3-py3.8.egg/autograd/tracer.py:48: RuntimeWarning: invalid value encountered in sqrt
  return f_raw(*args, **kwargs)
[1] 0.9887754       NaN       NaN

> reticulate::py_run_string("betahat = get_betahat(w0)")

> PyPrint("get_standard_error_matrix_local(betahat, y, x, w=w0, se_group=se_group)")
get_se_local betahat = [ 1.90362055 -0.19357143 -0.30374769]
get_se_local y = [ 0.4537174   0.71119636 -0.61011909  0.68769675 -0.9676639   1.81117498
  1.83409516  0.53692678  0.02391491  0.20022749]
get_se_local x = [[ 0.19683489 -0.32661248  0.34076902]
 [ 0.00813278 -0.11974848 -0.52346165]
 [-0.11451679  0.35797322 -0.09168743]
 [ 0.10297558  0.39618512 -0.21569574]
 [-0.49894614 -0.46455395  0.29983892]
 [ 0.02337955 -0.10704423 -0.15892958]
 [ 0.07145205 -0.02170856  0.23010505]
 [-0.17587096  0.32199008  0.13169641]
 [ 0.08549952 -0.44333114  0.20515593]
 [ 0.30105951  0.40685042 -0.21779093]]
get_se_local w = [1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
get_se_local resid = [ 0.11930341  0.51353448 -0.35067903  0.50284335 -0.01670877  1.69767398
  1.76376929  0.97404891 -0.16234433 -0.36027446]
get_se_local xtx_bar = [[ 0.0446005   0.01902273 -0.01569602]
 [ 0.01902273  0.10996104 -0.04313223]
 [-0.01569602 -0.04313223  0.0720043 ]]
get_se_local sigma2hat = 1.107423007435789
           [,1]       [,2]         [,3]
[1,] 0.97767675 -0.4691968  2.477526302
[2,] 1.00699179 -0.3182413 -0.003030384
[3,] 0.06094292 -2.5648870 -0.040139607

What is common is the warning given by the autograd package. Will try to reinstall just that one and see if that helps (or not)

jmcastagnetto commented 3 years ago

OK, recloned the repo into a new path, and rerun the procedure, now they are some new errors on the python part (when creating the modules wheels), and the R test seems to pass although there are some steps skipped.

Here is the whole session log

$ git clone https://github.com/rgiordan/zaminfluence.git
Cloning into 'zaminfluence'...
remote: Enumerating objects: 286, done.
remote: Counting objects: 100% (286/286), done.
remote: Compressing objects: 100% (167/167), done.
remote: Total 286 (delta 135), reused 252 (delta 108), pack-reused 0
Receiving objects: 100% (286/286), 89.87 KiB | 76.00 KiB/s, done.
Resolving deltas: 100% (135/135), done.

$ cd zaminfluence

$ export REPO=$(pwd)

$ python3 -m venv venv 

$ source venv/bin/activate

$ python3 -m pip install -e regression_sensitivity
Obtaining file:///home/jesus/devel/others/zaminfluence/regression_sensitivity
Collecting autograd>=1.3
  Using cached autograd-1.3.tar.gz (38 kB)
Collecting ipykernel
  Using cached ipykernel-5.4.2-py3-none-any.whl (119 kB)
Collecting numpy
  Using cached numpy-1.19.4-cp38-cp38-manylinux2010_x86_64.whl (14.5 MB)
Collecting paragami>=0.42
  Using cached paragami-0.42-py3-none-any.whl (41 kB)
Collecting vittles>=0.171
  Using cached vittles-0.173-py3-none-any.whl (26 kB)
Collecting future>=0.15.2
  Using cached future-0.18.2.tar.gz (829 kB)
Collecting traitlets>=4.1.0
  Using cached traitlets-5.0.5-py3-none-any.whl (100 kB)
Collecting tornado>=4.2
  Using cached tornado-6.1-cp38-cp38-manylinux2010_x86_64.whl (427 kB)
Collecting ipython>=5.0.0
  Using cached ipython-7.19.0-py3-none-any.whl (784 kB)
Collecting jupyter-client
  Using cached jupyter_client-6.1.7-py3-none-any.whl (108 kB)
Collecting pytest
  Using cached pytest-6.2.1-py3-none-any.whl (279 kB)
Collecting scipy
  Using cached scipy-1.5.4-cp38-cp38-manylinux1_x86_64.whl (25.8 MB)
Collecting ipython-genutils
  Using cached ipython_genutils-0.2.0-py2.py3-none-any.whl (26 kB)
Collecting prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0
  Using cached prompt_toolkit-3.0.8-py3-none-any.whl (355 kB)
Collecting pexpect>4.3; sys_platform != "win32"
  Using cached pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Requirement already satisfied: setuptools>=18.5 in ./venv/lib/python3.8/site-packages (from ipython>=5.0.0->ipykernel->regsens-rgiordandev==0.1) (44.0.0)
Collecting pygments
  Using cached Pygments-2.7.3-py3-none-any.whl (950 kB)
Collecting pickleshare
  Using cached pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)
Collecting decorator
  Using cached decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Collecting backcall
  Using cached backcall-0.2.0-py2.py3-none-any.whl (11 kB)
Collecting jedi>=0.10
  Using cached jedi-0.17.2-py2.py3-none-any.whl (1.4 MB)
Collecting python-dateutil>=2.1
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting pyzmq>=13
  Using cached pyzmq-20.0.0-cp38-cp38-manylinux1_x86_64.whl (1.1 MB)
Collecting jupyter-core>=4.6.0
  Using cached jupyter_core-4.7.0-py3-none-any.whl (82 kB)
Collecting pluggy<1.0.0a1,>=0.12
  Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting toml
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting iniconfig
  Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting packaging
  Using cached packaging-20.8-py2.py3-none-any.whl (39 kB)
Collecting attrs>=19.2.0
  Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)
Collecting py>=1.8.2
  Using cached py-1.10.0-py2.py3-none-any.whl (97 kB)
Collecting wcwidth
  Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting ptyprocess>=0.5
  Using cached ptyprocess-0.6.0-py2.py3-none-any.whl (39 kB)
Collecting parso<0.8.0,>=0.7.0
  Using cached parso-0.7.1-py2.py3-none-any.whl (109 kB)
Collecting six>=1.5
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting pyparsing>=2.0.2
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Building wheels for collected packages: autograd, future
  Building wheel for autograd (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/jesus/devel/others/zaminfluence/venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ir6tp3hl/autograd/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ir6tp3hl/autograd/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-1et65h4u
       cwd: /tmp/pip-install-ir6tp3hl/autograd/
  Complete output (6 lines):
  usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: setup.py --help [cmd1 cmd2 ...]
     or: setup.py --help-commands
     or: setup.py cmd --help

  error: invalid command 'bdist_wheel'
  ----------------------------------------
  ERROR: Failed building wheel for autograd
  Running setup.py clean for autograd
  Building wheel for future (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/jesus/devel/others/zaminfluence/venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ir6tp3hl/future/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ir6tp3hl/future/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-4md3o1ra
       cwd: /tmp/pip-install-ir6tp3hl/future/
  Complete output (6 lines):
  usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: setup.py --help [cmd1 cmd2 ...]
     or: setup.py --help-commands
     or: setup.py cmd --help

  error: invalid command 'bdist_wheel'
  ----------------------------------------
  ERROR: Failed building wheel for future
  Running setup.py clean for future
Failed to build autograd future
Installing collected packages: future, numpy, autograd, ipython-genutils, traitlets, tornado, wcwidth, prompt-toolkit, ptyprocess, pexpect, pygments, pickleshare, decorator, backcall, parso, jedi, ipython, six, python-dateutil, pyzmq, jupyter-core, jupyter-client, ipykernel, pluggy, toml, iniconfig, pyparsing, packaging, attrs, py, pytest, scipy, paragami, vittles, regsens-rgiordandev
    Running setup.py install for future ... done
    Running setup.py install for autograd ... done
  Running setup.py develop for regsens-rgiordandev
Successfully installed attrs-20.3.0 autograd-1.3 backcall-0.2.0 decorator-4.4.2 future-0.18.2 iniconfig-1.1.1 ipykernel-5.4.2 ipython-7.19.0 ipython-genutils-0.2.0 jedi-0.17.2 jupyter-client-6.1.7 jupyter-core-4.7.0 numpy-1.19.4 packaging-20.8 paragami-0.42 parso-0.7.1 pexpect-4.8.0 pickleshare-0.7.5 pluggy-0.13.1 prompt-toolkit-3.0.8 ptyprocess-0.6.0 py-1.10.0 pygments-2.7.3 pyparsing-2.4.7 pytest-6.2.1 python-dateutil-2.8.1 pyzmq-20.0.0 regsens-rgiordandev scipy-1.5.4 six-1.15.0 toml-0.10.2 tornado-6.1 traitlets-5.0.5 vittles-0.173 wcwidth-0.2.5

$ cd regression_sensitivity/

$ python3 -m pytest
============================================================== test session starts ===============================================================
platform linux -- Python 3.8.5, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /home/jesus/devel/others/zaminfluence/regression_sensitivity
collected 2 items                                                                                                                                

regsens_rgiordandev/tests/test_basics.py ..                                                                                                [100%]

=============================================================== 2 passed in 0.84s ================================================================

$ cd ../zaminfluence/tests/

$ ./testthat.R 
Loading required package: reticulate
Loading required package: dplyr

Attaching package: ‘dplyr’

The following object is masked from ‘package:testthat’:

    matches

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

Loading required package: ggplot2
Loading required package: latex2exp
Loading required package: stringr
Loading required package: broom
── Skip (test_end_to_end.R:16:1): python runs ────────────────────────────────────────────────────────────────────────────────────────────────────
Reason: empty test

── Skip (test_end_to_end.R:91:1): plotting works ─────────────────────────────────────────────────────────────────────────────────────────────────
Reason: empty test

── Warning (test_end_to_end.R:276:11): aggregated pairing works ──────────────────────────────────────────────────────────────────────────────────
Unknown or uninitialised column: `row_0`.

── Warning (test_end_to_end.R:277:11): aggregated pairing works ──────────────────────────────────────────────────────────────────────────────────
Unknown or uninitialised column: `row_1`.

── Warning (test_end_to_end.R:276:11): aggregated pairing works ──────────────────────────────────────────────────────────────────────────────────
Unknown or uninitialised column: `row_0`.

── Warning (test_end_to_end.R:277:11): aggregated pairing works ──────────────────────────────────────────────────────────────────────────────────
Unknown or uninitialised column: `row_1`.

── Warning (test_end_to_end.R:276:11): aggregated pairing works ──────────────────────────────────────────────────────────────────────────────────
Unknown or uninitialised column: `row_0`.

── Warning (test_end_to_end.R:277:11): aggregated pairing works ──────────────────────────────────────────────────────────────────────────────────
Unknown or uninitialised column: `row_1`.

── Warning (test_end_to_end.R:276:11): aggregated pairing works ──────────────────────────────────────────────────────────────────────────────────
Unknown or uninitialised column: `row_0`.

── Warning (test_end_to_end.R:277:11): aggregated pairing works ──────────────────────────────────────────────────────────────────────────────────
Unknown or uninitialised column: `row_1`.

── Skipped tests  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
● empty test (2)

══ testthat results  ═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
Warning (test_end_to_end.R:276:11): aggregated pairing works
Warning (test_end_to_end.R:277:11): aggregated pairing works
Warning (test_end_to_end.R:276:11): aggregated pairing works
Warning (test_end_to_end.R:277:11): aggregated pairing works
Warning (test_end_to_end.R:276:11): aggregated pairing works
Warning (test_end_to_end.R:277:11): aggregated pairing works
Warning (test_end_to_end.R:276:11): aggregated pairing works
Warning (test_end_to_end.R:277:11): aggregated pairing works

[ FAIL 0 | WARN 8 | SKIP 2 | PASS 85 ]

What is more the simple_examples.R now runs w/o a hitch:

$ Rscript  simple_examples.R 

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

Attaching package: ‘gridExtra’

The following object is masked from ‘package:dplyr’:

    combine

Loading required package: reticulate
Loading required package: latex2exp
Loading required package: stringr
Loading required package: broom
Loading required package: car
Loading required package: carData

Attaching package: ‘car’

The following object is masked from ‘package:dplyr’:

    recode

Loading required package: lmtest
Loading required package: zoo

Attaching package: ‘zoo’

The following objects are masked from ‘package:base’:

    as.Date, as.Date.numeric

Loading required package: survival
                 change       beta  beta_pzse  beta_mzse prop_removed
1              original  0.9183526  0.9865258  0.8501794           NA
2                  sign -0.3595436 -0.2999995 -0.4190876       0.1875
3 sign and significance -0.5228577 -0.4641184 -0.5815969       0.2178
4          significance -0.2223906 -0.1616786 -0.2831025       0.1622
                 change        beta   beta_pzse  beta_mzse prop_removed
1              original  0.48217551  0.53516292  0.4291881           NA
2                  sign -0.17895735 -0.11675094 -0.2411638       0.0907
3 sign and significance -0.31502621 -0.24966329 -0.3803891       0.1129
4          significance -0.07877113 -0.01742793 -0.1401143       0.0725
                 change         beta    beta_pzse     beta_mzse prop_removed
1              original  0.092414667  0.160886232  0.0239431030           NA
2                  sign -0.001881734  0.065654834 -0.0694183030       0.0074
3 sign and significance -0.072743032 -0.005691437 -0.1397946283       0.0142
4          significance  0.067929299  0.136096415 -0.0002378169       0.0016
[1] 10000
# A tibble: 6 x 8
    arm group  se_grad beta_grad beta_pzse_grad beta_mzse_grad obs_per_row
  <int> <int>    <dbl>     <dbl>          <dbl>          <dbl>       <dbl>
1     0     1  4.33e-6   3.94e-4      0.000402       0.000385            6
2     0     2 -5.39e-6  -2.37e-4     -0.000248      -0.000227            7
3     0     3 -5.38e-6  -2.23e-4     -0.000234      -0.000213            4
4     0     4 -5.87e-6  -6.29e-5     -0.0000744     -0.0000514           3
5     0     5 -5.48e-6  -7.49e-4     -0.000760      -0.000739            7
6     0     6 -6.37e-6   2.35e-4      0.000222       0.000247            4
# … with 1 more variable: grouped_row <int>
# A tibble: 30 x 4
     arm group beta_grad grouped_row
   <int> <int>     <dbl>       <int>
 1     0    46  0.00224           46
 2     0    54  0.000213          54
 3     0   152  0.00160          152
 4     0   161  0.00157          161
 5     0   179  0.00249          179
 6     0   241  0.00213          241
 7     0   247  0.00106          247
 8     0   338  0.00180          337
 9     0   361  0.00240          360
10     0   495  0.00200          494
# … with 20 more rows

 0  1 
15 15 

  0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16  17  18  19 
205 199 197 172 203 209 193 191 187 203 187 205 182 195 209 178 206 195 197 191 
 20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39 
193 192 222 214 226 189 194 174 197 229 206 250 193 201 204 210 218 198 208 177 
 40  41  42  43  44  45  46  47  48  49 
202 181 184 224 218 189 212 199 179 213 
vcovCL:     0.047676
zaminfluence:   0.047676
                 change       beta   beta_pzse  beta_mzse prop_removed
1              original  0.9073391  1.00078136  0.8138968           NA
2                  sign -0.1733668  0.08850384 -0.4352375       0.0893
3 sign and significance -0.4166331 -0.11799088 -0.7152752       0.1226
4          significance  0.0441852  0.31560318 -0.2272328       0.0756

:man_shrugging: ???

Seems like a bit of a Heisenbug :-)

rgiordan commented 3 years ago

So strange! Though arguably no stranger than the fact that you were seeing the errors you saw in the first place.

Incidentally, I see you have some wheel errors on your Python install. Could you please put my mind at ease, and try to do a clean install & test run one more time, this time upgrading pip and installing wheel before installing? (Some venv versions seem to have wheel, some don't. These steps are also listed in the new "troubleshooting" section of the install instructions.)

python3 -m pip install --upgrade pip
python3 -m pip install wheel
python3 -m pip install -e regression_sensitivity

The skipped tests and warnings are normal, I see them too. (I'll make an issue to remind myself to clean those up, actually.)

jmcastagnetto commented 3 years ago

OK. I removed the pip cache (~/.cache/pip/), and then redid everything, including the python lines above:

jesus@aleph0:~/tmp/test$ git clone https://github.com/rgiordan/zaminfluence.git
Cloning into 'zaminfluence'...
remote: Enumerating objects: 286, done.
remote: Counting objects: 100% (286/286), done.
remote: Compressing objects: 100% (167/167), done.
remote: Total 286 (delta 135), reused 252 (delta 108), pack-reused 0
Receiving objects: 100% (286/286), 89.87 KiB | 216.00 KiB/s, done.
Resolving deltas: 100% (135/135), done.
jesus@aleph0:~/tmp/test$ cd zaminfluence/
✔ ~/tmp/test/zaminfluence [master {origin/master}|✔] 
23:51 $ export REPO=$(pwd)
✔ ~/tmp/test/zaminfluence [master {origin/master}|✔] 
23:51 $ 
✔ ~/tmp/test/zaminfluence [master {origin/master}|✔] 
23:51 $ python3 -m venv venv     
✔ ~/tmp/test/zaminfluence [master {origin/master}|✔] 
23:51 $ source venv/bin/activate  
(venv) ✔ ~/tmp/test/zaminfluence [master {origin/master}|✔] 
23:51 $ python3 -m pip install --upgrade pip
Collecting pip
  Downloading pip-20.3.3-py2.py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 342 kB/s 
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.0.2
    Uninstalling pip-20.0.2:
      Successfully uninstalled pip-20.0.2
Successfully installed pip-20.3.3
(venv) ✔ ~/tmp/test/zaminfluence [master {origin/master}|✔] 
23:52 $ python3 -m pip install wheel
Collecting wheel
  Downloading wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel
Successfully installed wheel-0.36.2
(venv) ✔ ~/tmp/test/zaminfluence [master {origin/master}|✔] 
23:52 $ python3 -m pip install -e regression_sensitivity
Obtaining file:///home/jesus/tmp/test/zaminfluence/regression_sensitivity
Collecting autograd>=1.3
  Downloading autograd-1.3.tar.gz (38 kB)
Collecting future>=0.15.2
  Downloading future-0.18.2.tar.gz (829 kB)
     |████████████████████████████████| 829 kB 1.0 MB/s 
Collecting numpy
  Downloading numpy-1.19.4-cp38-cp38-manylinux2010_x86_64.whl (14.5 MB)
     |████████████████████████████████| 14.5 MB 241 kB/s 
Collecting paragami>=0.42
  Downloading paragami-0.42-py3-none-any.whl (41 kB)
     |████████████████████████████████| 41 kB 6.3 kB/s 
Collecting vittles>=0.171
  Downloading vittles-0.173-py3-none-any.whl (26 kB)
Collecting ipykernel
  Downloading ipykernel-5.4.2-py3-none-any.whl (119 kB)
     |████████████████████████████████| 119 kB 1.2 MB/s 
Collecting ipython>=5.0.0
  Downloading ipython-7.19.0-py3-none-any.whl (784 kB)
     |████████████████████████████████| 784 kB 1.1 MB/s 
Requirement already satisfied: setuptools>=18.5 in ./venv/lib/python3.8/site-packages (from ipython>=5.0.0->ipykernel->regsens-rgiordandev==0.1) (44.0.0)
Collecting jedi>=0.10
  Downloading jedi-0.17.2-py2.py3-none-any.whl (1.4 MB)
     |████████████████████████████████| 1.4 MB 986 kB/s 
Collecting parso<0.8.0,>=0.7.0
  Downloading parso-0.7.1-py2.py3-none-any.whl (109 kB)
     |████████████████████████████████| 109 kB 806 kB/s 
Collecting pexpect>4.3
  Downloading pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
     |████████████████████████████████| 59 kB 216 kB/s 
Collecting prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0
  Downloading prompt_toolkit-3.0.8-py3-none-any.whl (355 kB)
     |████████████████████████████████| 355 kB 1.2 MB/s 
Collecting ptyprocess>=0.5
  Downloading ptyprocess-0.6.0-py2.py3-none-any.whl (39 kB)
Collecting tornado>=4.2
  Downloading tornado-6.1-cp38-cp38-manylinux2010_x86_64.whl (427 kB)
     |████████████████████████████████| 427 kB 1.7 MB/s 
Collecting traitlets>=4.1.0
  Downloading traitlets-5.0.5-py3-none-any.whl (100 kB)
     |████████████████████████████████| 100 kB 465 kB/s 
Collecting backcall
  Downloading backcall-0.2.0-py2.py3-none-any.whl (11 kB)
Collecting decorator
  Downloading decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Collecting ipython-genutils
  Downloading ipython_genutils-0.2.0-py2.py3-none-any.whl (26 kB)
Collecting jupyter-client
  Downloading jupyter_client-6.1.7-py3-none-any.whl (108 kB)
     |████████████████████████████████| 108 kB 983 kB/s 
Collecting jupyter-core>=4.6.0
  Downloading jupyter_core-4.7.0-py3-none-any.whl (82 kB)
     |████████████████████████████████| 82 kB 61 kB/s 
Collecting python-dateutil>=2.1
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
     |████████████████████████████████| 227 kB 1.5 MB/s 
Collecting pyzmq>=13
  Downloading pyzmq-20.0.0-cp38-cp38-manylinux1_x86_64.whl (1.1 MB)
     |████████████████████████████████| 1.1 MB 454 kB/s 
Collecting six>=1.5
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting pickleshare
  Downloading pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)
Collecting pygments
  Downloading Pygments-2.7.3-py3-none-any.whl (950 kB)
     |████████████████████████████████| 950 kB 2.1 MB/s 
Collecting pytest
  Downloading pytest-6.2.1-py3-none-any.whl (279 kB)
     |████████████████████████████████| 279 kB 1.1 MB/s 
Collecting attrs>=19.2.0
  Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)
     |████████████████████████████████| 49 kB 292 kB/s 
Collecting pluggy<1.0.0a1,>=0.12
  Downloading pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting py>=1.8.2
  Downloading py-1.10.0-py2.py3-none-any.whl (97 kB)
     |████████████████████████████████| 97 kB 374 kB/s 
Collecting iniconfig
  Downloading iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting packaging
  Downloading packaging-20.8-py2.py3-none-any.whl (39 kB)
Collecting pyparsing>=2.0.2
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
     |████████████████████████████████| 67 kB 265 kB/s 
Collecting scipy
  Downloading scipy-1.5.4-cp38-cp38-manylinux1_x86_64.whl (25.8 MB)
     |████████████████████████████████| 25.8 MB 223 kB/s 
Collecting toml
  Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting wcwidth
  Downloading wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Building wheels for collected packages: autograd, future
  Building wheel for autograd (setup.py) ... done
  Created wheel for autograd: filename=autograd-1.3-py3-none-any.whl size=47988 sha256=de90e5fb6e14d77108c41e9893d5452a2522cfb276a22fc096ee117e2647a357
  Stored in directory: /home/jesus/.cache/pip/wheels/85/f5/d2/3ef47d3a836b17620bf41647222825b065245862d12aa62885
  Building wheel for future (setup.py) ... done
  Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491059 sha256=0abba752cfe207e2ee3fec1ae47fb97faf6540e9b7c919cd181ed067aac63968
  Stored in directory: /home/jesus/.cache/pip/wheels/8e/70/28/3d6ccd6e315f65f245da085482a2e1c7d14b90b30f239e2cf4
Successfully built autograd future
Installing collected packages: pyparsing, ipython-genutils, wcwidth, traitlets, toml, six, py, ptyprocess, pluggy, parso, packaging, numpy, iniconfig, future, attrs, tornado, scipy, pyzmq, python-dateutil, pytest, pygments, prompt-toolkit, pickleshare, pexpect, jupyter-core, jedi, decorator, backcall, autograd, paragami, jupyter-client, ipython, vittles, ipykernel, regsens-rgiordandev
  Running setup.py develop for regsens-rgiordandev
Successfully installed attrs-20.3.0 autograd-1.3 backcall-0.2.0 decorator-4.4.2 future-0.18.2 iniconfig-1.1.1 ipykernel-5.4.2 ipython-7.19.0 ipython-genutils-0.2.0 jedi-0.17.2 jupyter-client-6.1.7 jupyter-core-4.7.0 numpy-1.19.4 packaging-20.8 paragami-0.42 parso-0.7.1 pexpect-4.8.0 pickleshare-0.7.5 pluggy-0.13.1 prompt-toolkit-3.0.8 ptyprocess-0.6.0 py-1.10.0 pygments-2.7.3 pyparsing-2.4.7 pytest-6.2.1 python-dateutil-2.8.1 pyzmq-20.0.0 regsens-rgiordandev scipy-1.5.4 six-1.15.0 toml-0.10.2 tornado-6.1 traitlets-5.0.5 vittles-0.173 wcwidth-0.2.5
(venv) ✔ ~/tmp/test/zaminfluence [master {origin/master}|✔] 
23:54 $ cd regression_sensitivity/
(venv) ✔ ~/tmp/test/zaminfluence/regression_sensitivity [master {origin/master}|✔] 
23:56 $ python3 -m pytest
============================================================== test session starts ===============================================================
platform linux -- Python 3.8.5, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /home/jesus/tmp/test/zaminfluence/regression_sensitivity
collected 2 items                                                                                                                                

regsens_rgiordandev/tests/test_basics.py ..                                                                                                [100%]

=============================================================== 2 passed in 1.35s ================================================================
(venv) ✔ ~/tmp/test/zaminfluence/regression_sensitivity [master {origin/master}|✔] 
23:56 $ cd ../zaminfluence/tests/
(venv) ✔ ~/tmp/test/zaminfluence/zaminfluence/tests [master {origin/master}|✔] 
23:56 $ ./testthat.R 
Loading required package: reticulate
Loading required package: dplyr

Attaching package: ‘dplyr’

The following object is masked from ‘package:testthat’:

    matches

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

Loading required package: ggplot2
Loading required package: latex2exp
Loading required package: stringr
Loading required package: broom
══ Skipped tests ═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
● empty test (2)

══ Warnings ══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
── Warning (test_end_to_end.R:276:11): aggregated pairing works ─────────────────────
Unknown or uninitialised column: `row_0`.
Backtrace:
 1. df[infl_df$row_0, ] test_end_to_end.R:276:10
 4. tibble:::`$.tbl_df`(infl_df, row_0)
── Warning (test_end_to_end.R:277:11): aggregated pairing works ─────────────────────
Unknown or uninitialised column: `row_1`.
Backtrace:
 1. df[infl_df$row_1, ] test_end_to_end.R:277:10
 4. tibble:::`$.tbl_df`(infl_df, row_1)
── Warning (test_end_to_end.R:276:11): aggregated pairing works ─────────────────────
Unknown or uninitialised column: `row_0`.
Backtrace:
 1. df[infl_df$row_0, ] test_end_to_end.R:276:10
 4. tibble:::`$.tbl_df`(infl_df, row_0)
── Warning (test_end_to_end.R:277:11): aggregated pairing works ─────────────────────
Unknown or uninitialised column: `row_1`.
Backtrace:
 1. df[infl_df$row_1, ] test_end_to_end.R:277:10
 4. tibble:::`$.tbl_df`(infl_df, row_1)
── Warning (test_end_to_end.R:276:11): aggregated pairing works ─────────────────────
Unknown or uninitialised column: `row_0`.
Backtrace:
 1. df[infl_df$row_0, ] test_end_to_end.R:276:10
 4. tibble:::`$.tbl_df`(infl_df, row_0)
── Warning (test_end_to_end.R:277:11): aggregated pairing works ─────────────────────
Unknown or uninitialised column: `row_1`.
Backtrace:
 1. df[infl_df$row_1, ] test_end_to_end.R:277:10
 4. tibble:::`$.tbl_df`(infl_df, row_1)
── Warning (test_end_to_end.R:276:11): aggregated pairing works ─────────────────────
Unknown or uninitialised column: `row_0`.
Backtrace:
 1. df[infl_df$row_0, ] test_end_to_end.R:276:10
 4. tibble:::`$.tbl_df`(infl_df, row_0)
── Warning (test_end_to_end.R:277:11): aggregated pairing works ─────────────────────
Unknown or uninitialised column: `row_1`.
Backtrace:
 1. df[infl_df$row_1, ] test_end_to_end.R:277:10
 4. tibble:::`$.tbl_df`(infl_df, row_1)

[ FAIL 0 | WARN 8 | SKIP 2 | PASS 85 ]

Wheel errors are gone, and all seems to be working. Now I have to figure out how to use this code effectively in my analysis :-)

Thanks for your help -- I think we can close this now.

rgiordan commented 3 years ago

I'm glad it worked out in the end! Please feel quite free to post another issue if you have more problems or questions. I'd really like this package to be useful, and I'll be happy to get feedback or provide help.