missing headeres error (but vowpal wabbit installed in the system)

[jgalanp]

Hello.

I'm trying to install the library with remotes::install_github("rvw-org/rvw") and I have already installed the vowpal wabbit library from source. However, I'm still getting the missing headers error but for just four header files: Missing headers in /usr/local/include/vowpalwabbit comp_io.h, ezexample.h, floatbits.h, v_hashmap.h

OS: Ubuntu 20.04 R: 4.1.0 Using master branch of vowpal wabbit repo (last release is 8.11 on 14th July)

Can someone provide some help? Is the package incompatible with the latest version of vowpal wabbit? Am I doing something wrong?

Thank you!

[eddelbuettel]

You can help by decomposing the high-level command:

• instead of install_remotes() get a tarball (from GitHub releases and unpack it, or by cloning the repo
• run what R CMD INSTALL rvw_*tar.gz run, namely ./configure (which here is a shell calling an Rscript)

When I do that (Ubuntu 21.04, R 4.1.1) I get

edd@rob:~/git/rvw(master)$ Rscript tools/r_configure.R 
Valid path: /usr/include/vowpalwabbit
edd@rob:~/git/rvw(master)$ 

Similarly, installation works (despite some compiler noise which I calm down via ~/.R/Makevars):

edd@rob:~/git/rvw(master)$ R CMD INSTALL rvw_0.6.0.tar.gz 
* installing to library ‘/usr/local/lib/R/site-library’
* installing *source* package ‘rvw’ ...
** using staged installation
Valid path: /usr/include/vowpalwabbit
** libs
ccache g++  -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -Iextra/ -I/usr/include/vowpalwabbit -I'/usr/local/lib/R/site-library/Rcpp/include' -I'/usr/local/lib/R/site-library/RApiSerialize/include'    -fpic  -g -O3 -Wall -pipe -pedantic -Wno-misleading-indentation -Wno-unused -Wno-ignored-attributes -Wno-class-memaccess -c RcppExports.cpp -o RcppExports.o
ccache g++  -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -Iextra/ -I/usr/include/vowpalwabbit -I'/usr/local/lib/R/site-library/Rcpp/include' -I'/usr/local/lib/R/site-library/RApiSerialize/include'    -fpic  -g -O3 -Wall -pipe -pedantic -Wno-misleading-indentation -Wno-unused -Wno-ignored-attributes -Wno-class-memaccess -c helpers.cpp -o helpers.o
ccache gcc -I"/usr/share/R/include" -DNDEBUG -Iextra/ -I/usr/include/vowpalwabbit -I'/usr/local/lib/R/site-library/Rcpp/include' -I'/usr/local/lib/R/site-library/RApiSerialize/include'    -fpic  -g -O3 -Wall -pipe -pedantic -std=gnu99 -c md5.c -o md5.o
ccache g++  -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -Iextra/ -I/usr/include/vowpalwabbit -I'/usr/local/lib/R/site-library/Rcpp/include' -I'/usr/local/lib/R/site-library/RApiSerialize/include'    -fpic  -g -O3 -Wall -pipe -pedantic -Wno-misleading-indentation -Wno-unused -Wno-ignored-attributes -Wno-class-memaccess -c rvw.cpp -o rvw.o
ccache g++ -std=gnu++11 -Wl,-S -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -flto=auto -Wl,-z,relro -o rvw.so RcppExports.o helpers.o md5.o rvw.o -lvw -L/usr/lib/R/lib -lR
installing to /usr/local/lib/R/site-library/00LOCK-rvw/00new/rvw/libs
** R
** demo
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (rvw)
edd@rob:~/git/rvw(master)$ 

So when you install VW 'from source' as you said you did you have to put it somewhere our tools/r_configure.r looks for it, and/or adjust tools/r_configure.r. You can see in the file where we look -- the standard spots.

[jgalanp]

Thank you for your help

Sorry, with 'install from source' I meant building from source following the steps in VW repo: https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Building

Knowing you are able to install it I have been trying again but without success. However, I am stuck in a different point. Regarding the missing header files, I think they are missing in VW repository. So could it be possible that you had already them in /usr/include/vowpalwabbit from previous installations? I have been looking for them in past versions in order to take the last version of each of them (from the last release they appear in) and this is the result:

master

git checkout master
root@kalgan:~/vowpal_wabbit# find ./ -type f \( -iname comp_io.h -o -iname ezexample.h -o -iname floatbits.h -o -iname v_hashmap.h \)
root@kalgan:~/vowpal_wabbit# 

8.11.0

root@kalgan:~/vowpal_wabbit# git checkout 8.11.0
Note: switching to '8.11.0'.
root@kalgan:~/vowpal_wabbit# find ./ -type f \( -iname comp_io.h -o -iname ezexample.h -o -iname floatbits.h -o -iname v_hashmap.h \)
./vowpalwabbit/ezexample.h

8.8.0

root@kalgan:~/vowpal_wabbit# git checkout 8.8.0
root@kalgan:~/vowpal_wabbit# find ./ -type f \( -iname comp_io.h -o -iname ezexample.h -o -iname floatbits.h -o -iname v_hashmap.h \)
./vowpalwabbit/comp_io.h
./vowpalwabbit/ezexample.h
./vowpalwabbit/v_hashmap.h

8.7.0 (the last release having all four files)

root@kalgan:~/vowpal_wabbit# git checkout 8.7.0
root@kalgan:~/vowpal_wabbit# find ./ -type f \( -iname comp_io.h -o -iname ezexample.h -o -iname floatbits.h -o -iname v_hashmap.h \)
./vowpalwabbit/comp_io.h
./vowpalwabbit/ezexample.h
./vowpalwabbit/v_hashmap.h
./vowpalwabbit/floatbits.h

So what I did now was perform a regular install of VW building from the source in master. Then I took the missing files from the above releases and copied them to /usr/local/include/vowpalwabbit (that is where VW gets installed in my system). This worked to pass the test:

root@kalgan:~/rvw-0.6.0# Rscript tools/r_configure.R
Valid path: /usr/local/include/vowpalwabbit

But now I get another install error:

root@kalgan:~# R CMD INSTALL v0.6.0.tar.gz 
Warning in untar2(tarfile, files, list, exdir, restore_times) :
  skipping pax global extended headers
* installing to library ‘/usr/local/lib/R/site-library’
* installing *source* package ‘rvw’ ...
** using staged installation
Valid path: /usr/local/include/vowpalwabbit
** libs
g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -Iextra/ -I/usr/local/include/vowpalwabbit -I'/usr/local/lib/R/site-library/Rcpp/include' -I'/usr/local/lib/R/site-library/RApiSerialize/include'    -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-tbZjLv/r-base-4.1.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c RcppExports.cpp -o RcppExports.o
g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -Iextra/ -I/usr/local/include/vowpalwabbit -I'/usr/local/lib/R/site-library/Rcpp/include' -I'/usr/local/lib/R/site-library/RApiSerialize/include'    -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-tbZjLv/r-base-4.1.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c helpers.cpp -o helpers.o
helpers.cpp: In function ‘int get_num_example(vw&)’:
helpers.cpp:151:18: error: invalid use of incomplete type ‘struct shared_data’
  151 |     return all.sd->example_number + all.sd->weighted_holdout_examples;
      |                  ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:151:43: error: invalid use of incomplete type ‘struct shared_data’
  151 |     return all.sd->example_number + all.sd->weighted_holdout_examples;
      |                                           ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp: In function ‘vw* setup_model(std::string)’:
helpers.cpp:180:15: error: ‘struct vw’ has no member named ‘quiet’
  180 |     if (!all->quiet && !all->bfgs && !all->searchstr && !all->opts_n_args.vm.count("audit_regressor"))
      |               ^~~~~
helpers.cpp:180:63: error: ‘struct vw’ has no member named ‘opts_n_args’
  180 |     if (!all->quiet && !all->bfgs && !all->searchstr && !all->opts_n_args.vm.count("audit_regressor"))
      |                                                               ^~~~~~~~~~~
helpers.cpp:183:47: error: incomplete type ‘shared_data’ used in nested name specifier
  183 |                     << std::setw(shared_data::col_avg_loss) << std::left << "average"
      |                                               ^~~~~~~~~~~~
helpers.cpp:185:47: error: incomplete type ‘shared_data’ used in nested name specifier
  185 |                     << std::setw(shared_data::col_since_last) << std::left << "since"
      |                                               ^~~~~~~~~~~~~~
helpers.cpp:188:47: error: incomplete type ‘shared_data’ used in nested name specifier
  188 |                     << std::setw(shared_data::col_example_counter) << "example"
      |                                               ^~~~~~~~~~~~~~~~~~~
helpers.cpp:190:47: error: incomplete type ‘shared_data’ used in nested name specifier
  190 |                     << std::setw(shared_data::col_example_weight) << "example"
      |                                               ^~~~~~~~~~~~~~~~~~
helpers.cpp:192:47: error: incomplete type ‘shared_data’ used in nested name specifier
  192 |                     << std::setw(shared_data::col_current_label) << "current"
      |                                               ^~~~~~~~~~~~~~~~~
helpers.cpp:194:47: error: incomplete type ‘shared_data’ used in nested name specifier
  194 |                     << std::setw(shared_data::col_current_predict) << "current"
      |                                               ^~~~~~~~~~~~~~~~~~~
helpers.cpp:196:47: error: incomplete type ‘shared_data’ used in nested name specifier
  196 |                     << std::setw(shared_data::col_current_features) << "current"
      |                                               ^~~~~~~~~~~~~~~~~~~~
helpers.cpp:199:47: error: incomplete type ‘shared_data’ used in nested name specifier
  199 |                     << std::setw(shared_data::col_avg_loss) << std::left << "loss"
      |                                               ^~~~~~~~~~~~
helpers.cpp:201:47: error: incomplete type ‘shared_data’ used in nested name specifier
  201 |                     << std::setw(shared_data::col_since_last) << std::left << "last"
      |                                               ^~~~~~~~~~~~~~
helpers.cpp:204:47: error: incomplete type ‘shared_data’ used in nested name specifier
  204 |                     << std::setw(shared_data::col_example_counter) << "counter"
      |                                               ^~~~~~~~~~~~~~~~~~~
helpers.cpp:206:47: error: incomplete type ‘shared_data’ used in nested name specifier
  206 |                     << std::setw(shared_data::col_example_weight) << "weight"
      |                                               ^~~~~~~~~~~~~~~~~~
helpers.cpp:208:47: error: incomplete type ‘shared_data’ used in nested name specifier
  208 |                     << std::setw(shared_data::col_current_label) << "label"
      |                                               ^~~~~~~~~~~~~~~~~
helpers.cpp:210:47: error: incomplete type ‘shared_data’ used in nested name specifier
  210 |                     << std::setw(shared_data::col_current_predict) << "predict"
      |                                               ^~~~~~~~~~~~~~~~~~~
helpers.cpp:212:47: error: incomplete type ‘shared_data’ used in nested name specifier
  212 |                     << std::setw(shared_data::col_current_features) << "features"
      |                                               ^~~~~~~~~~~~~~~~~~~~
helpers.cpp: In function ‘Rcpp::List get_eval(vw&)’:
helpers.cpp:222:30: error: invalid use of incomplete type ‘struct shared_data’
  222 |     int num_examples = all.sd->example_number;
      |                              ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:223:41: error: invalid use of incomplete type ‘struct shared_data’
  223 |     double weighted_example_sum = all.sd->weighted_examples();
      |                                         ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:224:39: error: invalid use of incomplete type ‘struct shared_data’
  224 |     double weighted_label_sum = all.sd->weighted_labels;
      |                                       ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:229:31: error: invalid use of incomplete type ‘struct shared_data’
  229 |     int total_feature = all.sd->total_features;
      |                               ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:232:19: error: invalid use of incomplete type ‘struct shared_data’
  232 |         if (all.sd->weighted_labeled_examples > 0) {
      |                   ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:233:30: error: invalid use of incomplete type ‘struct shared_data’
  233 |             avg_loss = all.sd->sum_loss / all.sd->weighted_labeled_examples;
      |                              ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:233:49: error: invalid use of incomplete type ‘struct shared_data’
  233 |             avg_loss = all.sd->sum_loss / all.sd->weighted_labeled_examples;
      |                                                 ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:237:22: error: invalid use of incomplete type ‘struct shared_data’
  237 |     } else if((all.sd->holdout_best_loss == FLT_MAX) || (all.sd->holdout_best_loss == FLT_MAX * 0.5)) {
      |                      ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:237:64: error: invalid use of incomplete type ‘struct shared_data’
  237 |     } else if((all.sd->holdout_best_loss == FLT_MAX) || (all.sd->holdout_best_loss == FLT_MAX * 0.5)) {
      |                                                                ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:240:26: error: invalid use of incomplete type ‘struct shared_data’
  240 |         avg_loss = all.sd->holdout_best_loss;
      |                          ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:242:15: error: invalid use of incomplete type ‘struct shared_data’
  242 |     if (all.sd->report_multiclass_log_loss)
      |               ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:245:45: error: invalid use of incomplete type ‘struct shared_data’
  245 |             avg_multiclass_log_loss = all.sd->multiclass_log_loss / all.sd->weighted_labeled_examples;
      |                                             ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:245:75: error: invalid use of incomplete type ‘struct shared_data’
  245 |             avg_multiclass_log_loss = all.sd->multiclass_log_loss / all.sd->weighted_labeled_examples;
      |                                                                           ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:247:45: error: invalid use of incomplete type ‘struct shared_data’
  247 |             avg_multiclass_log_loss = all.sd->holdout_multiclass_log_loss / all.sd->weighted_labeled_examples;
      |                                             ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:247:83: error: invalid use of incomplete type ‘struct shared_data’
  247 |             avg_multiclass_log_loss = all.sd->holdout_multiclass_log_loss / all.sd->weighted_labeled_examples;
      |                                                                                   ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp: In function ‘bool copy_get_best_constant(vw&, float&, float&)’:
helpers.cpp:270:15: error: invalid use of incomplete type ‘struct shared_data’
  270 |     if (all.sd->first_observed_label == FLT_MAX || // no non-test labels observed or function was never called
      |               ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:273:26: error: invalid use of incomplete type ‘struct shared_data’
  273 |     float label1 = all.sd->first_observed_label; // observed labels might be inside [sd->Min_label, sd->Max_label], so can't use Min/Max
      |                          ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:274:27: error: invalid use of incomplete type ‘struct shared_data’
  274 |     float label2 = (all.sd->second_observed_label == FLT_MAX)?0: all.sd->second_observed_label; // if only one label observed, second might be 0
      |                           ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:274:72: error: invalid use of incomplete type ‘struct shared_data’
  274 |     float label2 = (all.sd->second_observed_label == FLT_MAX)?0: all.sd->second_observed_label; // if only one label observed, second might be 0
      |                                                                        ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:282:37: error: invalid use of incomplete type ‘struct shared_data’
  282 |         label1_cnt = (float) (all.sd->weighted_labels - label2*all.sd->weighted_labeled_examples)/(label1 - label2);
      |                                     ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:282:70: error: invalid use of incomplete type ‘struct shared_data’
  282 |         label1_cnt = (float) (all.sd->weighted_labels - label2*all.sd->weighted_labeled_examples)/(label1 - label2);
      |                                                                      ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:283:35: error: invalid use of incomplete type ‘struct shared_data’
  283 |         label2_cnt = (float)all.sd->weighted_labeled_examples - label1_cnt;
      |                                   ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:291:5: error: ‘po’ has not been declared
  291 |     po::variables_map& vm = all.opts_n_args.vm;
      |     ^~
helpers.cpp:291:24: error: ‘vm’ was not declared in this scope; did you mean ‘vw’?
  291 |     po::variables_map& vm = all.opts_n_args.vm;
      |                        ^~
      |                        vw
helpers.cpp:291:33: error: ‘struct vw’ has no member named ‘opts_n_args’
  291 |     po::variables_map& vm = all.opts_n_args.vm;
      |                                 ^~~~~~~~~~~
helpers.cpp:295:54: error: expected primary-expression before ‘>’ token
  295 |         funcName = vm["loss_function"].as<std::string>();
      |                                                      ^
helpers.cpp:295:56: error: expected primary-expression before ‘)’ token
  295 |         funcName = vm["loss_function"].as<std::string>();
      |                                                        ^
helpers.cpp:300:39: error: invalid use of incomplete type ‘struct shared_data’
  300 |         best_constant = (float) all.sd->weighted_labels / (float) (all.sd->weighted_labeled_examples);
      |                                       ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:300:74: error: invalid use of incomplete type ‘struct shared_data’
  300 |         best_constant = (float) all.sd->weighted_labels / (float) (all.sd->weighted_labeled_examples);
      |                                                                          ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:301:20: error: invalid use of incomplete type ‘struct shared_data’
  301 |     else if (all.sd->is_more_than_two_labels_observed)
      |                    ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
helpers.cpp:330:41: error: expected primary-expression before ‘float’
  330 |             tau = vm["quantile_tau"].as<float>();
      |                                         ^~~~~
helpers.cpp:339:16: error: invalid use of incomplete type ‘struct shared_data’
  339 |     if (!all.sd->is_more_than_two_labels_observed)
      |                ^~
In file included from /usr/local/include/vowpalwabbit/global_data.h:35,
                 from /usr/local/include/vowpalwabbit/vw.h:29,
                 from helpers.cpp:1:
/usr/local/include/vowpalwabbit/loss_functions.h:8:8: note: forward declaration of ‘struct shared_data’
    8 | struct shared_data;
      |        ^~~~~~~~~~~
make: *** [/usr/lib/R/etc/Makeconf:177: helpers.o] Error 1
ERROR: compilation failed for package ‘rvw’
* removing ‘/usr/local/lib/R/site-library/rvw’

I don't know what am doing wrong. But in case this becomes too complex, I think I can start working with the command line or Python.

Thank you!

[eddelbuettel]

All of that is possible. I currently use

edd@rob:~$ dpkg -l | grep libvw | cut -c-80
ii  libvw-dev                                  8.6.1.dfsg1-1.1804.2             
ii  libvw0                                     8.6.1.dfsg1-1.1804.2             
edd@rob:~$ 

and while these are available in my flavour of Ubuntu they may not be for yours, you need to check.

edd@rob:~$ apt-cache policy libvw0
libvw0:
  Installed: 8.6.1.dfsg1-1.1804.2
  Candidate: 8.6.1.dfsg1-1.1804.2
  Version table:
 *** 8.6.1.dfsg1-1.1804.2 100
        100 /var/lib/dpkg/status
     8.6.1.dfsg1-1build3 500
        500 http://us.archive.ubuntu.com/ubuntu hirsute/universe amd64 Packages
edd@rob:~$ 

I may have 'ported' them for an older Ubuntu too -- I don't quite recall but the (local package) version number 1.1804.2 is a hint. So they may be in my Launchpad repo. Would that help?

Otherwise, the 'does not compile'. Such is life sometimes. The R package has been moderately inactive but if you're interested we could look into reviving it.

Are you aiming at particular VW versions or features?

[jgalanp]

Hello

I have been trying also installing the ubuntu repository version (8.6.1) without success. It is not something critical for me to work with R, it is just that some colleagues feel very comfortable in R but they can work out of it too. After your first response, the problem seemed simple so I kept trying. But with this compile error I think it is time to stop. If in the future something tries to solve this I will be happy to help testing installations in my system if it can be useful (I think that I can't do much more).

Thank you very much for your help.

[eddelbuettel]

Sure, whatever works for you, But what you just said makes no sense -- you must have wires crossed on your system, and (longer term) you will be better off figuring out how/why/where.

edd@rob:~/git/rvw(master)$ dkrrr rocker/r-ubuntu:20.04   # standard Ubuntu + R, dkrrr just a local docker alias
root@0ba25110ddd3:/work# apt update -qq
36 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@0ba25110ddd3:/work# apt install libvw-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libboost-program-options1.71.0 libvw0
The following NEW packages will be installed:
  libboost-program-options1.71.0 libvw-dev libvw0
0 upgraded, 3 newly installed, 0 to remove and 36 not upgraded.
Need to get 2,814 kB of archives.
After this operation, 25.0 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libboost-program-options1.71.0 amd64 1.71.0-6ubuntu6 [342 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal/universe amd64 libvw0 amd64 8.6.1.dfsg1-1build2 [849 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal/universe amd64 libvw-dev amd64 8.6.1.dfsg1-1build2 [1,623 kB]
Fetched 2,814 kB in 1s (2,274 kB/s)  
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libboost-program-options1.71.0:amd64.
(Reading database ... 20547 files and directories currently installed.)
Preparing to unpack .../libboost-program-options1.71.0_1.71.0-6ubuntu6_amd64.deb ...
Unpacking libboost-program-options1.71.0:amd64 (1.71.0-6ubuntu6) ...
Selecting previously unselected package libvw0.
Preparing to unpack .../libvw0_8.6.1.dfsg1-1build2_amd64.deb ...
Unpacking libvw0 (8.6.1.dfsg1-1build2) ...
Selecting previously unselected package libvw-dev.
Preparing to unpack .../libvw-dev_8.6.1.dfsg1-1build2_amd64.deb ...
Unpacking libvw-dev (8.6.1.dfsg1-1build2) ...
Setting up libboost-program-options1.71.0:amd64 (1.71.0-6ubuntu6) ...
Setting up libvw0 (8.6.1.dfsg1-1build2) ...
Setting up libvw-dev (8.6.1.dfsg1-1build2) ...
Processing triggers for libc-bin (2.31-0ubuntu9) ...
root@0ba25110ddd3:/work#
... short break here to install Rcpp + data.table + libboost-program-options-deve via apt, then RApiSerialize via R...
root@0ba25110ddd3:/work# R CMD INSTALL rvw_0.6.0.tar.gz 
* installing to library ‘/usr/local/lib/R/site-library’
* installing *source* package ‘rvw’ ...
** using staged installation
Valid path: /usr/include/vowpalwabbit
** libs
g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -Iextra/ -I/usr/include/vowpalwabbit -I'/usr/lib/R/site-library/Rcpp/include' -I'/usr/local/lib/R/site-library/RApiSerialize/include'    -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-QwogzP/r-base-4.1.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c RcppExports.cpp -o RcppExports.o
g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -Iextra/ -I/usr/include/vowpalwabbit -I'/usr/lib/R/site-library/Rcpp/include' -I'/usr/local/lib/R/site-library/RApiSerialize/include'    -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-QwogzP/r-base-4.1.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c helpers.cpp -o helpers.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -Iextra/ -I/usr/include/vowpalwabbit -I'/usr/lib/R/site-library/Rcpp/include' -I'/usr/local/lib/R/site-library/RApiSerialize/include'    -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-QwogzP/r-base-4.1.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c md5.c -o md5.o
g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -Iextra/ -I/usr/include/vowpalwabbit -I'/usr/lib/R/site-library/Rcpp/include' -I'/usr/local/lib/R/site-library/RApiSerialize/include'    -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-QwogzP/r-base-4.1.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c rvw.cpp -o rvw.o
g++ -std=gnu++11 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o rvw.so RcppExports.o helpers.o md5.o rvw.o -lvw -L/usr/lib/R/lib -lR
installing to /usr/local/lib/R/site-library/00LOCK-rvw/00new/rvw/libs
** R
** demo
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (rvw)
root@0ba25110ddd3:/work# 

So it can all work as claimed (once we install neeed pieces: three CRAN packages,one more Ubuntu -dev package).

[jgalanp]

Yes, you are right. Some 8.11 installation was still present. Sorry for this. After removing every previously installed version and installing only libvw-dev 8.6.1 again, I could finally install rvw.

Thanks again

[eddelbuettel]

Closing this as it wasn't "missing" headers per se. Package could of course benefit from a more powerful configuration script (contributions welcome) and/or updates to newer VW versions (as needed) but builds fine if and when VW headers are present (as per the referenced packages in Debian/Ubuntu).