search

samhelmholtz / skinny-dip

Code and supplementary material from the KDD 2016 paper "Skinny-dip: Clustering in a sea of noise" by Samuel Maurus and Claudia Plant.
15 stars 10 forks source link

problems with skinnyDipClusteringSubspace #2

Closed alexWhitworth closed 7 years ago

alexWhitworth commented 7 years ago

It's impossible to run the sparse version of skinny dip given the current code. The problem can be can be traced back to generateRegularSparseGrid, specifically the call to the temp folder filename <- sprintf("%s/%s",tempdir(),"sparse_grid_temp.csv"). This file does not exist outside your local computer and hasn't been shipped with this repo.

It is suggested that you save data files in with your packages as .Rda files instead of as raw csv's which need to be imported. See Hadley's suggested handling for instance.

Similarly, you could replace getSparseGridSizes with an .Rda file and a call to data("sgsizes") instead of getSparseGridSizes

samhelmholtz commented 7 years ago

Many thanks for taking the time to report this Alex. I'll have a look at it in a few weeks when I'm back from holidays. Sam.

alexWhitworth commented 7 years ago

@samhelmholtz Great!

FYI-- I am interested in getting this work into a formal R package not just as a source(...) file. Please let me know if you'd be interested in collaborating on this or would rather I just fork it into my own Github account and attribute the work appropriately.

samhelmholtz commented 7 years ago

@alexWhitworth Great to hear that you're interested in furthering SkinnyDip! From my side things are a bit up in the air at the moment. Having recently submitted my PhD thesis, I am now preparing my defense (taking place in March/April) and organizing my post-PhD adventures. Given this I don't think I'll be able to dedicate much time to this project at the moment, so if you're interested I'd invite you to fork it as you suggested.

I will however try to fix this issue that you described above in the next few days though.

samhelmholtz commented 7 years ago

@alexWhitworth Just having a closer look at your original comment above. Unless I'm missing something, the line...

filename <- sprintf("%s/%s",tempdir(),"sparse_grid_temp.csv")

...just generates a filename in a temporary folder on the local machine. On my Linux machine I get a result like "/tmp/RtmpdfbYEG/sparse_grid_temp.csv". The following line simply uses that string as the destination filename to dump the location of the sparse grid nodes:

system(sprintf("cd sgpp/examples;python python_simple.py %d %d \"%s\";",d,l,filename))

Here the python_simple.py call just creates a sparse grid of the given dimensionality and level (d, l) and writing the results to "filename", which is the temporary filename we created above. The next line then just reads in the result from the file that the python utility created for us:

sparseGridDataFrame <- read.csv(filename,header=FALSE)

Given this, I don't quite understand what you mean when you're suggesting that "sparse_grid_temp.csv" needs to be included in this repo. The contents of this file are created on-demand when generating a sparse grid, so unless I'm missing something, it wouldn't make sense to include it in the repo (its purpose is just to be a short-lived file to aid in getting the sparse grid points from python into R).

I hope my comments have made sense. Please let me know what you think. Happy to continue discussing this until you get it to work for you.

alexWhitworth commented 7 years ago

It's possible I traced this incorrectly 3 weeks ago. Here's a reproducible error.

library(diptest)
source("../skinnydip/func.R")
data("iris")
c_mat <- as.matrix(iris[,1:4])
clust1 <- skinnyDipClusteringSubspace(c_mat) 
sh: line 0: cd: sgpp/examples: No such file or directory
python: can't open file 'python_simple.py': [Errno 2] No such file or directory
In addition: Warning message:
In file(file, "rt") :
  cannot open file '/tmp/Rtmp51VR6D/sparse_grid_temp.csv': No such file or directory

trace:

7 file(file, "rt") 
6 read.table(file = file, header = header, sep = sep, quote = quote, 
    dec = dec, fill = fill, comment.char = comment.char, ...) 
5 read.csv(filename, header = FALSE) at func.R#456
4 generateRegularSparseGrid(sparseGridDimension, sparseGridLevel) at func.R#508
3 findDipPursuitStartingPoint(dataInOrthComplement, NULL, NULL, 
    NULL, numSparseGridPoints, debug) at func.R#807
2 findOptimalSubspaceForClustering(t(data), numSparseGridPoints, 
    significanceLevel, debug) at func.R#1078
1 skinnyDipClusteringSubspace(c_mat)

sessionInfo()


R version 3.3.2 (2016-10-31)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] diptest_0.75-7

loaded via a namespace (and not attached): [1] rsconnect_0.5 tools_3.3.2

samhelmholtz commented 7 years ago

Thanks for the additional info Alex. I might be wrong, but it looks like you might not have built the sgpp (sparse grids) code. From the README_WEB.html:

Importantly, in order to call skinnyDipClusteringSubspace, you must build the sparse grids "sgpp" code in the directory $location$/code/skinny-dip/sgpp. Do this by going into that directory and calling "scons" (SConstruct) from bash.

Have you completed this step already?

alexWhitworth commented 7 years ago

Aha. Yes, I believe that I missed this in your README. But I'm unable to get sgpp to install properly currently. If you have any suggestions, they'd be appreciated. The installation guide doesn't have much in the way of troubleshooting

`~/R/skinnydip/sgpp $scons scons: Reading SConscript files ... Using default gcc Checking for C++ library m... yes Checking for swig...(cached) Error: swig cannot be found. Check PATH environment variable! Checking for C++ header file Python.h... yes Checking for javac...(cached) /usr/local/bin/javac Checking for java...(cached) /usr/local/bin/java Checking for C++ header file jni.h... no Trying to locate jni.h... ... JNI_CPPINCLUDE not set ... JAVA_HOME not set ... trying to guess ... found in /usr/local/fbprojects/packages/fb-jdk-8/6/bin/../include finished configuration

scons: warning: The build_dir keyword has been deprecated; use the variant_dir keyword instead. File "/home/alexwhitworth/R/skinnydip/sgpp/SConstruct", line 299, in

scons: *** Import of non-existent variable ''pysgpp'' File "/home/alexwhitworth/R/skinnydip/sgpp/tests/SConscript", line 7, in `

samhelmholtz commented 7 years ago

Ok, sounds like we're getting closer! Based on your dump, the first error I can see is that swig isn't found. Open a terminal and type:

swig -version

You should get an output like this...

SWIG Version 3.0.8

Compiled with g++ [x86_64-pc-linux-gnu]

Configured options: +pcre

Please see http://www.swig.org for reporting bugs and further information

If not, it probably means you haven't got it installed. You can try installing it with:

sudo apt-get install swig

As you said, the SGPP library isn't mine, so I'm not an expert on its installation process, but that seems to me to be a good place to start.

alexWhitworth commented 7 years ago

Closer indeed :). With swig and spgg installed:

library(diptest)
source("./func.R")
data("iris")
c_mat <- as.matrix(iris[,1:4])
clust1 <- skinnyDipClusteringSubspace(c_mat) 
dimensionality:         3
number of grid points:  1023
Error: could not find function "Null"

Stepping into the debugger:

# Fails: 
subspace <- findOptimalSubspaceForClustering(t(data),numSparseGridPoints, significanceLevel,debug)

Further into the debugger:

# debug at ./func.R#840: nullSpace <- Null(subspace)
# fails      
    ...
    nullSpace <- Null(subspace)
    ...
samhelmholtz commented 7 years ago

The function "Null" is for finding the null space of a matrix. It is documented here: http://stat.ethz.ch/R-manual/R-devel/library/MASS/html/Null.html

It looks like it can't find the function, which makes me think you haven't loaded/installed the MASS package.

From README_WEB.html:

make sure you have the following packages installed from CRAN:

MASS (for calculating the null space of a matrix, relevant for SparseDip), ...

You can install these packages using the R command e.g. install.packages("MASS"). ... Once you have the packages installed and loaded (i.e. library(diptest) to load a package...

Could you check to see if you have MASS loaded? You can do so with sessionInfo(), which should output something like:

other attached packages: [1] MASS_7.3-45

If you don't see that it's loaded, install it (if not already installed) and load it with the commands in the quote above, then try again :).

alexWhitworth commented 7 years ago

I can't recall that session. Undoubtedly MASS was installed, but perhaps not loaded. In any case, rerunning the same code from a fresh session in R with an explicit load of MASS works. Thank you for your patience with this!

library(diptest); library(MASS)
source("../skinnydip/func.R")
data("iris")
c_mat <- as.matrix(iris[,1:4])
clust1 <- skinnyDipClusteringSubspace(c_mat) # works

That said, please let me know if I'm interpreting the return clust1 accurately. From examining the code: (Is the below correct?)

I'm only not closing this issue pending your response to this interpretation question. Otherwise, thanks for your help and patience!

samhelmholtz commented 7 years ago

Great to hear that things are working now...thanks for your patience as well (I concede that this is "research-grade" code and requires some time to understand the setup requirements).

Your comments about the nature of the returned list structure from the skinnyDipClusteringSubspace function are correct. The relevant line in func.R is

return(list(subspace,labels,dataInSubspace))

Note also the computation of dataInSubspace (just the projection of the data onto the subspace):

dataInSubspace <- data %*% subspace

From a "black box clustering" perspective it's probably the "labels" that are the most important: they give the group (cluster) labels as you correctly mentioned.

Based on your comments I'll close this issue now. If my answer here missed something, feel free to reopen it.

alexWhitworth commented 7 years ago

@samhelmholtz installing on a new machine, and getting the below error after installing sg++

I spoke with both a production engineer and a software engineer and they're pretty sure that what's going on is a dependency issue due to differences in versioning. Can you please let me know exactly what the setup / versions were for the required software to get this to run so that I can setup my system identically to avoid any changes in dependency structure?

I have the following installed:

Linux CentOS
Python 2.7.5
SWIG Version 3.0.12
sipp 3.3.990
scons 2.5.1

Please advise, thanks

Error below here:

~/R/skinnydip/sgpp $scons

scons: Reading SConscript files ...
Using default gcc
Checking for C++ library m... (cached) yes
Checking for swig...(cached) /usr/local/bin/swig
Checking for C++ header file Python.h... (cached) yes
Checking for javac...(cached) /usr/local/bin/javac
Checking for java...(cached) /usr/local/bin/java
Checking for C++ header file jni.h... (cached) no
Trying to locate jni.h...
... JNI_CPPINCLUDE not set
... JAVA_HOME not set
... trying to guess
... found in /usr/local/fbprojects/packages/fb-jdk-8/7/bin/../include
finished configuration

scons: warning: The build_dir keyword has been deprecated; use the variant_dir keyword instead.
File "/home/alexwhitworth/R/skinnydip/sgpp/SConstruct", line 299, in <module>

scons: warning: The build_dir keyword has been deprecated; use the variant_dir keyword instead.
File "/home/alexwhitworth/R/skinnydip/sgpp/SConstruct", line 306, in <module>
scons: done reading SConscript files.
scons: Building targets ...
g++ -o tmp/build_pysgpp/pysgpp_wrap.os -c -fPIC -Wall -ansi -pedantic -Wno-long-long -fno-strict-aliasing -O3 -funroll-loops -ffloat-store -Wno-deprecated -I/usr/include/python2.7 -I/usr/local/fbprojects/packages/fb-jdk-8/7/include -I/usr/local/fbprojects/packages/fb-jdk-8/7/include/linux -Isrc/sgpp tmp/build_pysgpp/pysgpp_wrap.cc
In file included from src/sgpp/algorithm/datadriven/AlgorithmDGEMV.hpp:15:0,
                 from src/sgpp/sgpp.hpp:14,
                 from tmp/build_pysgpp/pysgpp_wrap.cc:5765:
src/sgpp/algorithm/common/GetAffectedBasisFunctions.hpp: In member function ‘void sg::GetAffectedBasisFunctions<BASIS>::operator()(BASIS&, std::vector<double, std::allocator<double> >&, std::vector<std::pair<long unsigned int, double>, std::allocator<std::pair<long unsigned int, double> > >&)’:
src/sgpp/algorithm/common/GetAffectedBasisFunctions.hpp:75:53: warning: typedef ‘level_type’ locally defined but not used [-Wunused-local-typedefs]
         typedef GridStorage::index_type::level_type level_type;
                                                     ^
tmp/build_pysgpp/pysgpp_wrap.cc: In function ‘PyObject* _wrap_GridStorage_find(PyObject*, PyObject*)’:
tmp/build_pysgpp/pysgpp_wrap.cc:21360:149: error: ‘eqIndex’ was not declared in this scope
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                     ^
tmp/build_pysgpp/pysgpp_wrap.cc:21360:149: note: suggested alternative:
In file included from src/sgpp/grid/storage/hashmap/HashGridIndex.hpp:11:0,
                 from src/sgpp/grid/GridStorage.hpp:11,
                 from src/sgpp/sgpp.hpp:11,
                 from tmp/build_pysgpp/pysgpp_wrap.cc:5765:
src/sgpp/common/hash_map_config.hpp:65:9: note:   ‘sg::eqIndex’
  struct eqIndex { };
         ^
tmp/build_pysgpp/pysgpp_wrap.cc:21360:207: error: template argument 4 is invalid
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                                                                               ^
tmp/build_pysgpp/pysgpp_wrap.cc:21360:209: error: template argument 1 is invalid
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                                                                                 ^
tmp/build_pysgpp/pysgpp_wrap.cc:21360:221: error: expected initializer before ‘>’ token
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                                                                                             ^
tmp/build_pysgpp/pysgpp_wrap.cc:21375:7: error: ‘result’ was not declared in this scope
       result = (arg1)->find(arg2);
       ^
tmp/build_pysgpp/pysgpp_wrap.cc:21380:239: error: ‘result’ was not declared in this scope
   resultobj = SWIG_NewPointerObj((new sg::HashGridStorage< sg::HashGridIndex< unsigned int,unsigned int > >::grid_map_iterator(static_cast< const sg::HashGridStorage< sg::HashGridIndex< unsigned int,unsigned int > >::grid_map_iterator& >(result))), SWIGTYPE_p_std__hash_mapT_sg__HashGridIndexT_unsigned_int_unsigned_int_t_p_size_t_hashT_sg__HashGridIndexT_unsigned_int_unsigned_int_t_p_t_eqIndexT_sg__HashGridIndexT_unsigned_int_unsigned_int_t_p_t_t__iterator, SWIG_POINTER_OWN |  0 );
                                                                                                                                                                                                                                               ^
tmp/build_pysgpp/pysgpp_wrap.cc:1188:89: note: in definition of macro ‘SWIG_NewPointerObj’
 #define SWIG_NewPointerObj(ptr, type, flags)            SWIG_Python_NewPointerObj(NULL, ptr, type, flags)
                                                                                         ^
tmp/build_pysgpp/pysgpp_wrap.cc: In function ‘PyObject* _wrap_GridStorage_begin(PyObject*, PyObject*)’:
tmp/build_pysgpp/pysgpp_wrap.cc:21393:149: error: ‘eqIndex’ was not declared in this scope
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                     ^
tmp/build_pysgpp/pysgpp_wrap.cc:21393:149: note: suggested alternative:
In file included from src/sgpp/grid/storage/hashmap/HashGridIndex.hpp:11:0,
                 from src/sgpp/grid/GridStorage.hpp:11,
                 from src/sgpp/sgpp.hpp:11,
                 from tmp/build_pysgpp/pysgpp_wrap.cc:5765:
src/sgpp/common/hash_map_config.hpp:65:9: note:   ‘sg::eqIndex’
  struct eqIndex { };
         ^
tmp/build_pysgpp/pysgpp_wrap.cc:21393:207: error: template argument 4 is invalid
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                                                                               ^
tmp/build_pysgpp/pysgpp_wrap.cc:21393:209: error: template argument 1 is invalid
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                                                                                 ^
tmp/build_pysgpp/pysgpp_wrap.cc:21393:221: error: expected initializer before ‘>’ token
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                                                                                             ^
tmp/build_pysgpp/pysgpp_wrap.cc:21403:7: error: ‘result’ was not declared in this scope
       result = (arg1)->begin();
       ^
tmp/build_pysgpp/pysgpp_wrap.cc:21408:239: error: ‘result’ was not declared in this scope
   resultobj = SWIG_NewPointerObj((new sg::HashGridStorage< sg::HashGridIndex< unsigned int,unsigned int > >::grid_map_iterator(static_cast< const sg::HashGridStorage< sg::HashGridIndex< unsigned int,unsigned int > >::grid_map_iterator& >(result))), SWIGTYPE_p_std__hash_mapT_sg__HashGridIndexT_unsigned_int_unsigned_int_t_p_size_t_hashT_sg__HashGridIndexT_unsigned_int_unsigned_int_t_p_t_eqIndexT_sg__HashGridIndexT_unsigned_int_unsigned_int_t_p_t_t__iterator, SWIG_POINTER_OWN |  0 );
                                                                                                                                                                                                                                               ^
tmp/build_pysgpp/pysgpp_wrap.cc:1188:89: note: in definition of macro ‘SWIG_NewPointerObj’
 #define SWIG_NewPointerObj(ptr, type, flags)            SWIG_Python_NewPointerObj(NULL, ptr, type, flags)
                                                                                         ^
tmp/build_pysgpp/pysgpp_wrap.cc: In function ‘PyObject* _wrap_GridStorage_end__SWIG_0(PyObject*, PyObject*)’:
tmp/build_pysgpp/pysgpp_wrap.cc:21421:149: error: ‘eqIndex’ was not declared in this scope
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                     ^
tmp/build_pysgpp/pysgpp_wrap.cc:21421:149: note: suggested alternative:
In file included from src/sgpp/grid/storage/hashmap/HashGridIndex.hpp:11:0,
                 from src/sgpp/grid/GridStorage.hpp:11,
                 from src/sgpp/sgpp.hpp:11,
                 from tmp/build_pysgpp/pysgpp_wrap.cc:5765:
src/sgpp/common/hash_map_config.hpp:65:9: note:   ‘sg::eqIndex’
  struct eqIndex { };
         ^
tmp/build_pysgpp/pysgpp_wrap.cc:21421:207: error: template argument 4 is invalid
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                                                                               ^
tmp/build_pysgpp/pysgpp_wrap.cc:21421:209: error: template argument 1 is invalid
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                                                                                 ^
tmp/build_pysgpp/pysgpp_wrap.cc:21421:221: error: expected initializer before ‘>’ token
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                                                                                             ^
tmp/build_pysgpp/pysgpp_wrap.cc:21431:7: error: ‘result’ was not declared in this scope
       result = (arg1)->end();
       ^
tmp/build_pysgpp/pysgpp_wrap.cc:21436:239: error: ‘result’ was not declared in this scope
   resultobj = SWIG_NewPointerObj((new sg::HashGridStorage< sg::HashGridIndex< unsigned int,unsigned int > >::grid_map_iterator(static_cast< const sg::HashGridStorage< sg::HashGridIndex< unsigned int,unsigned int > >::grid_map_iterator& >(result))), SWIGTYPE_p_std__hash_mapT_sg__HashGridIndexT_unsigned_int_unsigned_int_t_p_size_t_hashT_sg__HashGridIndexT_unsigned_int_unsigned_int_t_p_t_eqIndexT_sg__HashGridIndexT_unsigned_int_unsigned_int_t_p_t_t__iterator, SWIG_POINTER_OWN |  0 );
                                                                                                                                                                                                                                               ^
tmp/build_pysgpp/pysgpp_wrap.cc:1188:89: note: in definition of macro ‘SWIG_NewPointerObj’
 #define SWIG_NewPointerObj(ptr, type, flags)            SWIG_Python_NewPointerObj(NULL, ptr, type, flags)
                                                                                         ^
scons: *** [tmp/build_pysgpp/pysgpp_wrap.os] Error 1
scons: building terminated because of errors.
alexwhitworth@devvm31820:~/R/skinnydip/sgpp  $sudo scons
scons: Reading SConscript files ...
Using default gcc
Checking for C++ library m... (cached) yes
Checking for swig...(cached) /usr/local/bin/swig
Checking for C++ header file Python.h... (cached) yes
Checking for javac...(cached) /usr/local/bin/javac
Checking for java...(cached) /usr/local/bin/java
Checking for C++ header file jni.h... (cached) no
Trying to locate jni.h...
... JNI_CPPINCLUDE not set
... JAVA_HOME not set
... trying to guess
... found in /usr/local/fbprojects/packages/fb-jdk-8/7/bin/../include
finished configuration

scons: warning: The build_dir keyword has been deprecated; use the variant_dir keyword instead.
File "/home/alexwhitworth/R/skinnydip/sgpp/SConstruct", line 299, in <module>

scons: warning: The build_dir keyword has been deprecated; use the variant_dir keyword instead.
File "/home/alexwhitworth/R/skinnydip/sgpp/SConstruct", line 306, in <module>
scons: done reading SConscript files.
scons: Building targets ...
g++ -o tmp/build_pysgpp/pysgpp_wrap.os -c -fPIC -Wall -ansi -pedantic -Wno-long-long -fno-strict-aliasing -O3 -funroll-loops -ffloat-store -Wno-deprecated -I/usr/include/python2.7 -I/usr/local/fbprojects/packages/fb-jdk-8/7/include -I/usr/local/fbprojects/packages/fb-jdk-8/7/include/linux -Isrc/sgpp tmp/build_pysgpp/pysgpp_wrap.cc
In file included from src/sgpp/algorithm/datadriven/AlgorithmDGEMV.hpp:15:0,
                 from src/sgpp/sgpp.hpp:14,
                 from tmp/build_pysgpp/pysgpp_wrap.cc:5765:
src/sgpp/algorithm/common/GetAffectedBasisFunctions.hpp: In member function ‘void sg::GetAffectedBasisFunctions<BASIS>::operator()(BASIS&, std::vector<double, std::allocator<double> >&, std::vector<std::pair<long unsigned int, double>, std::allocator<std::pair<long unsigned int, double> > >&)’:
src/sgpp/algorithm/common/GetAffectedBasisFunctions.hpp:75:53: warning: typedef ‘level_type’ locally defined but not used [-Wunused-local-typedefs]
         typedef GridStorage::index_type::level_type level_type;
                                                     ^
tmp/build_pysgpp/pysgpp_wrap.cc: In function ‘PyObject* _wrap_GridStorage_find(PyObject*, PyObject*)’:
tmp/build_pysgpp/pysgpp_wrap.cc:21360:149: error: ‘eqIndex’ was not declared in this scope
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                     ^
tmp/build_pysgpp/pysgpp_wrap.cc:21360:149: note: suggested alternative:
In file included from src/sgpp/grid/storage/hashmap/HashGridIndex.hpp:11:0,
                 from src/sgpp/grid/GridStorage.hpp:11,
                 from src/sgpp/sgpp.hpp:11,
                 from tmp/build_pysgpp/pysgpp_wrap.cc:5765:
src/sgpp/common/hash_map_config.hpp:65:9: note:   ‘sg::eqIndex’
  struct eqIndex { };
         ^
tmp/build_pysgpp/pysgpp_wrap.cc:21360:207: error: template argument 4 is invalid
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                                                                               ^
tmp/build_pysgpp/pysgpp_wrap.cc:21360:209: error: template argument 1 is invalid
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                                                                                 ^
tmp/build_pysgpp/pysgpp_wrap.cc:21360:221: error: expected initializer before ‘>’ token
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                                                                                             ^
tmp/build_pysgpp/pysgpp_wrap.cc:21375:7: error: ‘result’ was not declared in this scope
       result = (arg1)->find(arg2);
       ^
tmp/build_pysgpp/pysgpp_wrap.cc:21380:239: error: ‘result’ was not declared in this scope
   resultobj = SWIG_NewPointerObj((new sg::HashGridStorage< sg::HashGridIndex< unsigned int,unsigned int > >::grid_map_iterator(static_cast< const sg::HashGridStorage< sg::HashGridIndex< unsigned int,unsigned int > >::grid_map_iterator& >(result))), SWIGTYPE_p_std__hash_mapT_sg__HashGridIndexT_unsigned_int_unsigned_int_t_p_size_t_hashT_sg__HashGridIndexT_unsigned_int_unsigned_int_t_p_t_eqIndexT_sg__HashGridIndexT_unsigned_int_unsigned_int_t_p_t_t__iterator, SWIG_POINTER_OWN |  0 );
                                                                                                                                                                                                                                               ^
tmp/build_pysgpp/pysgpp_wrap.cc:1188:89: note: in definition of macro ‘SWIG_NewPointerObj’
 #define SWIG_NewPointerObj(ptr, type, flags)            SWIG_Python_NewPointerObj(NULL, ptr, type, flags)
                                                                                         ^
tmp/build_pysgpp/pysgpp_wrap.cc: In function ‘PyObject* _wrap_GridStorage_begin(PyObject*, PyObject*)’:
tmp/build_pysgpp/pysgpp_wrap.cc:21393:149: error: ‘eqIndex’ was not declared in this scope
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                     ^
tmp/build_pysgpp/pysgpp_wrap.cc:21393:149: note: suggested alternative:
In file included from src/sgpp/grid/storage/hashmap/HashGridIndex.hpp:11:0,
                 from src/sgpp/grid/GridStorage.hpp:11,
                 from src/sgpp/sgpp.hpp:11,
                 from tmp/build_pysgpp/pysgpp_wrap.cc:5765:
src/sgpp/common/hash_map_config.hpp:65:9: note:   ‘sg::eqIndex’
  struct eqIndex { };
         ^
tmp/build_pysgpp/pysgpp_wrap.cc:21393:207: error: template argument 4 is invalid
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                                                                               ^
tmp/build_pysgpp/pysgpp_wrap.cc:21393:209: error: template argument 1 is invalid
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                                                                                 ^
tmp/build_pysgpp/pysgpp_wrap.cc:21393:221: error: expected initializer before ‘>’ token
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                                                                                             ^
tmp/build_pysgpp/pysgpp_wrap.cc:21403:7: error: ‘result’ was not declared in this scope
       result = (arg1)->begin();
       ^
tmp/build_pysgpp/pysgpp_wrap.cc:21408:239: error: ‘result’ was not declared in this scope
   resultobj = SWIG_NewPointerObj((new sg::HashGridStorage< sg::HashGridIndex< unsigned int,unsigned int > >::grid_map_iterator(static_cast< const sg::HashGridStorage< sg::HashGridIndex< unsigned int,unsigned int > >::grid_map_iterator& >(result))), SWIGTYPE_p_std__hash_mapT_sg__HashGridIndexT_unsigned_int_unsigned_int_t_p_size_t_hashT_sg__HashGridIndexT_unsigned_int_unsigned_int_t_p_t_eqIndexT_sg__HashGridIndexT_unsigned_int_unsigned_int_t_p_t_t__iterator, SWIG_POINTER_OWN |  0 );
                                                                                                                                                                                                                                               ^
tmp/build_pysgpp/pysgpp_wrap.cc:1188:89: note: in definition of macro ‘SWIG_NewPointerObj’
 #define SWIG_NewPointerObj(ptr, type, flags)            SWIG_Python_NewPointerObj(NULL, ptr, type, flags)
                                                                                         ^
tmp/build_pysgpp/pysgpp_wrap.cc: In function ‘PyObject* _wrap_GridStorage_end__SWIG_0(PyObject*, PyObject*)’:
tmp/build_pysgpp/pysgpp_wrap.cc:21421:149: error: ‘eqIndex’ was not declared in this scope
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                     ^
tmp/build_pysgpp/pysgpp_wrap.cc:21421:149: note: suggested alternative:
In file included from src/sgpp/grid/storage/hashmap/HashGridIndex.hpp:11:0,
                 from src/sgpp/grid/GridStorage.hpp:11,
                 from src/sgpp/sgpp.hpp:11,
                 from tmp/build_pysgpp/pysgpp_wrap.cc:5765:
src/sgpp/common/hash_map_config.hpp:65:9: note:   ‘sg::eqIndex’
  struct eqIndex { };
         ^
tmp/build_pysgpp/pysgpp_wrap.cc:21421:207: error: template argument 4 is invalid
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                                                                               ^
tmp/build_pysgpp/pysgpp_wrap.cc:21421:209: error: template argument 1 is invalid
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                                                                                 ^
tmp/build_pysgpp/pysgpp_wrap.cc:21421:221: error: expected initializer before ‘>’ token
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                                                                                             ^
tmp/build_pysgpp/pysgpp_wrap.cc:21431:7: error: ‘result’ was not declared in this scope
       result = (arg1)->end();
       ^
tmp/build_pysgpp/pysgpp_wrap.cc:21436:239: error: ‘result’ was not declared in this scope
   resultobj = SWIG_NewPointerObj((new sg::HashGridStorage< sg::HashGridIndex< unsigned int,unsigned int > >::grid_map_iterator(static_cast< const sg::HashGridStorage< sg::HashGridIndex< unsigned int,unsigned int > >::grid_map_iterator& >(result))), SWIGTYPE_p_std__hash_mapT_sg__HashGridIndexT_unsigned_int_unsigned_int_t_p_size_t_hashT_sg__HashGridIndexT_unsigned_int_unsigned_int_t_p_t_eqIndexT_sg__HashGridIndexT_unsigned_int_unsigned_int_t_p_t_t__iterator, SWIG_POINTER_OWN |  0 );
                                                                                                                                                                                                                                               ^
tmp/build_pysgpp/pysgpp_wrap.cc:1188:89: note: in definition of macro ‘SWIG_NewPointerObj’
 #define SWIG_NewPointerObj(ptr, type, flags)            SWIG_Python_NewPointerObj(NULL, ptr, type, flags)
                                                                                         ^
scons: *** [tmp/build_pysgpp/pysgpp_wrap.os] Error 1
scons: building terminated because of errors.
samhelmholtz commented 7 years ago

Hi Alex,

The first error in the output is:

tmp/build_pysgpp/pysgpp_wrap.cc:21360:149: error: ‘eqIndex’ was not declared in this scope
   SwigValueWrapper< std::hash_map< sg::HashGridIndex< unsigned int,unsigned int > *,size_t,hash< sg::HashGridIndex< unsigned int,unsigned int > * >,eqIndex< sg::HashGridIndex< unsigned int,unsigned int > * > >::iterator > result;
                                                                                                                                                     ^
tmp/build_pysgpp/pysgpp_wrap.cc:21360:149: note: suggested alternative:
In file included from src/sgpp/grid/storage/hashmap/HashGridIndex.hpp:11:0,
                 from src/sgpp/grid/GridStorage.hpp:11,
                 from src/sgpp/sgpp.hpp:11,
                 from tmp/build_pysgpp/pysgpp_wrap.cc:5765:
src/sgpp/common/hash_map_config.hpp:65:9: note:   ‘sg::eqIndex’

Note the "suggested alternative" last line: the "eqIndex" should be qualified with "sg::" before it. It seems that the compiler you're using isn't happy with the potentially-ambiguous reference to "eqIndex", and wants it qualified.

This "eqIndex" seems to be being used in five files without qualification:

src/sgpp/grid/GridDataBase.hpp:  typedef std::hash_map<index_pointer, double, hash<index_pointer>, eqIndex<index_pointer> > grid_map;
src/sgpp/grid/GridDataBase.hpp:  typedef std::tr1::unordered_map<index_pointer, double, hash<index_pointer>, eqIndex<index_pointer> > grid_map;
src/sgpp/grid/storage/hashmap/HashGridIndex.hpp~:struct eqIndex<HashGridIndex<LT, IT>* > {
src/sgpp/grid/storage/hashmap/HashGridStorage.hpp:    typedef std::hash_map<index_pointer, size_t, hash<index_pointer>, eqIndex<index_pointer> > grid_map;
src/sgpp/grid/storage/hashmap/HashGridStorage.hpp:    typedef std::tr1::unordered_map<index_pointer, size_t, hash<index_pointer>, eqIndex<

I suggest you go into those files and modify the source to include the "sg::" prefix in each usage. Then you can try to recompile, and it should hopefully no longer fail with that error. It may give you another similar error which you can track down and fix analogously. I'm no expert on the SGPP code (I didn't write it), but that's my best suggestion on how to proceed.

Hope that helps!

pfluegdk commented 7 years ago

Hi Alex, the SG++ version that is shipped with skinny-dip is a rather old one. The new one can be found at sgpp.sparsegrids.org/downloads.html However, since the version in skinny-dip, the interfaces have changed significantly (to the better, as we hope). Thus, an update of SG++ will not work for skinny-dip.

The errors are due to the fact that your software (in particular swig) is too new :-) I just installed an Ubuntu 14.04 with

gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3)
Python 2.7.6 (and the corresponding python-dev)
SWIG Version 2.0.11
SCons v2.3.0

all from the standard repos. Thus (fresh Ubuntu):

sudo apt-get install g++
sudo apt-get install python-dev
sudo apt-get install scons
sudo apt-get install swig

With that configuration, SG++ compiles without errors (only typedef-warnings).

Cheers from the SG++ developers, Dirk

alexWhitworth commented 7 years ago

Per @pfluegdk, I am able to install skinny-dip running the following:

gcc version 4.8.5
Python 2.7.5 
SWIG Version 2.0.11
SCons v2.3.0