search

scikit-learn / scikit-learn-benchmarks

Results of scikit-learn asv benchmarks
https://scikit-learn.org/scikit-learn-benchmarks/
35 stars 26 forks source link

vbench missing some classes #1

Closed erg closed 4 years ago

erg commented 11 years ago

And here they are...

This is using your vbench repository.

[erg@pliny scikit-learn-speed]$ [master*] make PYTHON=python2
python2 benchmarks/run_suite.py quick
Traceback (most recent call last):
  File "benchmarks/run_suite.py", line 11, in <module>
    benchmarks, _ = gather(quick=quick)
  File "/home/erg/python/scikit-learn-speed/benchmarks/suite.py", line 23, in gather
    __import__(mod).__dict__)]
  File "/home/erg/python/scikit-learn-speed/benchmarks/linear_model/__init__.py", line 1, in <module>
    from benchmarks import suite
  File "/home/erg/python/scikit-learn-speed/benchmarks/linear_model/benchmarks.py", line 1, in <module>
    from templates import make_suite as _make_suite
  File "/home/erg/python/scikit-learn-speed/benchmarks/templates.py", line 1, in <module>
    from vbench.benchmark import BenchmarkSuite, PythonBenchmark, \
ImportError: cannot import name PythonBenchmark
make: *** [run] Error 1
vene commented 11 years ago

Did you read the quick-start guide at http://vene.github.com/scikit-learn-speed/quick_start.html ?

It links to a specific branch on my vbench repo. Sorry for having it so convoluted.

jenkins-scikit-learn commented 11 years ago

If you want to reuse vbench, I would appreciate any little help in cleaning it up! The problem is my changes are no longer backwards compatible with the pandas vbench suite (and I don't intend them to be, since that version is pretty hackish) so I couldn't push towards a new official vbench API without having at least another project (except scikit-learn) using it!

However, the way my branch is designed now allows for vbench to be extended and used with arbitrary code as long as an appropriate Benchmark subclass is implemented.

Thanks, Vlad

On Fri, Feb 22, 2013 at 8:12 AM, Vlad Niculae notifications@github.comwrote:

Did you read the quick-start guide at http://vene.github.com/scikit-learn-speed/quick_start.html ?

It links to a specific branch on my vbench repo. Sorry for having it so convoluted.

— Reply to this email directly or view it on GitHubhttps://github.com/scikit-learn/scikit-learn-speed/issues/1#issuecomment-13932882.

erg commented 11 years ago

Ah! I think the quick start guide doesn't mention the abstract_benchmarks branch, it just links to it.

I don't have any other projects to apply your vbench to at this time. I mainly wanted to add a few more benchmarks.

erg commented 11 years ago

Now I'm getting this error:

[erg@pliny scikit-learn-speed]$ [master*] make PYTHON=python2
python2 benchmarks/run_suite.py quick
Traceback (most recent call last):
  File "benchmarks/run_suite.py", line 16, in <module>
    module_dependencies=dependencies)
  File "/usr/lib/python2.7/site-packages/vbench/runner.py", line 42, in __init__
    self.checksums = [b.checksum for b in benchmarks]
  File "/usr/lib/python2.7/site-packages/vbench/benchmark.py", line 175, in checksum
    return hashlib.md5(self.setup + self.code + self.cleanup).hexdigest()
TypeError: cannot concatenate 'str' and 'list' objects
make: *** [run] Error 1
vene commented 11 years ago

I think you actually need to use this branch. Allow me to update the quick start guide, it was pointing to the wrong branch.

https://github.com/vene/vbench/tree/abstract_multistep_benchmarks

vene commented 11 years ago

Actually the quick start guide is accurate, just not explicit enough: simply click on the link to my repo and it will take you to the correct branch

erg commented 11 years ago

I wasn't observant enough to realize I was in a branch on the github site.

erg commented 11 years ago

scikit-learn-speed requires sphinx-build.

erg commented 11 years ago

Another problem--I don't have a python executable, just python2and the scripts are hardcoded to python instead of using the environment variable PYTHON.

Here's where I hardcoded it to python2.

[erg@pliny scikit-learn-speed]$ [master*] git diff
diff --git a/benchmarks/suite.py b/benchmarks/suite.py
index fb9daea..0be63ea 100644
--- a/benchmarks/suite.py
+++ b/benchmarks/suite.py
@@ -68,10 +68,10 @@ except:
     print "Cannot load configuration file. Reverting to defaults."

 PREPARE = """
-python setup.py clean
+python2 setup.py clean
 """
 BUILD = """
-python setup.py build_ext --inplace
+python2 setup.py build_ext --inplace
 """

 RST_BASE = 'doc'
erg commented 11 years ago

Finally, it sorta runs, but there are a ton of errors and I don't get any benchmark results in index.html.

[erg@pliny scikit-learn-speed]$ [master*] make PYTHON=python2
python2 benchmarks/run_suite.py quick
git clone git@github.com:scikit-learn/scikit-learn.git /tmp/vb_sklearn_tmp
fatal: destination path '/tmp/vb_sklearn_tmp' already exists and is not an empty directory.
Deleting /tmp/vb_sklearn first
rm -rf /tmp/vb_sklearn
git clone /tmp/vb_sklearn_tmp /tmp/vb_sklearn
Cloning into '/tmp/vb_sklearn'...
done.
python2 setup.py clean
Partial import of sklearn during the build process.
/usr/lib/python2.7/site-packages/numpy/distutils/system_info.py:1494: UserWarning: 
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  warnings.warn(AtlasNotFoundError.__doc__)
sklearn/setup.py:64: UserWarning: 
    Blas (http://www.netlib.org/blas/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [blas]) or by setting
    the BLAS environment variable.
  warnings.warn(BlasNotFoundError.__doc__)
blas_opt_info:
blas_mkl_info:
  libraries mkl,vml,guide not found in ['/usr/local/lib', '/usr/lib64', '/usr/lib']
  NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
  libraries ptf77blas,ptcblas,atlas not found in ['/usr/local/lib', '/usr/lib64', '/usr/lib']
  NOT AVAILABLE

atlas_blas_info:
  libraries f77blas,cblas,atlas not found in ['/usr/local/lib', '/usr/lib64', '/usr/lib']
  NOT AVAILABLE

blas_info:
  FOUND:
    libraries = ['blas']
    library_dirs = ['/usr/lib64']
    language = f77

  FOUND:
    libraries = ['blas']
    library_dirs = ['/usr/lib64']
    define_macros = [('NO_ATLAS_INFO', 1)]
    language = f77

running clean

cp -R /usr/lib/python2.7/site-packages/vbench/scripts/vb_run_benchmarks.py /tmp/vb_sklearn
cp -R benchmarks/deps.py /tmp/vb_sklearn
cp -R benchmarks/templates.py /tmp/vb_sklearn
cp -R benchmarks/data /tmp/vb_sklearn
SKIPPING BLACKLISTED edb7592
python2 benchmarks/generate_rst_files.py quick
Generating rst file for Ridge-arcene
Generating rst file for Ridge-madelon
Generating rst file for Lars-minimadelon-oney
Generating rst file for Lars-madelon
Generating rst file for LassoLars-minimadelon-oney
Generating rst file for LassoLars-madelon
Generating rst file for Lasso-minimadelon-oney
Generating rst file for Lasso-madelon
Generating rst file for ElasticNet-minimadelon-oney
Generating rst file for OrthogonalMatchingPursuit-minimadelon
Generating rst file for OrthogonalMatchingPursuit-madelon
Generating rst file for SGDClassifier-madelon
Generating rst file for SGDClassifier-newsgroups
Generating rst file for LogisticRegression-arcene
Generating rst file for LogisticRegression-madelon
Generating rst file for ARDRegression-minimadelon-oney
Generating rst file for ARDRegression-blobs
Generating rst file for BayesianRidge-arcene
Generating rst file for BayesianRidge-madelon
sphinx-build -b html -P -d benchmarks/build/doctrees   doc benchmarks/build/html
Running Sphinx v1.2pre
loading pickled environment... done
building [html]: targets for 2 source files that are out of date
updating environment: 0 added, 2 changed, 0 removed
reading sources... [100%] vb_linear_model                                                                                                                                                     
doc/vbench/Ridge-arcene.txt:38: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/Ridge-madelon.txt:38: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/Lars-minimadelon-oney.txt:38: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/Lars-madelon.txt:38: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/LassoLars-minimadelon-oney.txt:38: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/LassoLars-madelon.txt:38: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/Lasso-minimadelon-oney.txt:38: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/Lasso-madelon.txt:38: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/ElasticNet-minimadelon-oney.txt:38: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/ElasticNet-minimadelon-oney.txt:61: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/OrthogonalMatchingPursuit-minimadelon.txt:38: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/OrthogonalMatchingPursuit-madelon.txt:38: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/SGDClassifier-madelon.txt:38: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/SGDClassifier-madelon.txt:61: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/SGDClassifier-newsgroups.txt:38: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/SGDClassifier-newsgroups.txt:61: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/LogisticRegression-arcene.txt:38: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/LogisticRegression-arcene.txt:61: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/LogisticRegression-madelon.txt:38: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/LogisticRegression-madelon.txt:61: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/ARDRegression-minimadelon-oney.txt:38: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/ARDRegression-blobs.txt:38: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/BayesianRidge-arcene.txt:38: ERROR: Content block expected for the "container" directive; none found.
doc/vbench/BayesianRidge-madelon.txt:38: ERROR: Content block expected for the "container" directive; none found.
looking for now-outdated files... none found
pickling environment... done
checking consistency... /home/erg/python/scikit-learn-speed/doc/quick_start.rst:: WARNING: document isn't included in any toctree
done
preparing documents... done
writing output... [100%] vb_linear_model                                                                                                                                                      
writing additional files... genindex search
copying images... [100%] vbench/figures/LogisticRegression-madelon-step1-timing.png                                                                                                           
copying static files... done
dumping search index... done
dumping object inventory... done
build succeeded, 25 warnings.
touch benchmarks/build/html/.nojekyll

Build finished. The HTML pages are in benchmarks/build/html.
vene commented 11 years ago

Judging by the log I believe the current commit is blacklisted because it failed before you changed the hardcoded python call. Try deleting the database and running again.

jeremiedbb commented 4 years ago

This repo is being repurposed to host results of a new asv benchmark suite. I'm closing this.