search

RGF-team / rgf

Home repository for the Regularized Greedy Forest (RGF) library. It includes original implementation from the paper and multithreaded one written in C++, along with various language-specific wrappers.
378 stars 58 forks source link

Python cant find executables #202

Open casperkaae opened 6 years ago

casperkaae commented 6 years ago

Hi there

I'm trying to install rgf/fastrgf and use the python wrapper to launch the executables.

I've installed using pip install rgf_python

However when i import the rgf module i get a user warning 

UserWarning: Cannot find FastRGF executable files. FastRGF estimators will be unavailable for usage.
  warnings.warn("Cannot find FastRGF executable files. FastRGF estimators will be unavailable for usage.")

To fix this issue i've compiled the rgf and fastrgf binaries* and added them to my $PATH variable (confirmed in bash that they are in the PATH) however i still get the same error. I've looked a bit into the rgf/utils get_paths and is_fastrgf_executable functions however i'm not completely sure why it fails?

*binaries: i was not sure which binaries are needed so i've added the following rgf, forest_predict, forest_train, discretized_trainer, discretized_gendata, auc

System Python: conda 3.6.1 OS: ubuntu 16.04

StrikerRUS commented 6 years ago

Hi @casperkaae !

Let's start from the beginning: please try to install through pip again but with -v option now pip install rgf_python -v and post the whole log here.

casperkaae commented 6 years ago

Thanks for the quick response!, the log is below

(py36) caskaae@hyperion:~$ pip install rgf_python -v
Created temporary directory: /tmp/pip-ephem-wheel-cache-hrq_c3ya
Created temporary directory: /tmp/pip-install-gdn0ohun
Collecting rgf_python
  1 location(s) to search for versions of rgf-python:
  * https://pypi.org/simple/rgf-python/
  Getting page https://pypi.org/simple/rgf-python/
  Looking up "https://pypi.org/simple/rgf-python/" in the cache
  Current age based on date: 10278
  Freshness lifetime from max-age: 600
  Freshness lifetime from request max-age: 600
  Starting new HTTPS connection (1): pypi.org
  https://pypi.org:443 "GET /simple/rgf-python/ HTTP/1.1" 304 0
  Analyzing links from page https://pypi.org/simple/rgf-python/
    Found link https://files.pythonhosted.org/packages/38/b9/cccb61463017359d3d725ee84d4900b5f90f4ed60e4b9946bd084161f986/rgf_python-1.2.0.tar.gz#sha256=a029c6a5bad9391cf3e2c0b9a5cf98c6e34a760de4774460be81d92e1dd853a9 (from https://pypi.org/simple/rgf-python/), version: 1.2.0
    Found link https://files.pythonhosted.org/packages/a8/82/620c00f985bf37e8a500f8d55d9dbae5fbd0a457e1032d3bb9678b0681a5/rgf_python-1.2.1.tar.gz#sha256=e168687745424520c73cc614557a44f0bb09678374b15f1448a4d0f77280a918 (from https://pypi.org/simple/rgf-python/), version: 1.2.1
    Found link https://files.pythonhosted.org/packages/f1/d5/216df0c9d30b2126dd658547d1ec0f6f1b883202d50298c270f95645e1c1/rgf_python-1.2.2.tar.gz#sha256=56609b18c3d0fc5c01c9044770b0f075d10436d1c796df50ed06888a573688c4 (from https://pypi.org/simple/rgf-python/), version: 1.2.2
    Found link https://files.pythonhosted.org/packages/c6/64/324ad7f9b043e6094e548773dccb10fc363bf920a99fd1d7da93be0f3a4f/rgf_python-1.2.3.tar.gz#sha256=2125ff16890ac93d9181c9f4b0b701c7b12de05a8df53aa4b7daa5c883d2c5f7 (from https://pypi.org/simple/rgf-python/), version: 1.2.3
    Found link https://files.pythonhosted.org/packages/19/87/b1c5c72c589059b89c397092ab6bb3105e499982274ae7d6e5e53d01c5a3/rgf_python-1.2.5.tar.gz#sha256=4156040e8db8a16c397518f4dbb78f5a309b9c964355ed3f4362d4a0fdf11583 (from https://pypi.org/simple/rgf-python/), version: 1.2.5
    Found link https://files.pythonhosted.org/packages/8f/5f/81730e1aeaa54e6038902d7b0b464ffcd6fe8dfbdad93b9d010deef65a59/rgf_python-1.3.0.tar.gz#sha256=43d6edf700eaa720340f2aa78a8e9c7ea0468eb0df99bcd82a1c2d74e940dc85 (from https://pypi.org/simple/rgf-python/), version: 1.3.0
    Found link https://files.pythonhosted.org/packages/e2/36/767ad3555af45f69171bfd1d997ab1054871e61aff6b76a900765eb7dec2/rgf_python-1.3.1.tar.gz#sha256=7ff3738cb08356a055ccaa395c453d35088dfe61ca2b5f02e82da9fbea5439f1 (from https://pypi.org/simple/rgf-python/), version: 1.3.1
    Found link https://files.pythonhosted.org/packages/3a/4d/e4a7d32aad0400ace51a8005505fd5f0a1ce4e45199899989ea7a65e2c47/rgf_python-2.0.0.tar.gz#sha256=e40bf5cd76fe1515e5f0a7ea7fcdb5a9505405d5626b177ec8d21826313f81dc (from https://pypi.org/simple/rgf-python/), version: 2.0.0
    Found link https://files.pythonhosted.org/packages/70/e7/00446a94b5f4ab1d56554152dbc3e6d13fe611ab9b9c51ef6bc7a0ff8b61/rgf_python-2.0.1.tar.gz#sha256=bd4f73a142680e719c3be80375ba0447064c6e1e067a0a35de01ed2e042dc9ef (from https://pypi.org/simple/rgf-python/), version: 2.0.1
    Found link https://files.pythonhosted.org/packages/7b/88/18be049ac6ffbb368affa964c00089818c561a58c049b577e75385e1995f/rgf_python-2.0.2.tar.gz#sha256=ce527ec6501b29bfd97ae4156aeeebf450230aa37d84a773af6642a1c68dde41 (from https://pypi.org/simple/rgf-python/), version: 2.0.2
    Found link https://files.pythonhosted.org/packages/cf/27/515562eab19e8d001596652f8b6a48c97580b94cdae956cccca9abe095de/rgf_python-2.0.3.tar.gz#sha256=40d0e79652bd99cd075b653693467e711e78e7319872c089b9ffcdf387156e22 (from https://pypi.org/simple/rgf-python/), version: 2.0.3
    Found link https://files.pythonhosted.org/packages/1c/de/3409d5ecd095fb628aaf84927cada5248f06edba214ab549bc6bb45a495a/rgf_python-2.1.0.tar.gz#sha256=dcd086e0a116e3a7b786360ad00c33e0276896802329c384908986337d62ac8b (from https://pypi.org/simple/rgf-python/), version: 2.1.0
    Skipping link https://files.pythonhosted.org/packages/27/1c/af21c48dcb0c2aa25e3537d110f56eaa2cfa30ef33d35112951dcb29a77d/rgf_python-2.1.2-py2.py3-none-macosx_10_6_x86_64.macosx_10_7_x86_64.macosx_10_8_x86_64.macosx_10_9_x86_64.macosx_10_10_x86_64.macosx_10_11_x86_64.macosx_10_12_x86_64.macosx_10_13_x86_64.whl#sha256=f5c341e6cdb6457c645a8903c1853864e6bf2a41f49be34864b5d8b463e61f4a (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/8a/62/1862aa41399970d273d50fa19996969a298e1ca0d58cab94b9eccd439292/rgf_python-2.1.2-py2.py3-none-manylinux1_i686.whl#sha256=fad54ab32641a0a274517e0775459f0811e2debffd8c47ee6b785cb81fc9af53 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/d8/0d/e409a98d86c20efd171d07c3841c2a64e276996d8c0844096d043f8a89e7/rgf_python-2.1.2-py2.py3-none-manylinux1_x86_64.whl#sha256=e2b0e074889815fb9cf2b4412c375254492ebb74311569dfe7370b7419991f73 (from https://pypi.org/simple/rgf-python/), version: 2.1.2
    Skipping link https://files.pythonhosted.org/packages/98/ea/07ea0cd3d07b4a32fc94f2d4b734bc4d195e21b28cd1fbdee44c6d1f355f/rgf_python-2.1.2-py2.py3-none-win32.whl#sha256=52982ae01d775b37829891c4725b9f50a298fe780e51261f51ef9cc1309319d4 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/cb/83/e095b828f497460428becd30c307b1b29218360867cf729a8368948058f1/rgf_python-2.1.2-py2.py3-none-win_amd64.whl#sha256=c83f54c40e7ee5928227fb0cd6240e651a88ae8e91aee2ae9e287a941d1b9227 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/07/8e/a88bd589bef0843180a9c7254b0ce465ea2342d7840c3881e2134b7d40df/rgf_python-2.1.2.tar.gz#sha256=5ba150e00a166121c80e4e35ae5e573e082231cf2f5b772b1e8a7478d7bbfede (from https://pypi.org/simple/rgf-python/), version: 2.1.2
    Skipping link https://files.pythonhosted.org/packages/fe/6e/77ed62c5d01b4e365652dbfb42e5a638af1f927f92c77cc0408443e3fedd/rgf_python-2.2.0-py2.py3-none-macosx_10_6_x86_64.macosx_10_7_x86_64.macosx_10_8_x86_64.macosx_10_9_x86_64.macosx_10_10_x86_64.macosx_10_11_x86_64.macosx_10_12_x86_64.macosx_10_13_x86_64.whl#sha256=92d93dcb915c03f32e6a81011483569a39dada25e51f619555699c5eb294b6d8 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/39/95/7db1cf6097899c62c98b39437eba5713942ecaf11c4254a67d868afef928/rgf_python-2.2.0-py2.py3-none-manylinux1_i686.whl#sha256=cb3a7db0961a8db7f77a42f42605b702e0a8e142c8f90396862e69892d1c81e7 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/8d/1b/2cfa4fb5d45cf532c289e6d84cf2d5c5848016245ddcc7d21154c6b3989a/rgf_python-2.2.0-py2.py3-none-manylinux1_x86_64.whl#sha256=7a8672fe97417b26844629c21926339062a8c9a20622b1e9f911790c744500a6 (from https://pypi.org/simple/rgf-python/), version: 2.2.0
    Skipping link https://files.pythonhosted.org/packages/50/9e/5465dd83facaaeedf09b06feec5167276cb0a9361e6c8e9bf5e28e73130c/rgf_python-2.2.0-py2.py3-none-win32.whl#sha256=224a20cacbda901c3a9542975bea2c0b8784aa7ae0de78e357cef87ec375f96f (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/cb/4d/0b4504e73f59f239bee317762944ffe077b078dbb57aa17aa178ef67d5b2/rgf_python-2.2.0-py2.py3-none-win_amd64.whl#sha256=bc60da4e0b299fce98b62fe255bc8390787993e56a124796ac59500e1312ad3e (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/0c/28/b7282f39d0b811aa0857aace516c681cbab03a885cc66e6afefb04ed8d36/rgf_python-2.2.0.tar.gz#sha256=b9dec479a177277c08bb511515bf1bc299e29ba21063f754480c41e6eb8ac4a7 (from https://pypi.org/simple/rgf-python/), version: 2.2.0
    Skipping link https://files.pythonhosted.org/packages/d4/ba/93a63d7777fc03048e0c3200eb6063daef934b47fd6057b14ff3e357de98/rgf_python-2.3.0-py2.py3-none-macosx_10_6_x86_64.macosx_10_7_x86_64.macosx_10_8_x86_64.macosx_10_9_x86_64.macosx_10_10_x86_64.macosx_10_11_x86_64.macosx_10_12_x86_64.macosx_10_13_x86_64.whl#sha256=7a7e09161ddb5caeac4e2e2bcd79b4f2c3068a2d77781fb1986e8defeeaae9ce (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/25/ab/335fa23bc031c10d1dab0e678cd0d8e6f2da78609b0de70f7d57b7c10405/rgf_python-2.3.0-py2.py3-none-manylinux1_i686.whl#sha256=3e2e01ab24f8a24be9289c3bbbc47ba5324b7dc3559968e9c1d776ed163b22d3 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/6e/9a/5a7c608201b45f32e172cd77c1c1e2a5c1162808cb1d4151b4c0d0085403/rgf_python-2.3.0-py2.py3-none-manylinux1_x86_64.whl#sha256=9568618ffd5c46454f6750bd362fbf2022662726ba91fc043cb8e2d1796d28ad (from https://pypi.org/simple/rgf-python/), version: 2.3.0
    Skipping link https://files.pythonhosted.org/packages/87/89/d1e37f86f9fc0d8993670f0cb29212b159ed2fbc3c2ce731a229de35cdbc/rgf_python-2.3.0-py2.py3-none-win32.whl#sha256=381a0d293df96a8207db3c4403161983e22f4734d010273a31a60ac095d05396 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/1d/5b/9f37c9e5be8210c06c48621ee5100c2d62144e1043f474ec0598b2cd62d4/rgf_python-2.3.0-py2.py3-none-win_amd64.whl#sha256=810dddc9d57f0168babcf3baf42ef02ce70c7a0e618a4a5bebf26638de70a42e (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/c5/d7/633d28ac0b98deb21ee92780e292026e684cd0ae7a22e2a88cac5caa8480/rgf_python-2.3.0.tar.gz#sha256=d9b83e79e38d5e50e313d3cea27af989a1f96cbb9176c4067e71aebbb77181a2 (from https://pypi.org/simple/rgf-python/), version: 2.3.0
    Skipping link https://files.pythonhosted.org/packages/fc/74/bae2bad70302916f10954f6a0698f9dd4550b8adc10eecd30c79b3334fca/rgf_python-3.0.0-py2.py3-none-macosx_10_6_x86_64.macosx_10_7_x86_64.macosx_10_8_x86_64.macosx_10_9_x86_64.macosx_10_10_x86_64.macosx_10_11_x86_64.macosx_10_12_x86_64.macosx_10_13_x86_64.whl#sha256=42b4a9454facf6b2caa2f113765eed2e905c5cac70e8c9307c5b071a67b7bd40 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/61/ee/d8e917ebe9b9c15e0034d17af8aff82f0afb476d2c2c97a92a45b26d6152/rgf_python-3.0.0-py2.py3-none-manylinux1_i686.whl#sha256=b7c2bad1e28ca129f40d9410d09e0a332d4f5c0c5efde7a0442afe79b9a02641 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/e9/68/d0fddc3068c9b973bed1f5f8a4f0a9b25248d133685183d5235315d0d085/rgf_python-3.0.0-py2.py3-none-manylinux1_x86_64.whl#sha256=5b68c9b9de491fe975887617340f2dd28781a69bbdecdaaf9c0a563718877241 (from https://pypi.org/simple/rgf-python/), version: 3.0.0
    Skipping link https://files.pythonhosted.org/packages/59/70/d9b5ce24c4a0a3923d95df38000b213c5e29b95601a20b8de87f69f4c582/rgf_python-3.0.0-py2.py3-none-win32.whl#sha256=2bd5f871e386ec72c13fc1e8273f17ef5972a2402b08a2f4e675cf31a0151964 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/b5/09/75d7fa21efe3298da8613192e740265d3c526bff5b45631c2bb64d66f6e1/rgf_python-3.0.0-py2.py3-none-win_amd64.whl#sha256=faf3f6ab043ae9186b13505f5e629c06f5739d56f4237cb44d4f4884da39802f (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/49/3f/5e72ad095c44a80f9878f34e62c4a6d541aa36772453b3c06c66a3bb8c77/rgf_python-3.0.0.tar.gz#sha256=f82bf4b429787bcf0f19e60a90f9e1faadf9e984d24c961098bd68a9f1ae9351 (from https://pypi.org/simple/rgf-python/), version: 3.0.0
    Skipping link https://files.pythonhosted.org/packages/52/1a/b9dee7540ee1da9be3a4c5008f2c57b45b4585bb20e2bdce0834787d8043/rgf_python-3.1.0-py2.py3-none-macosx_10_6_x86_64.macosx_10_7_x86_64.macosx_10_8_x86_64.macosx_10_9_x86_64.macosx_10_10_x86_64.macosx_10_11_x86_64.macosx_10_12_x86_64.macosx_10_13_x86_64.whl#sha256=d1c0163303b1ff773786d69e613d5bc99bc1cf364d2af6bea6f6075ee4bfdb1b (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/7f/9b/f488ff1547129261bc5e8c248c4813c6b3f30d1b0696829c838f20223cb4/rgf_python-3.1.0-py2.py3-none-manylinux1_i686.whl#sha256=ceba72ba952211210ce486ddfb42130fa2eb4a19bb5d707f404f29ba0dc590eb (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/33/c4/7c73917d9a250f1d476b7f585e6b8955179e916f8410ed8f77418665aabf/rgf_python-3.1.0-py2.py3-none-manylinux1_x86_64.whl#sha256=23f525d874772da297cddef9fb664de18f9727c4b9a55a9f8ff4ab3f18ab2475 (from https://pypi.org/simple/rgf-python/), version: 3.1.0
    Skipping link https://files.pythonhosted.org/packages/7b/13/2460bc545d347aa6d9e5c8b87a49771812b3b8cb288e8cb401f1b881ada9/rgf_python-3.1.0-py2.py3-none-win32.whl#sha256=d8260578b36d546d44976fd0b4662d482bccf7e868cc254dc0e82eb08310d616 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/15/5e/9a96a8ff4da1a68935b32ffc8a1a7d8ebcff9ee999a4668ef7deeb326b2a/rgf_python-3.1.0-py2.py3-none-win_amd64.whl#sha256=32bc342444340fcf8d6119f140d6d0b946b0f9ea60d05f11a35bf4cd80bd6fe1 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/ae/22/ad7e7d0ff9304d757170e62403e792334ee432b43c6a246a14830b5e1a54/rgf_python-3.1.0.tar.gz#sha256=811c6d78dc6056884f3d225459a77195689bfb963c4cce42c2b5e060dde39826 (from https://pypi.org/simple/rgf-python/), version: 3.1.0
    Skipping link https://files.pythonhosted.org/packages/48/47/5468d6c7c636c3614a5ea6ae717596f65094b9845c9cd037cf2e7f403f05/rgf_python-3.2.0-py2.py3-none-macosx_10_6_x86_64.macosx_10_7_x86_64.macosx_10_8_x86_64.macosx_10_9_x86_64.macosx_10_10_x86_64.macosx_10_11_x86_64.macosx_10_12_x86_64.macosx_10_13_x86_64.whl#sha256=e3548997362562bc8e208dec31b90cbea42697b4d5f5bdcc52f62c604aa59f7e (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/24/4d/015b4d8e8d0f673880921778b98155548e017e439ce778c0a3b7c070b573/rgf_python-3.2.0-py2.py3-none-manylinux1_i686.whl#sha256=c14e55c8d40fed6d5f4526bff2cd7b8597780c3ea62d942c2f8f7d7b1d40f3a7 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/fe/d1/4dec9c38a61ac6eb159fe5161b9e6f54a9255d2f0585f9f0ca9b31d0bf71/rgf_python-3.2.0-py2.py3-none-manylinux1_x86_64.whl#sha256=0995ab3038054b611b4fa97d2bdb87aac4a59310a0fb6b86b5e500beb9e94da1 (from https://pypi.org/simple/rgf-python/), version: 3.2.0
    Skipping link https://files.pythonhosted.org/packages/00/fc/bf212db2a1235b993f11dae289115180e2dcb023715adff2674bac85792d/rgf_python-3.2.0-py2.py3-none-win32.whl#sha256=35281bdffb527234fed60944dd39a6219e2d29951ddc005be8267fe6d0054bd9 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/c8/ba/eaddd7d4afa2e2cf7ac08595be70853b99e312f2b8d02a339de4e60b83d4/rgf_python-3.2.0-py2.py3-none-win_amd64.whl#sha256=9f6f70286f868d6a0ab31ee4b8a429b7664f3fbc01a999267af43688ec1cff66 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/f4/b9/d25a51f0a51d07d6d9d0d67f9ffeb1a057bc2113e281573c8160ce45b7ea/rgf_python-3.2.0.tar.gz#sha256=9a9f497c3e3f558257af20c43e00738d84969fa646780d5dc0fea43e5192ebb8 (from https://pypi.org/simple/rgf-python/), version: 3.2.0
  Using version 3.2.0 (newest of versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3, 1.2.5, 1.3.0, 1.3.1, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.1.0, 2.1.2, 2.2.0, 2.3.0, 3.0.0, 3.1.0, 3.2.0)
  Created temporary directory: /tmp/pip-unpack-nn17gdm4
  Looking up "https://files.pythonhosted.org/packages/fe/d1/4dec9c38a61ac6eb159fe5161b9e6f54a9255d2f0585f9f0ca9b31d0bf71/rgf_python-3.2.0-py2.py3-none-manylinux1_x86_64.whl" in the cache
  Ignoring unknown cache-control directive: 
  Current age based on date: 10278
  Ignoring unknown cache-control directive: immutable
  Freshness lifetime from max-age: 365000000
  The response is "fresh", returning cached response
  365000000 > 10278
  Using cached https://files.pythonhosted.org/packages/fe/d1/4dec9c38a61ac6eb159fe5161b9e6f54a9255d2f0585f9f0ca9b31d0bf71/rgf_python-3.2.0-py2.py3-none-manylinux1_x86_64.whl
  Downloading from URL https://files.pythonhosted.org/packages/fe/d1/4dec9c38a61ac6eb159fe5161b9e6f54a9255d2f0585f9f0ca9b31d0bf71/rgf_python-3.2.0-py2.py3-none-manylinux1_x86_64.whl#sha256=0995ab3038054b611b4fa97d2bdb87aac4a59310a0fb6b86b5e500beb9e94da1 (from https://pypi.org/simple/rgf-python/)
Requirement already satisfied: scikit-learn>=0.18 in ./.conda/envs/py36/lib/python3.6/site-packages (from rgf_python) (0.19.1)
Installing collected packages: rgf-python

Successfully installed rgf-python-3.2.0
Cleaning up...
StrikerRUS commented 6 years ago

OK, it uses cached wheel...

Using cached https://files.pythonhosted.org/packages/fe/d1/4dec9c38a61ac6eb159fe5161b9e6f54a9255d2f0585f9f0ca9b31d0bf71/rgf_python-3.2.0-py2.py3-none-manylinux1_x86_64.whl

But the version is fresh.

Then please do the couple of things:

And does the warning appear only for FastRGF or for RGF too?

casperkaae commented 6 years ago

Hi again

provide your gcc version: gcc --v

(py36) caskaae@hyperion:~$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.9' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9)

copy the content of ./.conda/envs/py36/lib/python3.6/site-packages/rgf_python here

.conda/envs/py36/lib/python3.6/site-packages/rgf_python-3.2.0.dist-info/ contains the following files: INSTALLER, METADATA, RECORD, top_level.txt, WHEEL

.conda/envs/py36/lib/python3.6/site-packages/rgf/ contains the following files: fastrgf_model.py, forest_predict, forest_train, __init__.py, __pycache__/, rgf, rgf_model.py, sklearn.py, utils.py, VERSION

try to run forest_train -h if it is in that folder


.conda/envs/py36/lib/python3.6/site-packages/rgf/forest_train -h
.conda/envs/py36/lib/python3.6/site-packages/rgf/forest_train version 0.4 (Aug 2017) by Tong Zhang
Read a dataset in appropriate format from a file.
Train a decision forest, and save the trained model to a file.

usage: .conda/envs/py36/lib/python3.6/site-packages/rgf/forest_train [options]

options: -h [-help | --help] : print this help

....[discriptions of all the input arguments]



> And does the warning appear only for FastRGF or for RGF too?

I have only seen the warning for the FastRGF executeables. If i run `from rgf.sklearn import RGFRegressor` it also only complains about missing FastRGF executeables

Thanks a lot for the help
Casper
StrikerRUS commented 6 years ago

Hmm...

Please try to grant executable permission for forest_predict and forest_train by

chmod a+x .conda/envs/py36/lib/python3.6/site-packages/rgf/forest_predict .conda/envs/py36/lib/python3.6/site-packages/rgf/forest_train

Also, you've said

To fix this issue i've compiled the rgf and fastrgf binaries* and added them to my $PATH variable (confirmed in bash that they are in the PATH) however i still get the same error.

Replace forest_predict and forest_train executables in .conda/envs/py36/lib/python3.6/site-packages/rgf/ by those you've compiled on yourself.

BTW,

*binaries: i was not sure which binaries are needed so i've added the following rgf, forest_predict, forest_train, discretized_trainer, discretized_gendata, auc

only rgf (RGF), forest_predict and forest_train (FastRGF) are used.

casperkaae commented 6 years ago

Hi again

I've tried to (regrant) executeable permissions as well as using my own compiled versions of the executeables, however no luck yet. Can there be anything in the python script detecting the files that doesn't work?

StrikerRUS commented 6 years ago

It's a pity.

Python package tries to find FastRGF as follows: https://github.com/RGF-team/rgf/blob/0dd8273a62348ac162dcd339975a7a37a861b2e6/python-package/rgf/utils.py#L188-L196

  1. directory where script is placed;
  2. directory from PATH ;
  3. specified in .rgfrc config file directory or $HOME directory.

I suppose the package finds executables correctly but they are broken. To be more precise https://github.com/RGF-team/rgf/blob/0dd8273a62348ac162dcd339975a7a37a861b2e6/python-package/rgf/utils.py#L137 function returns False.

To check this, you can call this function with path to the folder where FastRGF executables are located right now and could be run by console command (forest_train -h).

It often happens when the gcc version is less than 5, but it's not your case.

Maybe something wrong with the OpenMP library?..

casperkaae commented 6 years ago

I've tested the is_fastrgf_executable function (slightly modded last the try-except to actually raise the error)

from rgf.utils import is_fastrgf_executable
path = '/home/caskaae/.conda/envs/py36/lib/python3.6/site-packages/rgf/'
is_fastrgf_executable(path)

the subprocess call fails with

subprocess.check_output(cmd_train, stderr=subprocess.STDOUT)
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
<ipython-input-28-34524568c302> in <module>()
----> 1 subprocess.check_output(cmd_train, stderr=subprocess.STDOUT)

~/.conda/envs/py36/lib/python3.6/subprocess.py in check_output(timeout, *popenargs, **kwargs)
    334 
    335     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
--> 336                **kwargs).stdout
    337 
    338 

~/.conda/envs/py36/lib/python3.6/subprocess.py in run(input, timeout, check, *popenargs, **kwargs)
    416         if check and retcode:
    417             raise CalledProcessError(retcode, process.args,
--> 418                                      output=stdout, stderr=stderr)
    419     return CompletedProcess(process.args, retcode, stdout, stderr)
    420 

CalledProcessError: Command '['/home/caskaae/.conda/envs/py36/lib/python3.6/site-packages/rgf/forest_train', 'forest.ntrees=10', 'tst.target=BINARY', 'trn.x-file=/tmp/rgf/temp_fastrgf.train.data.x', 'trn.y-file=/tmp/rgf/temp_fastrgf.train.data.y', 'model.save=/tmp/rgf/temp_fastrgf.model']' died with <Signals.SIGSEGV: 11>.

.... so something is not working with the executeables.

Btw i get the same error if i reinstall the pacakage from scratch through pip, e.g. not using my own compiled versions of the binaries

StrikerRUS commented 6 years ago

Thank you very much!

Let's exclude or confirm that your problem is actually this issue #92.

How many processor cores do you have?

Please run is_fastrgf_executable() multiple times varying two lines: replace 14 with some other numbers (42, 140, 400, etc.) https://github.com/RGF-team/rgf/blob/0dd8273a62348ac162dcd339975a7a37a861b2e6/python-package/rgf/utils.py#L142-L143

4-5 times will be enough.

StrikerRUS commented 6 years ago

Hi @casperkaae !

Have you had a chance to check different #data?

dslate commented 6 years ago

I ran into this problem while trying to get the RGF R package working on some x86_64 workstations running Ubuntu Linux 16.04 and python3.6. I found that python reported errors trying to "import rgf":

dave@alicia:~$ python3 Python 3.6.3 (default, Oct 27 2017, 18:35:38) [GCC 5.4.0 20160609] on linux Type "help", "copyright", "credits" or "license" for more information.

import rgf /usr/local/lib/python3.6/site-packages/rgf/utils.py:196: UserWarning: Cannot find FastRGF executable files. FastRGF estimators will be unavailable for usage. warnings.warn("Cannot find FastRGF executable files. FastRGF estimators will be unavailable for usage.")

Of my six workstations, the error occurred on the three that had 16, 24, and 24 cores, but not on the machines with 2, 4, and 10 cores.

Applying the following mod to /usr/local/lib/python3.6/site-packages/rgf/utils.py "fixed" "import rgf" by limiting the number of threads:

  ~ ~ ~ ~ ~ ~ ~ ~
  *** utils.py.orig       Sun Aug 12 10:20:21 2018
  --- utils.py    Sun Aug 12 17:39:27 2018
  ***************
  *** 146,151 ****
  --- 146,152 ----
        UUIDS.append('temp_fastrgf')
        path_train = os.path.join(path, "forest_train")
        params_train = []
  +     params_train.append("set.nthreads=4")
        params_train.append("forest.ntrees=%s" % 10)
        params_train.append("tst.target=%s" % "BINARY")
        params_train.append("trn.x-file=%s" % temp_x_loc)

Of course this does not explain why the larger number of cores induces the failure.

Sorry about the formatting of my post; I'm not sure how to fix it. To clarify, the line I added to rgf/utils.py was:

params_train.append("set.nthreads=4")

StrikerRUS commented 6 years ago

@dslate Thank you very much for your information! It maybe be treated as another proof that the FastRGF error is caused by hidden multithreading issue (race condition?): #92.

Of my six workstations, the error occurred on the three that had 16, 24, and 24 cores ...

Did you want to say 16, 24 and 32, right?

We'll appreciate any help from you. Do you have some time and wish to dig deeper in this issue? Do you still have the access to those machines where the error happens? Before your post I could reproduce the error only on Windows, but your workstations run under Linux - it's interesting.

PS. I fixed the markdown in your post: just added the code block around the diff.

dslate commented 6 years ago

@StrikerRUS Thanks for your response. Actually I did mean 16, 24 and 24; two of my machines have 24 cores each. After installing my mod to rgf/utils.py I have been trying to get FastRGF_Regressor working for a current project in R. I am already using lightgbm, xgboost, and catboost with the same data, and recently got the command line version of rgf (version 1.3) running ok with it. This is my first attempt to use FastRGF_Regressor, and I am running into additional difficulties, such as predictions that are basically random numbers that have almost no correlation with the target, which is a 0-1 binary (by the way, should I be treating this as a regression or classification problem; the goal is to maximize AUC score?). I don't know yet whether this problem is due to bugs in my code or in FastRGF. This project has a deadline, so I need to either get FastRGF working pretty soon so I can add it to the algorithms whose predictions I ensemble, or just give up and go with the algorithms I'm already using.

I really don't have much time to try to debug this, but if you come up with some potential fixes I may be able to help test them in my environment. I couldn't tell from the posts about this issue whether the errors occur intermittently with the same data or are data dependent. If the latter, perhaps the problem is not a "race condition" but some other bug(s) in the multi-threading algorithm.

fukatani commented 6 years ago

@dslate Thanks for your report. Debugging FastRGF will be a tough job, so it is better to lower threads and if you use it.

I agree that there is a possibility that it is not a data race. Reproduction should be unstable if it is a data race, but it always happens first cycle of training.

StrikerRUS commented 6 years ago

@dslate AUC is a classification metric. For

the goal is to maximize AUC score

please take a look at this #199.

Unfortunately, FastRGF multithreading issue is rather old and I'm not sure we'll fix it soon. So, for your project you'd better use RGF, since FastRGF is unstable in your environment.

R wrapper (#208) will be integrated into this repo very soon (just after @fukatani's review).

ryancheunggit commented 5 years ago

Thank you very much!

Let's exclude or confirm that your problem is actually this issue #92.

How many processor cores do you have?

Please run is_fastrgf_executable() multiple times varying two lines: replace 14 with some other numbers (42, 140, 400, etc.) rgf/python-package/rgf/utils.py

Lines 142 to 143 in 0dd8273

X = np.tile(np.array([[1, 0, 1, 0], [0, 1, 0, 1]]), (14, 1)) y = np.tile(np.array([1, -1]), 14) 4-5 times will be enough.

I am on a 12 core machine(7920x) and getting the same error. this change solved my problem and it is good to go now...

StrikerRUS commented 5 years ago

Thanks a lot @ryancheunggit !

Your information is very helpful!

ryancheunggit commented 5 years ago

Sure thing.

I have done some further testing, in my case, the minimum number I have to replace 14 with is 23, which is the number of threads - 1. Maybe the test function in utils.py could use something like os.cpu_count() to replace the hard-coded number 14.

StrikerRUS commented 5 years ago

@ryancheunggit Thanks for your investigation! I'm not sure that the dependency is so determined. 14 came from my 8-threaded CPU...

albertnanda commented 5 years ago

Environment Info Operating System: Ubuntu 16.04, 48 cores

rgf_python version:'3.6.0'

Python version (for rgf_python errors): 3.6

Error Message Exception: FastRGF estimators are unavailable for usage.

Hi, I am successfully able to build fastRGF executables i.e forest_train & forest_predict. Output from: forest_train -h forest_train version 0.6 But I am unable to use it from python. I copied the files to /miniconda3/lib/python3.6/site-packages/rgf , I also started the ipython console from the directory containing the executables, still it's not working.

fukatani commented 5 years ago

@albertnanda

Thank you for your report!

To obtain information necessary for debugging, could you try as follows?

  1. Save this script as debug.py.
from __future__ import absolute_import

import atexit
import codecs
import glob
import numbers
import os
import platform
import stat
import subprocess
import warnings
from tempfile import gettempdir
from threading import Lock
from uuid import uuid4

import numpy as np
import scipy.sparse as sp
from sklearn.base import BaseEstimator
from sklearn.exceptions import NotFittedError
from sklearn.externals import six
from sklearn.utils.extmath import softmax
from sklearn.utils.multiclass import check_classification_targets
from sklearn.utils.validation import check_array, check_consistent_length, check_X_y, column_or_1d

CURRENT_DIR = os.path.abspath(os.path.dirname(__file__))
FLOATS = (float, np.float, np.float16, np.float32, np.float64, np.double)
INTS = (numbers.Integral, np.integer)
NOT_FITTED_ERROR_DESC = "Estimator not fitted, call `fit` before exploiting the model."
NOT_IMPLEMENTED_ERROR_DESC = "This method isn't implemented in base class."
SYSTEM = platform.system()

def get_paths():
    config = six.moves.configparser.RawConfigParser()
    path = os.path.join(os.path.expanduser('~'), '.rgfrc')

    try:
        with codecs.open(path, 'r', 'utf-8') as cfg:
            with six.StringIO(cfg.read()) as strIO:
                config.readfp(strIO)
    except six.moves.configparser.MissingSectionHeaderError:
        with codecs.open(path, 'r', 'utf-8') as cfg:
            with six.StringIO('[glob]\n' + cfg.read()) as strIO:
                config.readfp(strIO)
    except Exception:
        pass

    if SYSTEM in ('Windows', 'Microsoft'):
        try:
            rgf_exe = os.path.abspath(config.get(config.sections()[0], 'exe_location'))
        except Exception:
            rgf_exe = os.path.join(os.path.expanduser('~'), 'rgf.exe')
        def_rgf = 'rgf.exe'
    else:  # Linux, Darwin (macOS), etc.
        try:
            rgf_exe = os.path.abspath(config.get(config.sections()[0], 'exe_location'))
        except Exception:
            rgf_exe = os.path.join(os.path.expanduser('~'), 'rgf')
        def_rgf = 'rgf'

    try:
        fastrgf_path = os.path.abspath(config.get(config.sections()[0], 'fastrgf_location'))
    except Exception:
        fastrgf_path = os.path.expanduser('~')
    def_fastrgf = ''

    try:
        temp = os.path.abspath(config.get(config.sections()[0], 'temp_location'))
    except Exception:
        temp = os.path.join(gettempdir(), 'rgf')

    return def_rgf, rgf_exe, def_fastrgf, fastrgf_path, temp

DEFAULT_RGF_PATH, RGF_PATH, DEFAULT_FASTRGF_PATH, FASTRGF_PATH, TEMP_PATH = get_paths()

if not os.path.isdir(TEMP_PATH):
    os.makedirs(TEMP_PATH)
if not os.access(TEMP_PATH, os.W_OK):
    raise Exception("{0} is not writable directory. Please set "
                    "config flag 'temp_location' to writable directory".format(TEMP_PATH))

UUIDS = []

def is_fastrgf_executable(path):
    print("Seaching " + path)
    temp_x_loc = os.path.join(TEMP_PATH, 'temp_fastrgf.train.data.x')
    temp_y_loc = os.path.join(TEMP_PATH, 'temp_fastrgf.train.data.y')
    temp_model_loc = os.path.join(TEMP_PATH, "temp_fastrgf.model")
    temp_pred_loc = os.path.join(TEMP_PATH, "temp_fastrgf.predictions.txt")
    X = np.tile(np.array([[1, 0, 1, 0], [0, 1, 0, 1]]), (14, 1))
    y = np.tile(np.array([1, -1]), 14)
    np.savetxt(temp_x_loc, X, delimiter=' ', fmt="%s")
    np.savetxt(temp_y_loc, y, delimiter=' ', fmt="%s")
    UUIDS.append('temp_fastrgf')
    path_train = os.path.join(path, "forest_train")
    params_train = []
    params_train.append("forest.ntrees=%s" % 10)
    params_train.append("tst.target=%s" % "BINARY")
    params_train.append("trn.x-file=%s" % temp_x_loc)
    params_train.append("trn.y-file=%s" % temp_y_loc)
    params_train.append("model.save=%s" % temp_model_loc)
    cmd_train = [path_train]
    cmd_train.extend(params_train)
    path_pred = os.path.join(path, "forest_predict")
    params_pred = []
    params_pred.append("model.load=%s" % temp_model_loc)
    params_pred.append("tst.x-file=%s" % temp_x_loc)
    params_pred.append("tst.output-prediction=%s" % temp_pred_loc)
    cmd_pred = [path_pred]
    cmd_pred.extend(params_pred)
    try:
        os.chmod(path_train, os.stat(path_train).st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
        os.chmod(path_pred, os.stat(path_pred).st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
    except Exception:
        pass
    try:
        subprocess.check_output(cmd_train, stderr=subprocess.STDOUT)
        subprocess.check_output(cmd_pred, stderr=subprocess.STDOUT)
    except Exception as e:
        print(e)
        return False
    print("FastRGF found on " + path)
    return True

FASTRGF_AVAILABLE = True
if is_fastrgf_executable(CURRENT_DIR):
    FASTRGF_PATH = CURRENT_DIR
elif is_fastrgf_executable(DEFAULT_FASTRGF_PATH):
    FASTRGF_PATH = DEFAULT_FASTRGF_PATH
elif is_fastrgf_executable(FASTRGF_PATH):
    pass
else:
    FASTRGF_AVAILABLE = False
    warnings.warn("Cannot find FastRGF executable files. FastRGF estimators will be unavailable for usage.")
  1. Execute python ./debug.py.
  2. Please paste the result of debug.py.
fukatani commented 5 years ago

@StrikerRUS Is "set.nthreads=1" needed in is_fastrgf_executable? Or we may use maximum number of hardware logical threads.

albertnanda commented 5 years ago

@fukatani Output of debug.py /home/ubuntu/miniconda3/lib/python3.6/site-packages/sklearn/externals/six.py:31: DeprecationWarning: The module is deprecated in version 0.21 and will be removed in version 0.23 since we've dropped support for Python 2.7. Please rely on the official version of six (https://pypi.org/project/six/). "(https://pypi.org/project/six/).", DeprecationWarning) Seaching /home/ubuntu/rgf/FastRGF/bin Command '['/home/ubuntu/rgf/FastRGF/bin/forest_train', 'forest.ntrees=10', 'tst.target=BINARY', 'trn.x-file=/tmp/rgf/temp_fastrgf.train.data.x', 'trn.y-file=/tmp/rgf/temp_fastrgf.train.data.y', 'model.save=/tmp/rgf/temp_fastrgf.model']' died with <Signals.SIGSEGV: 11>. Seaching Command '['forest_train', 'forest.ntrees=10', 'tst.target=BINARY', 'trn.x-file=/tmp/rgf/temp_fastrgf.train.data.x', 'trn.y-file=/tmp/rgf/temp_fastrgf.train.data.y', 'model.save=/tmp/rgf/temp_fastrgf.model']' died with <Signals.SIGSEGV: 11>.Seaching /home/ubuntu/rgf/FastRGF/bin Command '['/home/ubuntu/rgf/FastRGF/bin/forest_train', 'forest.ntrees=10', 'tst.target=BINARY', 'trn.x-file=/tmp/rgf/temp_fastrgf.train.data.x', 'trn.y-file=/tmp/rgf/temp_fastrgf.train.data.y', 'model.save=/tmp/rgf/temp_fastrgf.model']' died with <Signals.SIGSEGV: 11>. ./debug.py:138: UserWarning: Cannot find FastRGF executable files. FastRGF estimators will be unavailable for usage. warnings.warn("Cannot find FastRGF executable files. FastRGF estimators will be unavailable for usage.")

albertnanda commented 5 years ago

Additional Info: GCC Version Linux-gnu Thread model: posix gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.11) cmake version cmake version 3.5.1 File Permissions -rw-rw-r-- 1 ubuntu ubuntu 5.0K Aug 12 05:56 debug.py -rwxr-xr-x 1 ubuntu ubuntu 597K Aug 2 09:28 forest_predict -rwxr-xr-x 1 ubuntu ubuntu 592K Aug 2 09:28 forest_train

fukatani commented 5 years ago

@casperkaae Thx! So building forest_train is suceeded and rgf_python recognize forest_train's location correctly. But forest_train program is crashed. It can be caused by FastRGF mutli threading issue.

Could you try another debug script here? It is almost same as predecessor, but different with thread number.

from __future__ import absolute_import

import atexit
import codecs
import glob
import numbers
import os
import platform
import stat
import subprocess
import warnings
from tempfile import gettempdir
from threading import Lock
from uuid import uuid4

import numpy as np
import scipy.sparse as sp
from sklearn.base import BaseEstimator
from sklearn.exceptions import NotFittedError
from sklearn.externals import six
from sklearn.utils.extmath import softmax
from sklearn.utils.multiclass import check_classification_targets
from sklearn.utils.validation import check_array, check_consistent_length, check_X_y, column_or_1d

CURRENT_DIR = os.path.abspath(os.path.dirname(__file__))
FLOATS = (float, np.float, np.float16, np.float32, np.float64, np.double)
INTS = (numbers.Integral, np.integer)
NOT_FITTED_ERROR_DESC = "Estimator not fitted, call `fit` before exploiting the model."
NOT_IMPLEMENTED_ERROR_DESC = "This method isn't implemented in base class."
SYSTEM = platform.system()

def get_paths():
    config = six.moves.configparser.RawConfigParser()
    path = os.path.join(os.path.expanduser('~'), '.rgfrc')

    try:
        with codecs.open(path, 'r', 'utf-8') as cfg:
            with six.StringIO(cfg.read()) as strIO:
                config.readfp(strIO)
    except six.moves.configparser.MissingSectionHeaderError:
        with codecs.open(path, 'r', 'utf-8') as cfg:
            with six.StringIO('[glob]\n' + cfg.read()) as strIO:
                config.readfp(strIO)
    except Exception:
        pass

    if SYSTEM in ('Windows', 'Microsoft'):
        try:
            rgf_exe = os.path.abspath(config.get(config.sections()[0], 'exe_location'))
        except Exception:
            rgf_exe = os.path.join(os.path.expanduser('~'), 'rgf.exe')
        def_rgf = 'rgf.exe'
    else:  # Linux, Darwin (macOS), etc.
        try:
            rgf_exe = os.path.abspath(config.get(config.sections()[0], 'exe_location'))
        except Exception:
            rgf_exe = os.path.join(os.path.expanduser('~'), 'rgf')
        def_rgf = 'rgf'

    try:
        fastrgf_path = os.path.abspath(config.get(config.sections()[0], 'fastrgf_location'))
    except Exception:
        fastrgf_path = os.path.expanduser('~')
    def_fastrgf = ''

    try:
        temp = os.path.abspath(config.get(config.sections()[0], 'temp_location'))
    except Exception:
        temp = os.path.join(gettempdir(), 'rgf')

    return def_rgf, rgf_exe, def_fastrgf, fastrgf_path, temp

DEFAULT_RGF_PATH, RGF_PATH, DEFAULT_FASTRGF_PATH, FASTRGF_PATH, TEMP_PATH = get_paths()

if not os.path.isdir(TEMP_PATH):
    os.makedirs(TEMP_PATH)
if not os.access(TEMP_PATH, os.W_OK):
    raise Exception("{0} is not writable directory. Please set "
                    "config flag 'temp_location' to writable directory".format(TEMP_PATH))

UUIDS = []

def is_fastrgf_executable(path):
    print("Seaching " + path)
    temp_x_loc = os.path.join(TEMP_PATH, 'temp_fastrgf.train.data.x')
    temp_y_loc = os.path.join(TEMP_PATH, 'temp_fastrgf.train.data.y')
    temp_model_loc = os.path.join(TEMP_PATH, "temp_fastrgf.model")
    temp_pred_loc = os.path.join(TEMP_PATH, "temp_fastrgf.predictions.txt")
    X = np.tile(np.array([[1, 0, 1, 0], [0, 1, 0, 1]]), (14, 1))
    y = np.tile(np.array([1, -1]), 14)
    np.savetxt(temp_x_loc, X, delimiter=' ', fmt="%s")
    np.savetxt(temp_y_loc, y, delimiter=' ', fmt="%s")
    UUIDS.append('temp_fastrgf')
    path_train = os.path.join(path, "forest_train")
    params_train = []
    params_train.append("forest.ntrees=%s" % 10)
    params_train.append("set.nthreads=1")
    params_train.append("tst.target=%s" % "BINARY")
    params_train.append("trn.x-file=%s" % temp_x_loc)
    params_train.append("trn.y-file=%s" % temp_y_loc)
    params_train.append("model.save=%s" % temp_model_loc)
    cmd_train = [path_train]
    cmd_train.extend(params_train)
    path_pred = os.path.join(path, "forest_predict")
    params_pred = []
    params_pred.append("model.load=%s" % temp_model_loc)
    params_pred.append("tst.x-file=%s" % temp_x_loc)
    params_pred.append("set.nthreads=1")
    params_pred.append("tst.output-prediction=%s" % temp_pred_loc)
    cmd_pred = [path_pred]
    cmd_pred.extend(params_pred)
    try:
        os.chmod(path_train, os.stat(path_train).st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
        os.chmod(path_pred, os.stat(path_pred).st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
    except Exception:
        pass
    try:
        subprocess.check_output(cmd_train, stderr=subprocess.STDOUT)
        subprocess.check_output(cmd_pred, stderr=subprocess.STDOUT)
    except Exception as e:
        print(e)
        return False
    print("FastRGF found on " + path)
    return True

FASTRGF_AVAILABLE = True
if is_fastrgf_executable(CURRENT_DIR):
    FASTRGF_PATH = CURRENT_DIR
elif is_fastrgf_executable(DEFAULT_FASTRGF_PATH):
    FASTRGF_PATH = DEFAULT_FASTRGF_PATH
elif is_fastrgf_executable(FASTRGF_PATH):
    pass
else:
    FASTRGF_AVAILABLE = False
    warnings.warn("Cannot find FastRGF executable files. FastRGF estimators will be unavailable for usage.")
albertnanda commented 5 years ago

@fukatani

Seaching /home/ubuntu/rgf/FastRGF/bin FastRGF found on /home/ubuntu/rgf/FastRGF/bin

albertnanda commented 5 years ago

@fukatani I tried with 12 threads, it crashed and then I tried with 6 threads and it worked.

fukatani commented 5 years ago

@albertnanda Thx!

Current FastRGF has trouble for handling small data with many thread.

I opened #301 , it is work around for this issue. This can work for your environment.

git clone https://github.com/RGF-team/rgf.git
cd rgf
git checkout reduce-checking-nthreads
cd python-package
pip install -e .

But I don't know it is appropriate process for Anaconda. Or you can edit rgf/python-package/utils.py directly as #301 .

We recommend trying small nthread (I mean n_jobs) first, and increase nthread gradually.

StrikerRUS commented 5 years ago

@fukatani

Or we may use maximum number of hardware logical threads.

According to this info from @albertnanda , it seems to be useless.

Operating System: Ubuntu 16.04, 48 cores

I tried with 12 threads, it crashed and then I tried with 6 threads and it worked.

I think that we need to fix #92 and this issue will disappear.