NBoost is a scalable, search-api-boosting platform for deploying transformer models to improve the relevance of search results on different platforms (i.e. Elasticsearch)
Ubuntu 18.04 install fails in 2 ways:
1) pip install nboost[pt] with default python 3.6 or pip3 install nboost[pt]
`sudo pip3 install nboost[pt]
[sudo] password for lad:
The directory '/home/lad/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/lad/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting nboost[pt]
Downloading https://files.pythonhosted.org/packages/f8/89/83e4ee2d2a0a673579e06024e20c22a9857cff2dad2303a339d5b7d0bed6/nboost-0.3.9.tar.gz (831kB)
100% |████████████████████████████████| 839kB 793kB/s
Requirement already satisfied: elasticsearch in /usr/local/lib/python3.6/dist-packages (from nboost[pt])
Requirement already satisfied: flask in /usr/local/lib/python3.6/dist-packages (from nboost[pt])
Requirement already satisfied: jsonpath-ng in /usr/local/lib/python3.6/dist-packages (from nboost[pt])
Requirement already satisfied: nltk in /usr/local/lib/python3.6/dist-packages (from nboost[pt])
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from nboost[pt])
Requirement already satisfied: termcolor in /usr/local/lib/python3.6/dist-packages (from nboost[pt])
Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from nboost[pt])
Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (from nboost[pt])
Collecting transformers==2.7.0 (from nboost[pt])
Downloading https://files.pythonhosted.org/packages/37/ba/dda44bbf35b071441635708a3dd568a5ca6bf29f77389f7c7c6818ae9498/transformers-2.7.0-py3-none-any.whl (544kB)
100% |████████████████████████████████| 552kB 2.0MB/s
Requirement already satisfied: urllib3<2,>=1.21.1 in /usr/lib/python3/dist-packages (from elasticsearch->nboost[pt])
Requirement already satisfied: certifi in /usr/lib/python3/dist-packages (from elasticsearch->nboost[pt])
Requirement already satisfied: click>=5.1 in /usr/lib/python3/dist-packages (from flask->nboost[pt])
Requirement already satisfied: Werkzeug>=0.15 in /usr/local/lib/python3.6/dist-packages (from flask->nboost[pt])
Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.6/dist-packages (from flask->nboost[pt])
Requirement already satisfied: Jinja2>=2.10.1 in /usr/local/lib/python3.6/dist-packages (from flask->nboost[pt])
Requirement already satisfied: ply in /usr/local/lib/python3.6/dist-packages (from jsonpath-ng->nboost[pt])
Requirement already satisfied: six in /usr/lib/python3/dist-packages (from jsonpath-ng->nboost[pt])
Requirement already satisfied: decorator in /usr/local/lib/python3.6/dist-packages (from jsonpath-ng->nboost[pt])
Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from nltk->nboost[pt])
Requirement already satisfied: regex in /usr/local/lib/python3.6/dist-packages (from nltk->nboost[pt])
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torch->nboost[pt])
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.6/dist-packages (from torch->nboost[pt])
Requirement already satisfied: dataclasses; python_version < "3.7" in /usr/local/lib/python3.6/dist-packages (from torch->nboost[pt])
Collecting sentencepiece (from transformers==2.7.0->nboost[pt])
Downloading https://files.pythonhosted.org/packages/95/bc/b39f9627c41027caf669cf4d1e47a7415cfa018a7dcb899aebd122af7c91/sentencepiece-0.1.95.tar.gz (508kB)
100% |████████████████████████████████| 512kB 2.3MB/s
Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers==2.7.0->nboost[pt])
Requirement already satisfied: tokenizers==0.5.2 in /usr/local/lib/python3.6/dist-packages (from transformers==2.7.0->nboost[pt])
Collecting boto3 (from transformers==2.7.0->nboost[pt])
Downloading https://files.pythonhosted.org/packages/b3/7f/c7bafbc7db335f6c5194d9b948b0c6fcfe013d14026992150ee749fc5877/boto3-1.16.53-py2.py3-none-any.whl (130kB)
100% |████████████████████████████████| 133kB 3.7MB/s
Requirement already satisfied: sacremoses in /usr/local/lib/python3.6/dist-packages (from transformers==2.7.0->nboost[pt])
Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.6/dist-packages (from Jinja2>=2.10.1->flask->nboost[pt])
Collecting jmespath<1.0.0,>=0.7.1 (from boto3->transformers==2.7.0->nboost[pt])
Downloading https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl
Collecting botocore<1.20.0,>=1.19.53 (from boto3->transformers==2.7.0->nboost[pt])
Downloading https://files.pythonhosted.org/packages/a2/89/f2511851beb7d687ac45638a0e62b5cdeb0abca572ac7d3d250d6f4c888f/botocore-1.19.53-py2.py3-none-any.whl (7.2MB)
100% |████████████████████████████████| 7.2MB 167kB/s
Collecting s3transfer<0.4.0,>=0.3.0 (from boto3->transformers==2.7.0->nboost[pt])
Downloading https://files.pythonhosted.org/packages/ea/43/4b4a1b26eb03a429a4c37ca7fdf369d938bd60018fc194e94b8379b0c77c/s3transfer-0.3.4-py2.py3-none-any.whl (69kB)
100% |████████████████████████████████| 71kB 5.3MB/s
Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.20.0,>=1.19.53->boto3->transformers==2.7.0->nboost[pt])
Downloading https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl (227kB)
100% |████████████████████████████████| 235kB 3.6MB/s
Installing collected packages: sentencepiece, jmespath, python-dateutil, botocore, s3transfer, boto3, transformers, nboost
Running setup.py install for sentencepiece ... error
Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;file='/tmp/pip-build-trbowu18/sentencepiece/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-s5s0aakj-record/install-record.txt --single-version-externally-managed --compile:
/usr/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'long_description_content_type'
warnings.warn(msg)
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/sentencepiece
copying src/sentencepiece/init.py -> build/lib.linux-x86_64-3.6/sentencepiece
copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece
copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece
running build_ext
/bin/sh: 1: pkg-config: not found
Cloning into 'sentencepiece'...
Note: checking out '0e6dfbf86e2fa6d86a3d9a8a08a628da71c073e0'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
./build_bundled.sh: 15: ./build_bundled.sh: cmake: not found
make: *** No targets specified and no makefile found. Stop.
make: *** No rule to make target 'install'. Stop.
env: ‘pkg-config’: No such file or directory
Failed to find sentencepiece pkg-config
----------------------------------------
Ubuntu 18.04 install fails in 2 ways: 1) pip install nboost[pt] with default python 3.6 or pip3 install nboost[pt]
`sudo pip3 install nboost[pt] [sudo] password for lad: The directory '/home/lad/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag. The directory '/home/lad/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag. Collecting nboost[pt] Downloading https://files.pythonhosted.org/packages/f8/89/83e4ee2d2a0a673579e06024e20c22a9857cff2dad2303a339d5b7d0bed6/nboost-0.3.9.tar.gz (831kB) 100% |████████████████████████████████| 839kB 793kB/s Requirement already satisfied: elasticsearch in /usr/local/lib/python3.6/dist-packages (from nboost[pt]) Requirement already satisfied: flask in /usr/local/lib/python3.6/dist-packages (from nboost[pt]) Requirement already satisfied: jsonpath-ng in /usr/local/lib/python3.6/dist-packages (from nboost[pt]) Requirement already satisfied: nltk in /usr/local/lib/python3.6/dist-packages (from nboost[pt]) Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from nboost[pt]) Requirement already satisfied: termcolor in /usr/local/lib/python3.6/dist-packages (from nboost[pt]) Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from nboost[pt]) Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (from nboost[pt]) Collecting transformers==2.7.0 (from nboost[pt]) Downloading https://files.pythonhosted.org/packages/37/ba/dda44bbf35b071441635708a3dd568a5ca6bf29f77389f7c7c6818ae9498/transformers-2.7.0-py3-none-any.whl (544kB) 100% |████████████████████████████████| 552kB 2.0MB/s Requirement already satisfied: urllib3<2,>=1.21.1 in /usr/lib/python3/dist-packages (from elasticsearch->nboost[pt]) Requirement already satisfied: certifi in /usr/lib/python3/dist-packages (from elasticsearch->nboost[pt]) Requirement already satisfied: click>=5.1 in /usr/lib/python3/dist-packages (from flask->nboost[pt]) Requirement already satisfied: Werkzeug>=0.15 in /usr/local/lib/python3.6/dist-packages (from flask->nboost[pt]) Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.6/dist-packages (from flask->nboost[pt]) Requirement already satisfied: Jinja2>=2.10.1 in /usr/local/lib/python3.6/dist-packages (from flask->nboost[pt]) Requirement already satisfied: ply in /usr/local/lib/python3.6/dist-packages (from jsonpath-ng->nboost[pt]) Requirement already satisfied: six in /usr/lib/python3/dist-packages (from jsonpath-ng->nboost[pt]) Requirement already satisfied: decorator in /usr/local/lib/python3.6/dist-packages (from jsonpath-ng->nboost[pt]) Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from nltk->nboost[pt]) Requirement already satisfied: regex in /usr/local/lib/python3.6/dist-packages (from nltk->nboost[pt]) Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torch->nboost[pt]) Requirement already satisfied: typing-extensions in /usr/local/lib/python3.6/dist-packages (from torch->nboost[pt]) Requirement already satisfied: dataclasses; python_version < "3.7" in /usr/local/lib/python3.6/dist-packages (from torch->nboost[pt]) Collecting sentencepiece (from transformers==2.7.0->nboost[pt]) Downloading https://files.pythonhosted.org/packages/95/bc/b39f9627c41027caf669cf4d1e47a7415cfa018a7dcb899aebd122af7c91/sentencepiece-0.1.95.tar.gz (508kB) 100% |████████████████████████████████| 512kB 2.3MB/s Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers==2.7.0->nboost[pt]) Requirement already satisfied: tokenizers==0.5.2 in /usr/local/lib/python3.6/dist-packages (from transformers==2.7.0->nboost[pt]) Collecting boto3 (from transformers==2.7.0->nboost[pt]) Downloading https://files.pythonhosted.org/packages/b3/7f/c7bafbc7db335f6c5194d9b948b0c6fcfe013d14026992150ee749fc5877/boto3-1.16.53-py2.py3-none-any.whl (130kB) 100% |████████████████████████████████| 133kB 3.7MB/s Requirement already satisfied: sacremoses in /usr/local/lib/python3.6/dist-packages (from transformers==2.7.0->nboost[pt]) Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.6/dist-packages (from Jinja2>=2.10.1->flask->nboost[pt]) Collecting jmespath<1.0.0,>=0.7.1 (from boto3->transformers==2.7.0->nboost[pt]) Downloading https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl Collecting botocore<1.20.0,>=1.19.53 (from boto3->transformers==2.7.0->nboost[pt]) Downloading https://files.pythonhosted.org/packages/a2/89/f2511851beb7d687ac45638a0e62b5cdeb0abca572ac7d3d250d6f4c888f/botocore-1.19.53-py2.py3-none-any.whl (7.2MB) 100% |████████████████████████████████| 7.2MB 167kB/s Collecting s3transfer<0.4.0,>=0.3.0 (from boto3->transformers==2.7.0->nboost[pt]) Downloading https://files.pythonhosted.org/packages/ea/43/4b4a1b26eb03a429a4c37ca7fdf369d938bd60018fc194e94b8379b0c77c/s3transfer-0.3.4-py2.py3-none-any.whl (69kB) 100% |████████████████████████████████| 71kB 5.3MB/s Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.20.0,>=1.19.53->boto3->transformers==2.7.0->nboost[pt]) Downloading https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl (227kB) 100% |████████████████████████████████| 235kB 3.6MB/s Installing collected packages: sentencepiece, jmespath, python-dateutil, botocore, s3transfer, boto3, transformers, nboost Running setup.py install for sentencepiece ... error Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;file='/tmp/pip-build-trbowu18/sentencepiece/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-s5s0aakj-record/install-record.txt --single-version-externally-managed --compile: /usr/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'long_description_content_type' warnings.warn(msg) running install running build running build_py creating build creating build/lib.linux-x86_64-3.6 creating build/lib.linux-x86_64-3.6/sentencepiece copying src/sentencepiece/init.py -> build/lib.linux-x86_64-3.6/sentencepiece copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece running build_ext /bin/sh: 1: pkg-config: not found Cloning into 'sentencepiece'... Note: checking out '0e6dfbf86e2fa6d86a3d9a8a08a628da71c073e0'.
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;file='/tmp/pip-build-trbowu18/sentencepiece/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-s5s0aakj-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-trbowu18/sentencepiece/ `
2) via docker run --uhost my_es_hostname --uport 9200 --search_route //_search --query_path url.query.size --default_topk 10 --choices_path body.hits.hits --cvalues_path _source.passage with error on http://nboos_hostname:8181/nboost/status
{"doc":null,"msg":"('no such table: searches',)","type":"OperationalError"}
192.x.y.z - - [13/Jan/2021 16:22:25] "GET /nboost/status?_=1610555537815 HTTP/1.1" 500 - E:Proxy:[pro:han:132]: Traceback (most recent call last): File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/opt/conda/lib/python3.6/site-packages/nboost/proxy.py", line 78, in status_path stats = db.get_stats() File "/opt/conda/lib/python3.6/site-packages/nboost/database.py", line 70, in get_stats ''').fetchone() sqlite3.OperationalError: no such table: searches