search

adobe / NLP-Cube

Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing
http://opensource.adobe.com/NLP-Cube/index.html
Apache License 2.0
550 stars 93 forks source link

ERROR in app: Exception on /nlp [GET] #123

Closed Spaskich closed 3 years ago

Spaskich commented 3 years ago

Describe the bug I'm getting the following errors on some of the requests when running in a docker container as a web server:

10.42.109.126 - - [04/Feb/2021 14:38:55] "GET /nlp?lang=id&text=WE+Online%2C+Jakarta+-+%0A++Indeks+Harga+Saham+Gabungan+%28IHSG%29+masih+mampu+bangkit+0.48%25+atau29.47+poin+ke+level+6%2C107.22+dengan+saham-saham+disektor+Aneka+Industri+naik+2.11%25%2C+Infrastruktur+0.96%25+dan+Keuangan+0.81%25.++%0A++Analis+Reliance+Sekuritas%2C+Lanjar+Nafi+mengungkapkan+bila+melesatnya+harga+saham+PT+Indah+Kiat+Pulp+and+Paper+Tbk+%28INKP%29+yang+sebesar+12.2%25+dan+PT+Tjiwi+Kimia+Tbk+%28TKIM%29+sebesar+19.9%25+menjadi+motor+pengerak+IHSG+pada+perdagangan+4+Februari+2021.++%0A++Baca+Juga%3A+Sempat+Anjlok+ke+Zona+Merah%2C+IHSG+Parkir+dengan+Apresiasi+0%2C48%25+pada+Penutupan+Sesi+II.++%0A++%22Ini+setelah+harga+pulp+naik+karena+Paper+Excellence+BV+yang+di+kendalikan+oleh+Jackson+Wijaya+dari+Indonesia+memenangkan+kasus+abritrase+terhadap+grub+brazil+J%26F+Investimentos+SA+untuk+menyelesaikan+akuisisi+pabrik+kertas+Eldorado+Brazil+serta.+Kemudian+juga+karena+aksi+beli+investor+asing+sseniali+Rp609%2C15+miliar%2C%22+terangnya%2C+di+Jakarta%2C+Kamis+%284%2F2%2F2021%29.++%0A++Baca+Juga%3A+Mayoritas+Saham+Bank-bank+BUMN+Sumringah+Setelah+Erick+Lempar+Pujian.++%0A++Menurutnya%2C+secara+teknikal+IHSG+bergerak+terkonsolidasi+namun+terlihat+cukup+kuat+diatas+level+rata-rata+50+hari+dan+5+hari+sebagai+support+pergerakan.+Selanjutnya+IHSG+berpotensi+menguji+resistance+rata-rata+20+hari+sebagai+indikasi+pembentukan+uptrend+jangka+pendek+menuju+upper+bollinger+bands.+Indikator+Stochastic+terkonfirmasi+alami+momentum+bullish+dengan+MACD+yang+bergerak+positif.++%0A++%22Sehingga+diperkirakan+IHSG+bergerak+melanjutkan+penguatan+diakhir+pekan+dengan+support+resistance+6051-6239.+Saham-saham+yang+dapat+dicermati+secara+teknikal+diantaranya%3B+ACST%2C+ADRO%2C+BMRI%2C+HOKI%2C+INDY%2C+PTBA%2C+PTPP%2C+RALS%2C+TBLA%2C+WSKT%2C+dan+WSBP%2C%22+tutup+Lanjar. HTTP/1.1" 500 -
[2021-02-04 14:38:55,773] ERROR in app: Exception on /nlp [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "webserver.py", line 71, in nlp
    result = thecube(query)
  File "../cube/api.py", line 184, in __call__
    sequences+=self._tokenizer.tokenize(input_line)
  File "../cube/generic_networks/tokenizers.py", line 384, in tokenize
    if np.argmax(y.npvalue()) == 1:
  File "_dynet.pyx", line 730, in _dynet.Expression.npvalue
  File "_dynet.pyx", line 745, in _dynet.Expression.npvalue
ValueError: Attempt to do tensor forward in different devices (nodes 20330 and 39)
10.42.109.126 - - [04/Feb/2021 14:38:55] "GET /nlp?lang=id&text=Grab-Gojek+Dapat+Saingan+Baru+di+Negeri+Singa HTTP/1.1" 500 -
[2021-02-04 14:38:55,872] ERROR in app: Exception on /nlp [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "webserver.py", line 71, in nlp
    result = thecube(query)
  File "../cube/api.py", line 184, in __call__
    sequences+=self._tokenizer.tokenize(input_line)
  File "../cube/generic_networks/tokenizers.py", line 410, in tokenize
    seq = self._get_tokens(w.strip(), space_after_end_of_sentence=space_after_end_of_sentence)
  File "../cube/generic_networks/tokenizers.py", line 320, in _get_tokens
    y_pred, _, _ = self._predict_tok(input_string, runtime=True)
  File "../cube/generic_networks/tokenizers.py", line 264, in _predict_tok
    word_lstm = word_lstm.add_input(dy.tanh(emb + hol))
  File "_dynet.pyx", line 6026, in _dynet.RNNState.add_input
  File "_dynet.pyx", line 6036, in _dynet.RNNState.add_input
  File "_dynet.pyx", line 4941, in _dynet._RNNBuilder.add_input_to_prev
ValueError: Using stale builder. Create .new_graph() after computation graph is renewed.

I can't reproduce the issue when running the image locally.

To Reproduce Steps to reproduce the behavior:

  1. Build and run the image in a production docker environment
  2. Wait for the models to download.
  3. Send the following request 10.42.109.126 - - [04/Feb/2021 14:38:55] "GET /nlp?lang=id&text=Grab-Gojek+Dapat+Saingan+Baru+di+Negeri+Singa HTTP/1.1" 500 -
  4. See error

Expected behavior When running locally the everything works fine and the Cube returns the expected output.

Screenshots image

tiberiu44 commented 3 years ago

Looking at the error, it seems to be related to DyNET itself. The framework pushed tensors on two different GPUs. We are no longer supporting DyNET and have moved NLP-Cube to PyTorch. We are currently working on releasing a new version and you will no longer have this type of issues. Until then, I suggest you use a CPU-only docker.

Spaskich commented 3 years ago

When deploying the project to a container, I get the following warning:

2/2/2021 12:39:24 PM   WARNING: This is a development server. Do not use it in a production deployment.
2/2/2021 12:39:24 PM   Use a production WSGI server instead.

Could this cause any of the aforementioned problems?

tiberiu44 commented 3 years ago

No, it has nothing to do with this. The actual error is ValueError: Attempt to do tensor forward in different devices (nodes 20330 and 39), which has to do with DyNet allocating tensors on two different GPUs.

Probably you cannot reproduce the error locally, because you have none or just one GPU.

tiberiu44 commented 3 years ago

This issue should be fixed in 3.0, which has been officially released