Spacy-stanza and Spacy conflict when calling pipelines on the GPU

[Tadusko]

If either spacy.prefer_gpu() or .require_gpu() are called anytime a Stanza pipeline is/will be loaded on the GPU, the successive pipeline runs will fail.

Is there any way to circumvent this or should one of the pipelines be on the CPU if the two need to be loaded at the same time?

How to reproduce the behaviour

import spacy_stanza
import spacy

snlp = spacy_stanza.load_pipeline('fi', processors='tokenize, mwt, lemma, pos, depparse')
spacy.prefer_gpu()
doc = snlp("Tämä on esimerkkilause. Toinen.")

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\x\anaconda3\envs\lang_ai\lib\site-packages\spacy\language.py", line 977, in __call__
    doc = self.make_doc(text)
  File "C:\Users\x\anaconda3\envs\lang_ai\lib\site-packages\spacy\language.py", line 1059, in make_doc
    return self.tokenizer(text)
  File "C:\Users\x\anaconda3\envs\lang_ai\lib\site-packages\spacy_stanza\tokenizer.py", line 83, in __call__
    snlp_doc = self.snlp(text)
  File "C:\Users\x\anaconda3\envs\lang_ai\lib\site-packages\stanza\pipeline\core.py", line 210, in __call__
    doc = self.process(doc)
  File "C:\Users\x\anaconda3\envs\lang_ai\lib\site-packages\stanza\pipeline\core.py", line 204, in process
    doc = process(doc)
  File "C:\Users\x\anaconda3\envs\lang_ai\lib\site-packages\stanza\pipeline\pos_processor.py", line 33, in process
    preds += self.trainer.predict(b)
  File "C:\Users\x\anaconda3\envs\lang_ai\lib\site-packages\stanza\models\pos\trainer.py", line 73, in predict
    _, preds = self.model(word, word_mask, wordchars, wordchars_mask, upos, xpos, ufeats, pretrained, word_orig_idx, sentlens, wordlens)
  File "C:\Users\x\anaconda3\envs\lang_ai\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\x\anaconda3\envs\lang_ai\lib\site-packages\stanza\models\pos\model.py", line 100, in forward
    word_emb = pack(word_emb)
  File "C:\Users\x\anaconda3\envs\lang_ai\lib\site-packages\stanza\models\pos\model.py", line 95, in pack
    return pack_padded_sequence(x, sentlens, batch_first=True)
  File "C:\Users\x\anaconda3\envs\lang_ai\lib\site-packages\torch\nn\utils\rnn.py", line 244, in pack_padded_sequence
    _VF._pack_padded_sequence(input, lengths, batch_first)
RuntimeError: 'lengths' argument should be a 1D CPU int64 tensor, but got 1D cuda:0 Long tensor

Only Stanza pipeline seems to be affected, since loading in and running Spacy pipelines appears to work normally. E.g.

...
nlp = spacy.load("en_core_web_sm")
doc = nlp("This is an example sentence. Another one.")
# no error

I used the Finnish Stanza pipeline since I have that downloaded, but the same issue has been reported previously for other languages as well in Stanza's issues.

Info about spaCy

• spaCy version: 3.0.3
• spaCy-stanza version 1.0.0
• spaCy-transformers version 1.0.1
• Platform: Windows-10-10.0.19041-SP0
• Python version: 3.8.5
• Pipelines: en_core_web_sm (3.0.0)

[adrianeboyd]

I don't think the GPU models will work simultaneously, but you can now switch spacy's CPU/GPU mode back and forth, which wasn't possible in spacy v2. You need to upgrade to the very latest thinc (v8.0.2) which has a bug fix related to torch for require_cpu.

import spacy
import spacy_stanza

stanza_nlp = spacy_stanza.load_pipeline("en")

# load spacy pipeline on GPU
spacy.require_gpu()
spacy_nlp = spacy.load("en_core_web_trf")
doc1 = spacy_nlp("stuff")

# switch back to CPU defaults
spacy.require_cpu()
doc2 = stanza_nlp("other stuff")

spacy models loaded on GPU stay on GPU, but ones that involve torch (all the transformer-based models) will only work if require_gpu is the current state. Ones that just use cupy (the non-trf ones) will keep working on GPU even if you call require_cpu at a later point.

[Tadusko]

Thank you for the answer! CPU/GPU switching should come in handy.