search

OscarKjell / text

Using Transformers from HuggingFace in R
https://r-text.org
132 stars 30 forks source link

CPU vs. GPU usage? #33

Closed sebsilas closed 1 year ago

sebsilas commented 1 year ago

I am trying to use text to produce text embeddings for nearly 700,000 items. Eventually, I am using the text embeddings to compute semantic similarity norms with textSimiliaryNorm. I don't store the text embedding, as this would require a lot of storage space, so it's just an interim step to calculating my text similarity norms.

I get my text embedding like this:

comment_embedding <- text::textEmbed(comments,
                                       keep_token_embeddings = FALSE,
                                       device = 'gpu',
                                       tokenizer_parallelism = TRUE)

where comments is a vector of a certain batch size.

Running this in batch sizes of 1,000 items (comments), takes about 30 minutes on my laptop (8-Core Macbook Pro, 32MB RAM). To try and speed up this process, I have tried running the same procedure on an Amazon Web Services instance with a GPU. However, there doesn't appear to be a performance gain, despite the fact that I can confirm the GPU is being used when I run textEmbed:

Screenshot 2022-12-14 at 14 21 24

(I also don't get the warning from text saying that CUDA is not being used, whereas I do on my laptop).

This 1077MB seems to be an upper bound on GPU usage, whether I change the batch size or get a bigger server with more GPU or CPU power.

The CPU usage is maxing out (100%), effectively, with whatever batch size I use. I have even tried a g3.8xlarge instance instance with 244 GB CPU memory. Again, the CPU is getting maxed out at 100% and the GPU is stuck on 1,077 MB usage, when textEmbed is being run. (Hence), I can't improve the processing times I get on my laptop, even though I am throwing way more power at the same processes.

Do you have any idea why I am not experiencing any performance improvement? Please let me know if I can provide more information.

All my code is working fine (I get the text similarity norms I want at the end), I just need performance improvements. FYI: I am running my code interactively in the R terminal of the server (or RStudio on my laptop).

All the best, Seb

OscarKjell commented 1 year ago

Thanks for the feedback – please note that we are working on increasing the speed of textEmbed() as well as GPU functionality.