search

guillaume-be / rust-bert

Rust native ready-to-use NLP pipelines and transformer-based models (BERT, DistilBERT, GPT2,...)
https://docs.rs/crate/rust-bert
Apache License 2.0
2.51k stars 211 forks source link

Double free or corruption (fasttop) #450

Closed AnBowell closed 4 months ago

AnBowell commented 4 months ago

When attempting to create a multithreaded REST API using Rocket to serve a model, I've run into the following error.

double free or corruption (fasttop)

This is all the error I receive and I presume it's coming from the underlying libtorch library. I've seen other GitHub issues on both this crate and the tch-rs crate talking about how tensors are Send but not Sync, so I wrapped a SentenceEmbeddingsModel in a Mutex on the server start up and then passed it to each method in the API. I assumed this would keep the model safely wrapped, but it doesn't appear so as I get the same error.

I've noticed on the tch-rs crate there was a similar double-free error that was patched a while ago: https://github.com/LaurentMazare/tch-rs/issues/475. I've also reported an issue there.

I understand this may be an issue for the tch-rs crate instead, but hoping there's a more sensible way I can wrap the model up for use across threads? I'm only using the encode method to 

struct SbertModel(Arc<Mutex<SentenceEmbeddingsModel>>);
...
let sbert_model = SbertModel(Arc::new(Mutex::new(
    SentenceEmbeddingsBuilder::remote(SentenceEmbeddingsModelType::AllMiniLmL6V2)
        .create_model()
        .unwrap())));
...
let my_model = model.0.lock().await;
let encoding = my_model.encode(&[some_string])?;
drop(my_model); // Have tried both manually dropping and letting it go out of scope.
AnBowell commented 4 months ago

After some deeper investigations, I've found this is actually originating in a different crate!