search

pavlin-policar / openTSNE

Extensible, parallel implementations of t-SNE
https://opentsne.rtfd.io
BSD 3-Clause "New" or "Revised" License
1.42k stars 157 forks source link

process crashes when /tmp gets full #240

Closed s-baumann closed 1 year ago

s-baumann commented 1 year ago

I have seen an issue on linux where doing the neighest neighbour step many times can fill up the /tmp directory. Eventually the program crashes.

It would be good if there were a way to not store anything in a temporary folder or if it is possible to manually call a function like openTSNE.clear_temp() to clear all of the objects created in a process.

pavlin-policar commented 1 year ago

I've never run across this issue before. openTSNE doesn't really use tmp files during regular usage, so it's surprising this comes up at all. The only time we use temporary files is when pickling openTSNE embeddings (here). Could this be an issue with Annoy? We use the Annoy library for nearest neighbor search, and I've not really looked at the internals. We bundle our own version here: https://github.com/pavlin-policar/openTSNE/tree/master/openTSNE/dependencies/annoy. Does this problem occur with pynndescent as well?

pavlin-policar commented 1 year ago

Closing due to inactivity and I can't really reproduce this. Please feel free to re-open if the issue persists.