search

pavlin-policar / openTSNE

Extensible, parallel implementations of t-SNE
https://opentsne.rtfd.io
BSD 3-Clause "New" or "Revised" License
1.42k stars 157 forks source link

Pickling/unpickling embeddings object does not work on Windows #260

Closed austinpatrickm closed 1 month ago

austinpatrickm commented 1 month ago

Hi Pavlin,

This is similar to issue #244.

I'll direct you to this thread:

https://stackoverflow.com/questions/70701836/opentsne-pickle-preserve-transformer

I've tested this extensively and on Mac and Linux and it works fine. But on Windows, there is a fundamental issue being invoked when I try to pickle the trained embeddings object, or unpickle the object if it was embedded in a Linux/Mac env. That is: 'WinErr 32 - The process cannot access the file because it is being used by another process: ...\AppData\Local\Temp...\tmp.ann'.

I've found that small datasets do work. So for example embeddings fitted on a dataset of shape (999, 50) did pickle fine, but (1000, 50) and anything above that leads to the same old issue.

pavlin-policar commented 1 month ago

Hi, thanks for bringing this up, this has been an ongoing issue that, unfortunately, I don't know how to solve. I don't have access to a Windows machine, so I can't really debug this. I've moved the discussion to #210 so we have all the stuff in one place.

austinpatrickm commented 1 month ago

Thanks. For what it's worth I did manage to find a workaround by downgrading to 0.6.0 and changing the handling of the annoy file.