search

josephsdavid / N2D

Library implementation of https://arxiv.org/abs/1908.05968v5
https://n2d.readthedocs.io/en/latest/
MIT License
23 stars 6 forks source link

Manifold learner takes too long? #7

Open dvaler01 opened 4 years ago

dvaler01 commented 4 years ago

Hello David,

After some additions in the library, I am running a test using Convolutional AE on a Dataset of ~10000 samples (256x256 resolution), where I set the dimensionality of the embedding to 36 and the dimensionality of the Manifold learner to 6. I trained the network for 15 epochs, each epoch consists of 39 steps with batch size 256. After ~3hours the training step ended, so after this it's the turn of the manifold learner. But after 12 hours it seems that the manifold learner still running. So my question is, is it possible to stack in local minimum or something like this? Or it's normal to take so long? My only feedback that the program is still working is from the command htop. Because I don't have any logs during the manifold learning step. Any suggestions/comments/questions?

p.s. I also have my custom data generator where for the Y(output) I applied a mask on the X(input).

Thanks, Dimitris.

josephsdavid commented 4 years ago

Hmmm, personally I have never had that problem, do you mind sending over a minimal example + the version of: UMAP-Learn, Numba, and sklearn you have installed so I can try to reproduce. Do you know if it’s stuck in the UMAP part (lots of output to the console) or the sklearn part? This sounds like an issue I’ve had with tsne but not with UMAP before.

Also, great idea with the custom generator! I was about to implement one for my own project :)