Open youssefavx opened 4 years ago
Hi @youssefavx, the parameter seems to be enforced from the client side. It is a bit unorthodoxed but you will be able to find the parameter here: https://github.com/tensorflow/tensorboard/blob/1b877f76759d0110dceb8313b7db0824f2a5acfd/tensorboard/plugins/projector/vz_projector/data.ts#L83-L89
I've seen many cases when projector slowed down a lot with high number of the samples so I get why we have this sampling but I agree it is not ideal that it is not tune-able.
@hfiller FYI :)
@stephanwlee Thank you for pointing me to that! For some reason I can't find vz_projector in my directory. I did pip install.
Inside tensorboard/plugins/projector, my directory looks like this:
__init__.py metadata.py projector_plugin.py
__pycache__ projector_config_pb2.py tf_projector_plugin
Should I download the vz_projector from that link and add it and see? Or re-install tensorboard some other way?
JavaScript gets compiled. You will need to clone the repo and build/run after making the change.
@stephanwlee Thanks so much! Will try and report back.
After building with bazel, and running, I get this warning:
INFO: Analyzed target //tensorboard:tensorboard (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //tensorboard:tensorboard up-to-date:
bazel-bin/tensorboard/tensorboard
INFO: Elapsed time: 0.208s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
WARNING: Logging before flag parsing goes to stderr.
W0520 11:44:34.761243 4394603968 __init__.py:329] Limited tf.compat.v2.summary API due to missing TensorBoard installation.
W0520 11:44:34.776326 4394603968 __init__.py:329] Limited tf.compat.v2.summary API due to missing TensorBoard installation.
W0520 11:44:34.777926 4394603968 __init__.py:329] Limited tf.compat.v2.summary API due to missing TensorBoard installation.
W0520 11:44:34.801923 4394603968 __init__.py:357] Limited tf.summary API due to missing TensorBoard installation.
Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
I don't know if that affects T-SNE or not. PCA seems to work fine. Yesterday I ran T-SNE for around 8-9 hours but it went from "Finding nearest neighbors..." (without any percentage near it) to "Initializing T-SNE" And my computer's temperature dropped from its usual high point. With the 10K limit before it took around 10-20 minutes.
So I don't know if it's actually stuck, or it's taking a while to process (which I'm fine waiting for).
Here are the diagnostics:
No action items identified. Please copy ALL of the above output, including the lines containing only backticks, into your GitHub issue or comment. Be sure to redact any sensitive information.
While running bazel, I got these warnings:
I'm now building and running again (these messages were from that build) and trying T-SNE again to see.
I got an idea. I'll try building with the 10K limit and see if the same thing happens. If it loads up quickly then at least I know there's nothing wrong and it's just that it's taking a while.
I'm doing git clone https://github.com/tensorflow/tensorboard
btw.
It works with the 10K limit. I'll try slowly incrementing this and building instead of setting it at a million.
Okay, setting it at 20K, it also gets stuck at "Finding nearest neighbors..." without percentage
10,001 points works fine:
11K points works fine:
15K works... 16K works...
I wonder if there's some kind of threshold or something.
Edit: 18K doesn't work, I get the ellipses again and no percentage. Same with 17K...
Is this related to RAM / free hard drive space? Like if I don't have enough RAM or VRAM that's what dictates where the threshold lies? I do notice my hard drive space dropping by quite a bit and not recovering until I restart.
It seems unexpected and curious to me why it would suddenly stop working around 17K. The expected behavior for me would be that it gets progressively slower and slower the higher you go but not completely suddenly stop at a threshold. This is why I'm wondering if this points to a different parameter I have to tune or something or an error I may have made.
I am trying to do something similar with 50k points, did you get it working?
@artist-ai unfortunately, no. I only got it working up to 16K, after that it freezes on loading. I don't think the freeze is directly related to my computer's speed because if it was, 17K would probably load just fine but it didn't. The thing looks like it's loading except it just keeps loading for hours which isn't normal.
I still think you should give it a try though and try to push it as far as you can and see what happens so we have more experience data points to compare. Maybe this limit of 17K is universal or maybe (likely) it's not
I'm running tensorboard locally. I generated a lot of sentence embeddings (92K) for a list of 92K sentences. Now I want to visualize them with T-SNE. I got it working great so far on Tensorboard except for that message. If I understand correctly it's not showing all the sentences / data points, but only 10K. How can I get rid of this limit so that it shows all 92K?
Or did I misunderstand what this means?
I saw this: https://stackoverflow.com/questions/43702546/tensorboard-doesnt-show-all-data-points
But trying both suggestions:
--sames_per_plugin scalars=0
and:--sames_per_plugin sentence_embeddings=0
And changing
scalar_metadata.PLUGIN_NAME: 1000
toscalar_metadata.PLUGIN_NAME: 92324
Didn't seem to do anything. The message "For faster results, the data will be sampled down to 10,000 points." still shows up. Perhaps that message shows up regardless?