Tensorboard data server ready

[XciD]

Hello,

We currently have a process where we spawn tensorboard on demand. During the spawn, we hang the HTTP request and once tensorboard is up, we forward it.

On big dataset traces, the load of data can takes a few seconds resulting "no scalar" view. Besides some quick hack, could we have some HTTP endpoint that gives us the status of the data server ?

Thanks

[japie1235813]

To provide suggestions and/or support this request we will need more information and clarification.

1. Could you describe your overall set up and environment? How TB is spawned on demand? Are you launching TB through command lines? How is the http request being forward? What kind of "data server" you are referring to? (I would think about the web server but still need confirmation)
2. What tensorboard-data-server package are you using? Do you use RustBoard (i.e. with --load_fast flag) or traditional TB?
3. What kind of "data readiness" and return type you have in mind? For example, do you need the response as soon as there is data loaded for the first round (the most top folder) or you want to wait until all layers/data are scanned (all sub folder are walked). What are the response type in corresponding scenarios? Some background context with examples would be helpful too.

[XciD]

Hello @japie1235813, thank for you answer.

Here is some context:

1. Could you describe your overall set up and environment? How TB is spawned on demand? Are you launching TB through command lines? How is the http request being forward? What kind of "data server" you are referring to? (I would think about the web server but still need confirmation)

We have a proxy in front that catch all HTTP request:

• check if the tensorboard is started in the kubernetes cluster (1 pod = 1 tensorboard)
• forward to it otherwise spawn a new instance and forward to it

To know if the instance is ready, we just wait for the TCP port to be open.

2. What tensorboard-data-server package are you using? Do you use RustBoard (i.e. with --load_fast flag) or traditional TB?

Yes, we use the load_fast param

3. What kind of "data readiness" and return type you have in mind? For example, do you need the response as soon as there is data loaded for the first round (the most top folder) or you want to wait until all layers/data are scanned (all sub folder are walked). What are the response type in corresponding scenarios? Some background context with examples would be helpful too.

Our UX issue is that on restart, all experiments arent here, having a "data readiness" when all files have been walked could be great.

[bileschi]

@XciD would you be interested in attending the TensorBoard Special Interest Group (SIG) this Friday, 2022-02-11 at 11 AM Pacific? This feature is a bit ambiguous re what we would need, and what parts we could re-use. There are some medium-sized UX & technical design issues and we would want to make sure we are focused on the things you and our other users really want.

SIG details in this google doc : https://docs.google.com/document/d/1Uz0ho6YDemtQzqUqG4KrVHwK23axhuepEMENg-teVZg/edit