Any documentation On how to use it ?

[jeffin143]

I would like to know how to use it , How should I call the Logger from my c++ file

[RustingSword]

Please refer to the test file.

[jeffin143]

Thank you so much @RustingSword for the really quick response

Could you share me your email id , I have some questions regarding the internals , or may be twitter handle so that I could get in touch with you

Thanks

Please close the issue one you have seen the comments

[RustingSword]

There are barely any internals in this project. 😂 If you mean internals of tensorboard or some other things, I doubt I know enough to answer your questions. Anyways, feel free to send emails to i#toonaive.me.

[jeffin143]

Yeah I meant , internals of tensorboard/tensorflow writer

[RustingSword]

All I know currently is that every record is a corresponding pb message, which will be serialized and written to the event file. Maybe you can state your questions first, then I can see if I could help.

[jeffin143]

message Image {
    // Dimensions of the image.
    int32 height = 1;
    int32 width = 2;
    // Valid colorspace values are
    //   1 - grayscale
    //   2 - grayscale + alpha
    //   3 - RGB
    //   4 - RGBA
    //   5 - DIGITAL_YUV
    //   6 - BGRA
    int32 colorspace = 3;
    // Image data in encoded format.  All image formats supported by
    // image_codec::CoderUtil can be stored here.
    bytes encoded_image_string = 4;
  }

bytes encoded_image_string here means ?

int TensorBoardLogger::add_image(const string &tag, int step, string value) {
    auto summary = new Summary();
    auto imageo = new  Summary_Image();
    imageo->set_height(3);
    imageo->set_width(3);
    imageo->set_colorspace(3);
    imageo->set_encoded_image_string(value);

    auto v = summary->add_value();
    v->set_node_name(tag);
    v->set_tag(tag);
    v->set_allocated_image(imageo);
    return add_event(step, summary);
}

Tried this snippet of code but it didn't log properly

[RustingSword]

As the comment indicated, encoded_image_string should be the encoded byte string of an image.

Actually I was planning to add support for other record types, but didn't find the time. (Maybe I'll try this weekend.) You can refer to dmlc tensorboard project for this, and the relevent part is here, where the image is encoded as a PNG file, using PIL.

[RustingSword]

Hi @jeffin143, I just updated the code to support image, audio and text record. It turns out you just need to pass the content of an image file as encoded_image_string for it to work. Please see if you have any further questions.

[jeffin143]

Thanks @RustingSword , I never knew about .rbuf() function :)

Great work :)

[jeffin143]

@RustingSword By any chance is it possible to plot multiple images during one epoch or one step

https://github.com/tensorflow/tensorboard/blob/8277e9f8d50758bb4d2b7a0fce1cab7e736b3dd2/tensorboard/plugins/image/summary_v2.py#L45

I will take a look too

Thanks

[RustingSword]

I noticed this option too. However, it requires tf.image.encode_png operator, which requires libpng. I think it's too heavy a dependency for this small project. Maybe a better option is to merge these images manually before calling add_image, after all, you already need to create an image file in the first place.

[jeffin143]

Do you mean merge manually as in concatenate string or something else ?

    ifstream fin("./assets/test.jpeg");
    ostringstream ss;
    ss << fin.rdbuf();
    std::string temp = ss.str();
    ss.str("");
    fin.close();
    fin.open("./assets/Lenna_(test_image).png")  ;
    ss << fin.rdbuf();
    temp = temp + ss.str();
    fin.close();
    logger.add_image(
        "Image Sample", 1, temp, 512, 512, 3, "Lena Forsén",
        "Lenna or Lena is the name given to a standard test image widely used "
        "in the field of image processing since 1973.");

[RustingSword]

I don't think it's as simple as concatenating strings. If I'm understanding correctly, the right way is to decode each image to get raw pixel data as a tensor ([height, width, channel]), then concatenate them into a single tensor, then encode it into some image format, such as png or jpeg.

For example, png images are encoded in <length, type, data, crc> format, if you cancatenate two png images directly, you'll get a corrupted file (wrong data and crc code). So you'll need corresponding image manipulation library for the decoding/encoding part.

[jeffin143]

That makes sense, But what I noticed is they are using tf.summary.write instead of tf.summary.image.

The implementation of add_image is exactly same as mirror of tf.summary.image so that is correct and it is not possible to add that support there if I am not wrong.

But we don't have this tf.summary.write implemented which logs a tensor - https://github.com/tensorflow/tensorflow/blob/2b96f3662bd776e277f86997659e61046b56c315/tensorflow/python/ops/summary_ops_v2.py#L614-L678

this will allow us to log vector of values and I am assuming if that is possible then we can log vector encoded images

So instead of concatenating strings we can push to the vector and then pass that string of vector/tensor

Let my know what do you thing

[RustingSword]

this will allow us to log vector of values and I am assuming if that is possible then we can log vector encoded images

I don't quite understand this. Actually filling summay.value.tensor is the most general way to log data, and we can directly fill this field, only if we specify the pertained plugin name, just like this.

While thinking about this, I find that each time we only fill one value of a summary, so I'm wondering if we can add multiple values to log multiple images. I pushed a demo to multi-images branch, and I'm wondering if this is what you wanted. It will log multiple images in one step, and there's a slider to choose among images, which means you cannot see all images at the same time. This differs from the above talked method to merge all images into a big one.

[RustingSword]

Seems there's a bug with TensorBoard, it always shows the first display name and description, though I specified different ones for each image.

[jeffin143]

I pushed a demo to multi-images branch, and I'm wondering if this is what you wanted.

I will take a look in a while

Seems there's a bug with TensorBoard, it always shows the first display name and description, though I specified different ones for each image.

I am not sure if that is a bug , or they meant to pack multiple values under one tag-name and description Although they are multiple values but I guess they have written it that way

Any ways we can open an issue and let them answer it , If you don't have the bandwidth to open an issue I can get it done

Let me know

Is the most general way to log data, and we can directly fill this field, only if we specify the pertained plugin name

I am not sure How does that work So I want to log just a number 1.2 and 2.3 then I should Specify DT_FLOAT as my plugin type and the can log multiple values for summary

[jeffin143]

I don't quite understand this.

@RustingSword : https://github.com/tensorflow/tensorboard/blob/8277e9f8d50758bb4d2b7a0fce1cab7e736b3dd2/tensorboard/plugins/image/summary_v2.py#L17

This might help so they store width height and different png as an array of string that's it So we load multiple images and then store

[ dimesnion , png0, png1 ] as tensor of string and then log that value as normal tensor of string with apt tags that's it

[jeffin143]

Seems there's a bug with TensorBoard, it always shows the first display name and description, though I specified different ones for each image.

https://www.tensorflow.org/tensorboard/image_summaries

https://github.com/tensorflow/tensorboard/blob/master/docs/images/images_multiple.png?raw=1

Scroll down to log multiple images , They are all with same tag and description and step count that means , The only difference is

sample 1 of 25 and same 2 of 25

[jeffin143]

How should I log a string array ? Could you share a snippet of code ?

[RustingSword]

Seems there's a bug with TensorBoard, it always shows the first display name and description, though I specified different ones for each image.

Nevermind, seems this is intended behaviour, repeated values with the same tag will be stripped to keep the first metadata only. https://github.com/tensorflow/tensorflow/blob/ab0e8d2629e2eee10731014f7cc9ecce63d3a9ac/tensorflow/python/summary/writer/writer.py#L124

This might help so they store width height and different png as an array of string that's it So we load multiple images and then store [ dimesnion , png0, png1 ] as tensor of string and then log that value as normal tensor of string with apt tags that's it

Yeah this is worth trying I think. But I don't understand what the dimension should be, especially if these images are of different sizes. According to this comment, it seems that all the images must have the same width and height.

[RustingSword]

How should I log a string array ? Could you share a snippet of code ?

In binary or text format? This is for text string.

[jeffin143]

In binary or text format?

Binary format because the encoded image string is not in utf-8 format and hence we need to log binary string array

According to this comment, it seems that all the images must have the same width and height.

Yes exactly the images cannot be of different size to be logged .

[RustingSword]

I think it's OK to assign a binary string to tensor->add_string_val(). The problem here is how to display this as image. I tried concat [width, height, encoded_image] as a binary string (though not sure if I did it correctly), and set plugin_name to "images", but it didn't work.

[jeffin143]

It worked :)

Here is the following snippet of code

.h file

    int add_texts(const std::string &tag, int step, std::vector<std::string> text);

.cc file

int TensorBoardLogger::add_texts(const string &tag, int step, vector<std::string> text) {
    auto *plugin_data = new SummaryMetadata::PluginData();
    plugin_data->set_plugin_name("images");

    auto *meta = new SummaryMetadata();
    meta->set_allocated_plugin_data(plugin_data);

    auto *tensor = new TensorProto();
    tensor->set_dtype(tensorflow::DataType::DT_STRING);
    for (size_t i = 0; i < text.size(); ++i) {
        tensor->add_string_val(text[i]);
    }

    auto *summary = new Summary();
    auto *v = summary->add_value();
    v->set_tag(tag);
    v->set_allocated_tensor(tensor);
    v->set_allocated_metadata(meta);

    return add_event(step, summary);
}

.test file

vector<string> temp = {"512","512",image,image2};
logger.add_texts("image as text",1,temp);

[RustingSword]

Great! It looks like these images can have different sizes, so the width and height has no effect. I tried to set them both to 0, it still works, kind of strange.

Care to submit a PR so I can merge it?

I think the interface can be changed to something like this to make it clearer:

// .h
int add_images(const std::string &tag, int step, int height, int width,
               const std::vector<std::string> &encoded_images,
               const std::string &display_name = "",
               const std::string &description = ""
);

// .cc

// ...
meta->set_display_name(display_name);
meta->set_summary_description(description);
// ...
tensor->add_string_val(to_string(width));
tensor->add_string_val(to_string(height));
for (const auto &image: encoded_images)
    tensor->add_string_val(image);
// ...

[jeffin143]

@RustingSword Any plans on adding support for pr curve or projector boards for embeddings

https://github.com/tensorflow/tensorboard/blob/8277e9f8d50758bb4d2b7a0fce1cab7e736b3dd2/tensorboard/plugins/pr_curve/metadata.py#L27

https://github.com/tensorflow/tensorboard/blob/8277e9f8d50758bb4d2b7a0fce1cab7e736b3dd2/tensorboard/plugins/projector/metadata.py#L21

[RustingSword]

The remaining plugins seem more complex to implement, currently I haven't come up with a good plan with them. I may need some time to think about it, so no clear timeline. Any input is highly appreciated.

[jeffin143]

The remaining plugins seem more complex to implement,

@RustingSword I have been working on it, I am not sure why even after generating the embedding config file the tensorbaord is not visualising it

Here is the issued raised : https://github.com/tensorflow/tensorboard/issues/3671#issue-623938306

Mxnet uses the same logic : https://github.com/reminisce/tensorboard-mxnet-logger/blob/09c5498426ca4ee52a0da500bd1171db12c1bf36/tensorboardX/embedding.py#L38

So I am assuming if have tesnors.tsv and metadata.tsv , it should work but no clue

ps : Also I am trying to under the math or logic behind PR curve, once I am done I guess it isn't much complex to implement

[RustingSword]

Great, however I don't have much time to fiddle with this until weekend. I think you can try this demo which successfully shows up in tensorboard projector page, and compare the difference with your generated files.

[RustingSword]

This works:

In [1]: import numpy as np
In [2]: np.savetxt('demo/vec.tsv', np.random.random((1000, 16)), delimiter="\t")
In [3]: np.savetxt('demo/label.tsv', np.arange(1000))

> ls demo
label.tsv              projector_config.pbtxt vec.tsv
> cat demo/projector_config.pbtxt
embeddings {
    tensor_path: "vec.tsv"
    metadata_path: "label.tsv"
}
> tensorboard --logdir demo

[jeffin143]

huh, I wasted an entire day, because I gave wrong path for tensor_path and meta data path

instead of ../assets/ it was ./assets

Sorry for being dumb, I have raised a PR for the support

I will take a look at the PR curve late during the week

[RustingSword]

That happens :-)

For efficiency's sake, we should save embedding in binary format, especially for large ones. However it seems the open source tensorboard only accepts tsv format file, while the standalone projector accepts binary file, it's not opensourced. So we currently can do nothing about this.

[jeffin143]

Hi @RustingSword , Hope you are doing fine, I am currently working out the math for pr-curve plotting support , it would be little while till I raise a pr so feel free to work if you want, If you don't beat me to it , I will raise a pr as soon as I finish. Currently It gives out some faulty result and hence I have to debug it

Also heads-up , I am a GSOC scholar and I would be working for @mlpack this summer and would be building a backend to log so that we could use tensorbaord to visualise , Here is the blog and design ideas we came up with during the community bonding period : https://www.mlpack.org/gsocblog/Jeffin2020CBP.html

Thanks for this wonderful repo and idea :)

[RustingSword]

Sure, no hurry, just take your time. I’m kind of busy with other things, and there are several remaining plugins worth implementing, such as hparams, so if I have time maybe I will try these.

I came across the GSoC ideas on mlpack, they are all very interesting, and congratulations! The design seems rather solid, asynchronism is a must for efficiency. I noticed that in the figure, main thread is responsible to create summary. I don’t know if this main thread is also doing the training work. I want to remind that some heavy operations, such as histogram summary, may take some time, this may block the main thread during this period.

Hope you are enjoying it :-)