search

tensorflow / tfjs

A WebGL accelerated JavaScript library for training and deploying ML models.
https://js.tensorflow.org
Apache License 2.0
18.46k stars 1.92k forks source link

Allow user to insert custom metadata into the model.json file #3987

Closed pyu10055 closed 3 years ago

pyu10055 commented 4 years ago

Please make sure that this is a feature request. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template

System information

Describe the feature and the current behavior/state. User would like to insert custom defined metadata information into the model, currently the model definition contains the userDefinedMetadata field that can be used for this purpose. potentially user can specify an external file that contains the metadata information, which allow user to include metadata that is not json compatible. Will this change the current api? How?

  1. The python converter command line should provide a new parameter to inserting user defined metadata.
  2. The graph model class should expose the field. Who will benefit with this feature?

Any Other info.

plakal commented 4 years ago

A common need for any classifier model is a mapping between classifier outputs and class names. E.g., for our YAMNet model on TF-Hub, the TF2 SavedModel includes the "class map" as an asset and this can be looked up to annotate predictions, as shown in the usage https://tfhub.dev/google/yamnet/1 and we would like to do the same for the TF-JS version https://tfhub.dev/google/tfjs-model/yamnet/tfjs/1

Related issues: https://github.com/tensorflow/tfjs/issues/762 https://github.com/tensorflow/tfjs/issues/1596

One thought about using userDefinedMetadata: Inspecting model.json shows that this field is currently being used to hold a "signature". Is this only for informational purposes or does anything in TF-JS rely on the signature being present here? If we want to transport additional metadata through this field, we would need to merge whatever the user supplies with whatever is already present. I assume we want userDefinedMetadata to be a mapping from string keys to arbitrary json values, and the user can supply an arbitrary number of json files to include?

pyu10055 commented 4 years ago

@plakal Thank you for pointing out that signature is currently in the userDefinedMetadata. Signature is used by the GraphModel class, it would be better moved out of userDefinedMetadata field, and leave the userDefinedMetadata open.

lina128 commented 4 years ago

Hi @plakal Thank you for providing the YAMNET use case, I think it definitely fits the purpose of this proposal. A bit of more detail of what we are imagining to support: 1. During conversion, user can provide an arbitrary number of json files, which converter can digest and store in userDefinedMetadata field, each json object keyed by a user provided name. 2. In client side, during graph load, tfjs library will load the field in memory and return as part of the model. Then user can access the object through the model. Also thank you for pointing out the signature field, as Ping said, the signature is converted from SavedModel and is used to distill the inference graph for execution. And this field is not supposed to be accessible or mutated, so we should move this out.

plakal commented 4 years ago

Sounds great, thanks!

plakal commented 3 years ago

Thanks for the fast work! Is there any documentation for the new feature and how to use it when converting a model to TF-JS? I'd like to try this with YAMNet.

lina128 commented 3 years ago

Hi @plakal, we just merged it to master, it will be released with version 2.8.0, we are targeting a release next week. For the time being, if you want to try it out, you can check out master, and use the python converter from your local directory. I will update the readme once the 2.8.0 is released. For local testing for now, you can append this flag to the converter: --metadata=key:path/metadata.json. To provide multiple metadata files, separate each key/filepath pair with comma. And then after you load the model, you can access the metadata from model.metadata using the key, such as model.metadata['key'].