search

aws-neuron / aws-neuron-sdk

Powering AWS purpose-built machine learning chips. Blazing fast and cost effective, natively integrated into PyTorch and TensorFlow and integrated with your favorite AWS services
https://aws.amazon.com/machine-learning/neuron/
Other
460 stars 153 forks source link

Error while following the TF1 Bert tutorial #381

Closed Gabriel4256 closed 2 years ago

Gabriel4256 commented 2 years ago

Hi, team. I have some trouble following the TF 1 Bert tutorial here.

Here are my execution environments:

I tried to use pretrained model from https://github.com/google-research/bert, and followed the instructions here.

After setting all the other things properly, I exeucuted the example usage script in the tutorial, which gave me the following result:

WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

2022-01-04 04:00:39.198511: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2022-01-04 04:00:39.253927: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2999990000 Hz
2022-01-04 04:00:39.254247: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5589a3b59130 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-01-04 04:00:39.254282: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/aws_neuron_tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/contrib/predictor/saved_model_predictor.py:153: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
2022-01-04 04:00:46.407554: I tensorflow/core/grappler/devices.cc:60] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA support)
2022-01-04 04:00:46.407733: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2022-01-04 04:00:47.765241: I tensorflow/neuron/grappler/convert/segment.cc:456] There are 1639 ops of 36 different types in the graph that are not compiled by neuron-cc: LogSoftmax, GreaterEqual, RandomUniform, Tanh, ArgMax, Pow, Softmax, BatchMatMul, Fill, Cast, Mul, SquaredDifference, Mean, RealDiv, Transpose, Slice, LessEqual, ExpandDims, Sub, Const, Pack, GatherV2, NoOp, MatMul, BiasAdd, Shape, StridedSlice, Rsqrt, Reshape, Identity, Assert, Placeholder, OneHot, Squeeze, Add, All, (For more information see https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/neuron-cc-ops/neuron-cc-ops-tensorflow.html).
2022-01-04 04:00:47.780154: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:786] Optimization results for grappler item: graph_to_optimize
2022-01-04 04:00:47.780198: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788]   aws_neuron_static_shape_inference: Graph size after: 1638 nodes (0), 1934 edges (0), time = 84.051ms.
2022-01-04 04:00:47.780209: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788]   aws_neuron_fuse_supported_operators: Graph size after: 1638 nodes (0), 1934 edges (0), time = 24.706ms.
INFO:tensorflow:Number of operations in TensorFlow session: 8830
INFO:tensorflow:Number of operations after tf.neuron optimizations: 1638
INFO:tensorflow:Number of operations placed on Neuron runtime: 0
WARNING:tensorflow:Converted /home/ubuntu/AWS_Neuron_scripts/tensorflow/bert-2 to ./bert-saved-model-neuron but no operator will be running on AWS machine learning accelerators. This is probably not what you want. Please refer to https://github.com/aws/aws-neuron-sdk for current limitations of the AWS Neuron SDK. We are actively improving (and hiring)!
{'OnNeuronRatio': 0.0}

According to the message, it seems like no operator is compiled for Inferentia. Is this intended? Please let me know if I am missing something.

And this is the result of pip list:

Package                            Version
---------------------------------- ---------------------------
absl-py                            1.0.0
alabaster                          0.7.12
anyio                              3.3.2
argh                               0.26.2
argon2-cffi                        20.1.0
arrow                              1.2.1
asn1crypto                         1.4.0
astor                              0.8.1
astroid                            2.8.0
astropy                            4.1
async-generator                    1.10
atomicwrites                       1.4.0
attrs                              21.2.0
autopep8                           1.5.6
autovizwidget                      0.19.1
awscli                             1.22.17
Babel                              2.9.1
backcall                           0.2.0
backports.functools-lru-cache      1.6.4
backports.shutil-get-terminal-size 1.0.0
beautifulsoup4                     4.10.0
bert-tensorflow                    1.0.1
binaryornot                        0.4.4
bitarray                           2.0.1
bkcharts                           0.2
black                              21.12b0
bleach                             4.1.0
bokeh                              2.3.3
boto                               2.49.0
boto3                              1.20.17
botocore                           1.23.17
Bottleneck                         1.3.2
brotlipy                           0.7.0
cached-property                    1.5.2
cachetools                         4.2.4
certifi                            2021.10.8
cffi                               1.14.6
chardet                            4.0.0
charset-normalizer                 2.0.9
click                              7.1.2
cloudpickle                        2.0.0
clyent                             1.2.2
colorama                           0.4.3
contextlib2                        21.6.0
contextvars                        2.4
cookiecutter                       1.7.0
coverage                           6.2
cryptography                       35.0.0
cycler                             0.11.0
Cython                             0.29.24
cytoolz                            0.11.0
dask                               2021.3.0
dataclasses                        0.8
decorator                          5.1.0
defusedxml                         0.7.1
diff-match-patch                   20200713
dill                               0.3.4
distributed                        2021.3.0
dmlc-nnvm                          1.8.0.0+0
dmlc-topi                          1.8.0.0+0
dmlc-tvm                           1.8.0.0+0
docutils                           0.15.2
dparse                             0.5.1
entrypoints                        0.3
environment-kernels                1.1.1
et-xmlfile                         1.0.1
fastcache                          1.1.0
filelock                           3.4.0
flake8                             3.8.4
Flask                              2.0.2
Flask-Cors                         3.0.10
fsspec                             2021.11.1
future                             0.18.2
gast                               0.2.2
gevent                             21.1.2
glob2                              0.7
gmpy2                              2.1.0b5
google-auth                        1.35.0
google-auth-oauthlib               0.4.6
google-pasta                       0.2.0
greenlet                           1.1.0
grpcio                             1.43.0
gssapi                             1.7.2
h5py                               2.10.0
hdijupyterutils                    0.19.1
HeapDict                           1.0.1
html5lib                           1.1
idna                               3.3
imagecodecs-lite                   2019.12.3
imageio                            2.13.1
imagesize                          1.3.0
immutables                         0.15
importlib-metadata                 4.8.3
inferentia-hwm                     1.8.0.0+0
inflection                         0.5.1
iniconfig                          1.1.1
intervaltree                       3.0.2
ipykernel                          5.5.5
ipyparallel                        8.0.0
ipython                            7.16.1
ipython-genutils                   0.2.0
ipywidgets                         7.6.5
islpy                              2018.2+aws2018.x.870.0.bld0
isort                              5.10.1
itsdangerous                       2.0.1
jdcal                              1.4.1
jedi                               0.17.2
jeepney                            0.7.1
Jinja2                             3.0.3
jinja2-time                        0.2.0
jmespath                           0.10.0
joblib                             1.1.0
json5                              0.9.5
jsonschema                         4.1.2
jupyter                            1.0.0
jupyter-client                     7.1.0
jupyter-console                    6.4.0
jupyter-core                       4.8.1
jupyter-server                     1.13.1
jupyterlab                         3.2.5
jupyterlab-pygments                0.1.2
jupyterlab-server                  2.9.0
jupyterlab-widgets                 1.0.2
Keras                              2.2.4
Keras-Applications                 1.0.8
keras-mxnet                        2.2.4.2
Keras-Preprocessing                1.1.2
keyring                            23.2.1
kiwisolver                         1.3.1
krb5                               0.2.0
lazy-object-proxy                  1.6.0
libarchive-c                       3.1
llvmlite                           0.36.0
locket                             0.2.0
lxml                               4.7.1
Markdown                           3.3.6
MarkupSafe                         2.0.1
matplotlib                         3.3.4
mccabe                             0.6.1
mistune                            0.8.4
mkl-fft                            1.3.0
mkl-random                         1.2.0
mkl-service                        2.3.0
mock                               4.0.3
more-itertools                     8.12.0
mpmath                             1.2.1
msgpack                            1.0.2
multipledispatch                   0.6.0
multiprocess                       0.70.12.2
mypy-extensions                    0.4.3
nbclassic                          0.3.4
nbclient                           0.5.9
nbconvert                          6.0.7
nbformat                           5.1.3
nest-asyncio                       1.5.4
networkx                           2.4
neuron-cc                          1.8.2.0+5b63f6edb
nltk                               3.6.6
nose                               1.3.7
notebook                           6.4.6
numba                              0.53.1
numexpr                            2.7.3
numpy                              1.19.5
numpydoc                           1.1.0
oauthlib                           3.1.1
olefile                            0.46
opencv-python                      4.5.1.48
openpyxl                           3.0.9
opt-einsum                         3.3.0
packaging                          21.3
pandas                             1.1.5
pandocfilters                      1.5.0
parso                              0.7.0
partd                              1.2.0
path                               16.2.0
pathlib2                           2.3.6
pathos                             0.2.8
pathspec                           0.9.0
pathtools                          0.1.2
patsy                              0.5.2
pep8                               1.7.1
pexpect                            4.8.0
pickleshare                        0.7.5
Pillow                             8.4.0
pip                                21.3.1
pkginfo                            1.8.2
platformdirs                       2.3.0
plotly                             5.5.0
pluggy                             1.0.0
ply                                3.11
pox                                0.3.0
poyo                               0.5.0
ppft                               1.6.6.4
prometheus-client                  0.12.0
prompt-toolkit                     3.0.24
protobuf                           3.19.1
protobuf3-to-dict                  0.1.5
psutil                             5.8.0
psycopg2                           2.7.5
ptyprocess                         0.7.0
py                                 1.11.0
pyasn1                             0.4.8
pyasn1-modules                     0.2.8
pycodestyle                        2.6.0
pycosat                            0.6.3
pycparser                          2.21
pycryptodome                       3.12.0
pycurl                             7.43.0.6
pydocstyle                         6.1.1
pyflakes                           2.2.0
pygal                              2.4.0
Pygments                           2.10.0
pyinstrument                       3.4.2
pyinstrument-cext                  0.2.4
pykerberos                         1.2.1
pylint                             2.11.1
pyls-black                         0.4.6
pyls-spyder                        0.3.2
pyodbc                             4.0.31
pyOpenSSL                          21.0.0
pyparsing                          3.0.6
PyQt5                              5.12.3
PyQt5_sip                          4.19.18
PyQtChart                          5.12
PyQtWebEngine                      5.12.1
pyrsistent                         0.17.3
PySocks                            1.7.1
pyspnego                           0.3.1
pytest                             6.2.5
python-dateutil                    2.8.2
python-jsonrpc-server              0.4.0
python-language-server             0.36.2
pytz                               2021.3
PyWavelets                         1.1.1
pyxdg                              0.27
PyYAML                             6.0
pyzmq                              22.1.0
QDarkStyle                         3.0.2
qstylizer                          0.2.1
QtAwesome                          1.1.1
qtconsole                          5.2.2
QtPy                               1.11.3
regex                              2021.8.3
requests                           2.27.0
requests-kerberos                  0.14.0
requests-oauthlib                  1.3.0
rope                               0.22.0
rsa                                4.8
Rtree                              0.9.7
ruamel-yaml-conda                  0.15.80
s3transfer                         0.5.0
safety                             1.10.3
sagemaker                          2.70.0
scikit-image                       0.17.2
scikit-learn                       0.24.2
scipy                              1.4.1
seaborn                            0.11.2
SecretStorage                      3.3.1
Send2Trash                         1.8.0
setuptools                         59.6.0
shap                               0.40.0
simplegeneric                      0.8.1
singledispatch                     0.0.0
sip                                4.19.25
six                                1.16.0
slicer                             0.0.7
smdebug                            1.0.12
smdebug-rulesconfig                1.0.1
sniffio                            1.2.0
snowballstemmer                    2.2.0
sortedcollections                  2.1.0
sortedcontainers                   2.4.0
soupsieve                          2.3.1
sparkmagic                         0.15.0
Sphinx                             4.3.2
sphinxcontrib-applehelp            1.0.2
sphinxcontrib-devhelp              1.0.2
sphinxcontrib-htmlhelp             2.0.0
sphinxcontrib-jsmath               1.0.1
sphinxcontrib-qthelp               1.0.3
sphinxcontrib-serializinghtml      1.1.5
sphinxcontrib-websupport           1.2.4
spyder                             5.0.5
spyder-kernels                     2.0.5
SQLAlchemy                         1.4.22
statsmodels                        0.12.2
sympy                              1.8
tables                             3.6.1
tblib                              1.7.0
tenacity                           8.0.1
tensorboard                        2.4.0
tensorboard-data-server            0.6.1
tensorboard-plugin-neuron          2.2.0.0
tensorboard-plugin-wit             1.8.0
tensorflow                         1.15.5
tensorflow-estimator               1.15.1
tensorflow-hub                     0.12.0
tensorflow-neuron                  1.15.5.2.0.4.0
tensorflow-serving-api             1.15.0
termcolor                          1.1.0
terminado                          0.12.1
testpath                           0.5.0
textdistance                       4.2.2
threadpoolctl                      3.0.0
three-merge                        0.1.1
tifffile                           2020.6.3
tinycss2                           1.1.1
toml                               0.10.2
tomli                              1.2.2
toolz                              0.11.2
tornado                            6.1
tqdm                               4.62.3
traitlets                          4.3.3
typed-ast                          1.4.3
typing_extensions                  4.0.1
ujson                              4.0.2
unicodecsv                         0.14.1
urllib3                            1.26.7
watchdog                           2.1.6
wcwidth                            0.2.5
webencodings                       0.5.1
websocket-client                   1.2.3
Werkzeug                           2.0.2
wheel                              0.37.1
whichcraft                         0.6.1
widgetsnbextension                 3.5.1
wrapt                              1.12.1
wurlitzer                          3.0.2
xlrd                               2.0.1
XlsxWriter                         3.0.2
xlwt                               1.3.0
yapf                               0.31.0
zict                               2.0.0
zipp                               3.6.0
zope.event                         4.5.0
zope.interface                     5.4.0

Thanks in advance.

aws-joshim commented 2 years ago

@Gabriel4256 the no operator warning that you see above is likely the result of an unsuccessful compilation of the model. From the description above, it looks like you are compiling the BERT model on a Inf1 instance. For a BERT large model, we recommend that you run the compilation on a a c5.4xlarge instance as described in the tutorial - https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/neuron-frameworks/tensorflow-neuron/tutorials/bert_demo/bert_demo.html#tensorflow-bert-demo under the Launch instances tab. From the description above, it looks like you are compiling the BERT model on a Inf1 instance. Please let us know if addresses your issue

Gabriel4256 commented 2 years ago

@aws-joshim Execution on c5.4xlarge instance derives almost same result: 

WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

2022-01-05 02:42:24.876594: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2022-01-05 02:42:24.897271: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2999995000 Hz
2022-01-05 02:42:24.897869: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5654d52be700 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-01-05 02:42:24.897892: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/aws_neuron_tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/contrib/predictor/saved_model_predictor.py:153: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
2022-01-05 02:42:40.276883: I tensorflow/core/grappler/devices.cc:60] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA support)
2022-01-05 02:42:40.277032: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2022-01-05 02:42:42.047896: I tensorflow/neuron/grappler/convert/segment.cc:456] There are 1639 ops of 36 different types in the graph that are not compiled by neuron-cc: LogSoftmax, GreaterEqual, RandomUniform, Tanh, ArgMax, Pow, Softmax, BatchMatMul, Fill, Cast, Mul, SquaredDifference, Mean, RealDiv, Transpose, Slice, LessEqual, ExpandDims, Sub, Const, Pack, GatherV2, NoOp, MatMul, BiasAdd, Shape, StridedSlice, Rsqrt, Reshape, Identity, Assert, Placeholder, OneHot, Squeeze, Add, All, (For more information see https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/neuron-cc-ops/neuron-cc-ops-tensorflow.html).
2022-01-05 02:42:42.062828: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:786] Optimization results for grappler item: graph_to_optimize
2022-01-05 02:42:42.062863: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788]   aws_neuron_static_shape_inference: Graph size after: 1638 nodes (0), 1934 edges (0), time = 560.523ms.
2022-01-05 02:42:42.062870: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788]   aws_neuron_fuse_supported_operators: Graph size after: 1638 nodes (0), 1934 edges (0), time = 95.084ms.
INFO:tensorflow:Number of operations in TensorFlow session: 8830
INFO:tensorflow:Number of operations after tf.neuron optimizations: 1638
INFO:tensorflow:Number of operations placed on Neuron runtime: 0
WARNING:tensorflow:Converted /home/ubuntu/AWS_Neuron_scripts/tensorflow/bert-2 to ./bert-saved-model-neuron but no operator will be running on AWS machine learning accelerators. This is probably not what you want. Please refer to https://github.com/aws/aws-neuron-sdk for current limitations of the AWS Neuron SDK. We are actively improving (and hiring)!
{'OnNeuronRatio': 0.0}

It might be the problem of the Bert model I used. I created the model using the example on google bert github (https://colab.research.google.com/github/google-research/bert/blob/master/predicting_movie_reviews_with_bert_on_tf_hub.ipynb).

After training, I just added the following code for saving model in a SavedModel format:

    features = {
        "input_ids": tf.placeholder(shape=[None, FLAGS.max_seq_length], dtype=tf.int32, name='input_ids'),
        "input_mask": tf.placeholder(shape=[None, FLAGS.max_seq_length], dtype=tf.int32, name='input_mask'),
        "segment_ids": tf.placeholder(shape=[None, FLAGS.max_seq_length], dtype=tf.int32, name='segment_ids'),
        "label_ids": tf.placeholder(shape=[None], dtype=tf.int32, name='label_ids'),
        "is_real_example": tf.placeholder(shape=[None], dtype=tf.int32, name='is_real_example'),
    }
    serving_input_fn = tf.estimator.export.build_raw_serving_input_receiver_fn(features)
    estimator._export_to_tpu = False
    estimator.export_saved_model(
        export_dir_base='./bert_classifier_saved_model',
        serving_input_receiver_fn=serving_input_fn)

Please let me know if I am missing something. Also, I would appreciate if you share pretrained Bert model that is compiled well with Neuron.

aws-joshim commented 2 years ago

We looked at the attached Colab notebook content. The reason for the incompatible saved model format is that the model from tensorflow-hub contains different operator names, causing compilation to fail. We also noticed that the attached Jupyter notebook points to a tensorflow 2.x tutorial, while the compilation steps mentioned above use the tensorflow-neuron 1.x script. The following TF 2.x compilation scripts with the tensorflow-neuron 2.x version could address your compilation issue -

tf 2.x: bert_trace.py

import argparse import tensorflow as tf import tensorflow.neuron as tfn

def main(): parser = argparse.ArgumentParser() parser.add_argument('--input_saved_model', required=True, help='Original SaveModel') parser.add_argument('--output_saved_model', required=True, help='Output SavedModel that runs on Inferentia') parser.add_argument('--batch_size', type=int, default=1) parser.add_argument('--sequence_length', type=int, default=128) args = parser.parse_args() model = tf.saved_model.load(args.input_saved_model) wfunc = model.signatures['serving_default'] input_ids = tf.zeros([args.batch_size, args.sequence_length], dtype=tf.int32) input_mask = tf.zeros([args.batch_size, args.sequence_length], dtype=tf.int32) segment_ids = tf.zeros([args.batch_size, args.sequence_length], dtype=tf.int32) is_real_example = tf.zeros([args.batch_size], dtype=tf.int32) label_ids = tf.zeros([args.batch_size], dtype=tf.int32) print(wfunc) # to see it's calling signature example_inputs = [input_ids, input_mask, is_real_example, label_ids, segment_ids] wfunc_neuron = tfn.trace(wfunc, example_inputs) signatures = {'serving_default': wfunc_neuron.aws_neuron_function} tf.saved_model.save(model, args.output_saved_model, signatures)

if name == 'main': main()

Gabriel4256 commented 2 years ago

It works fine, thank you.

Gabriel4256 commented 2 years ago

Is there any way to compile and run Bert-large model on TF1? I need this because some profiling tools only work on TF 1. I've tried some bert models, but most of the operations were compiled to run on CPU as shown below:

INFO:tensorflow:Number of operations in TensorFlow session: 8830
INFO:tensorflow:Number of operations after tf.neuron optimizations: 1638
INFO:tensorflow:Number of operations placed on Neuron runtime: 26

For this, I've downloaded checkpoint file from google bert github, converted it to TF1 SavedModel forat using the following code:

with tf.Session(graph=tf.Graph()) as sess:
  # Initialize v1 since the saver will not.
    segment_ids = tf.saved_model.utils.build_tensor_info(tf.constant(np.zeros((1,128))))
    input_ids = tf.saved_model.utils.build_tensor_info(tf.constant(np.zeros((1,128))))
    input_mask = tf.saved_model.utils.build_tensor_info(tf.constant(np.zeros((1,128))))
    # label_ids = tf.saved_model.utils.build_tensor_info(tf.constant(np.zeros((1))))
    label = tf.saved_model.utils.build_tensor_info(tf.constant(np.zeros((1))))

    loader = tf.compat.v1.train.import_meta_graph('/home/ubuntu/models/wwm_uncased_L-24_H-1024_A-16/bert_model.ckpt.meta')
    loader.restore(sess, "/home/ubuntu/models/wwm_uncased_L-24_H-1024_A-16/bert_model.ckpt")
    shutil.rmtree("./saved_model_test", ignore_errors=True)
    builder = tf.compat.v1.saved_model.builder.SavedModelBuilder("./saved_model_test")

    signature_def = tf.compat.v1.saved_model.build_signature_def(
        inputs={
            "segment_ids": segment_ids,
            "input_ids": input_ids,
            "input_mask": input_mask,
        },
        outputs={"label": label})
    builder.add_meta_graph_and_variables(sess,
                                         [tf.saved_model.SERVING],
                                         signature_def_map={
                                              tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY: signature_def},
                                         strip_default_attrs=True)
    builder.save()

And fianlly, I compiled it with the following code:

with tf.Session(graph=tf.Graph()) as sess:
    tf.saved_model.loader.load(sess, ['serve'], MODEL_DIR)
    result = tfn.saved_model.compile(
        args.input_saved_model, args.output_saved_model,
    )

I've tried this on c5.4xlarge instance and all the other things are same as the earlier post.

Thanks in advance.

aws-owinop commented 2 years ago

Hi @Gabriel4256, to compile TF BERT large model you will need to follow the steps here. The default compile script doesn't work in this case.

jeffhataws commented 2 years ago

Hi @Gabriel4256, please let us know if you still have problem with compiling TF BERT large model after following the steps mentioned in the previous post. Thanks.