search

kamalkraj / BERT-NER-TF

Named Entity Recognition with BERT using TensorFlow 2.0
Apache License 2.0
212 stars 71 forks source link

Support for Multi Cased Model #4

Closed dafs1 closed 4 years ago

dafs1 commented 4 years ago

I have tried to train a model with the Multi Cased model tensorflow but there is an AssertionError when the Checkpoints are loaded. I was wondering if this code supports Multi-cased model.

kamalkraj commented 4 years ago

@dafs1 Please post the complete stack trace.

ricsharma commented 4 years ago

Hi Kamal I have same issue on loading checkpoints. I have installed tensorflow-gpu=2.0. Pls help.

2019-12-13 06:57:58.597123: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2019-12-13 06:57:58.611380: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2019-12-13 06:57:58.612076: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:04.0 2019-12-13 06:57:58.612365: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2019-12-13 06:57:58.613676: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2019-12-13 06:57:58.615071: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2019-12-13 06:57:58.615423: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2019-12-13 06:57:58.617124: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2019-12-13 06:57:58.618428: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2019-12-13 06:57:58.622556: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2019-12-13 06:57:58.622676: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2019-12-13 06:57:58.623270: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2019-12-13 06:57:58.623828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2019-12-13 06:57:58.624129: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2019-12-13 06:57:58.629352: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz 2019-12-13 06:57:58.629537: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2dc4f40 executing computations on platform Host. Devices: 2019-12-13 06:57:58.629565: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Host, Default Version 2019-12-13 06:57:58.722542: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2019-12-13 06:57:58.723208: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2dc5100 executing computations on platform CUDA. Devices: 2019-12-13 06:57:58.723232: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0 2019-12-13 06:57:58.723407: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2019-12-13 06:57:58.723931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:04.0 2019-12-13 06:57:58.723984: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2019-12-13 06:57:58.724001: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2019-12-13 06:57:58.724014: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2019-12-13 06:57:58.724028: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2019-12-13 06:57:58.724040: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2019-12-13 06:57:58.724052: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2019-12-13 06:57:58.724065: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2019-12-13 06:57:58.724110: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2019-12-13 06:57:58.724620: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2019-12-13 06:57:58.725177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2019-12-13 06:57:58.725248: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2019-12-13 06:57:58.726331: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-12-13 06:57:58.726356: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2019-12-13 06:57:58.726366: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2019-12-13 06:57:58.726464: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2019-12-13 06:57:58.727030: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2019-12-13 06:57:58.727527: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0. 2019-12-13 06:57:58.727559: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0) Traceback (most recent call last): File "/content/gdrive/My Drive/berttf/run_nercolab.py", line 515, in main() File "/content/gdrive/My Drive/berttf/run_nercolab.py", line 359, in main ner = BertNer(args.bert_model, tf.float32, num_labels, args.max_seq_length) File "/content/gdrive/My Drive/berttf/model.py", line 41, in init checkpoint.restore(init_checkpoint).assert_existing_objects_matched() File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/util.py", line 958, in assert_existing_objects_matched return self.assert_consumed() File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/util.py", line 943, in assert_consumed "".join(unused_attribute_strings))) AssertionError: Some objects had attributes which were not restored: <tf.Variable 'bert_model/word_embeddings/embeddings:0' shape=(28996, 768) dtype=float32, numpy= array([[-0.00047125, -0.00392444, -0.01718481, ..., 0.00761576, -0.01668709, -0.00266492], [-0.03945672, -0.03438417, 0.00060767, ..., -0.00086753, -0.0229597 , -0.01650486], [-0.01777249, 0.00659098, 0.00273569, ..., -0.02210987, 0.00029147, 0.02218284], ..., [-0.02312314, -0.03112865, 0.00977948, ..., -0.00647722, 0.01329653, -0.00929082], [-0.01568137, -0.01355687, -0.02006243, ..., -0.01109568, 0.03311502, -0.0019628 ], [ 0.00439565, -0.02604441, 0.0313138 , ..., 0.02466861, 0.00381696, 0.01435717]], dtype=float32)>: ['bert_model/word_embeddings/embeddings'] <tf.Variable 'bert_model/embedding_postprocessor/type_embeddings:0' shape=(2, 768) dtype=float32, numpy= array([[ 0.00173585, -0.00327145, 0.00719522, ..., 0.02184234, -0.01153607, -0.03336546], [-0.02611437, 0.00967791, 0.01031132, ..., 0.02880269, 0.01059028, 0.01333835]], dtype=float32)>: ['bert_model/embedding_postprocessor/type_embeddings'] <tf.Variable 'bert_model/embedding_postprocessor/position_embeddings:0' shape=(512, 768) dtype=float32, numpy= array([[-3.6883123e-02, 5.6215911e-03, -1.5609710e-02, ..., -9.8099494e-03, -2.4294211e-02, 1.3280443e-02], [ 8.2727615e-03, 2.4059916e-02, 1.9387554e-02, ..., 8.9767986e-05, -1.6971119e-02, -7.2881440e-03], [ 1.8983334e-02, -3.6079798e-02, 1.0769339e-02, ..., -2.8570689e-02, 9.8243924e-03, 3.6343865e-02], ..., [-2.2436766e-02, -2.0206356e-02, -1.2461357e-02, ..., 1.8608063e-02, -2.8008332e-02, 7.2287582e-03], [ 5.5570775e-03, 1.5468974e-02, 3.9679483e-02, ..., 1.6238807e-02, -1.4573895e-02, 3.0967349e-02], [-1.0606544e-04, 8.0070049e-03, -2.7877258e-02, ..., 1.7902822e-03, 2.4845375e-02, -1.3735524e-03]], dtype=float32)>: ['bert_model/embedding_postprocessor/position_embeddings'] <tf.Variable 'bert_model/pooler_transform/kernel:0' shape=(768, 768) dtype=float32, numpy= array([[-0.0186411 , 0.0064633 , 0.03519246, ..., 0.01129883, 0.0120767 , 0.01176439], [ 0.00984347, -0.01713336, 0.00414206, ..., -0.01745014, -0.00189776, 0.00987592], [-0.00576746, 0.0247864 , 0.00093574, ..., 0.01094402, 0.03005101, 0.00610278], ..., [-0.0061362 , -0.01489993, -0.01143325, ..., 0.00476282, 0.01582425, 0.01931809], [ 0.00599172, 0.00246412, -0.00416973, ..., 0.02900079, -0.02143396, -0.01735855], [-0.0240486 , 0.00501571, 0.03354624, ..., 0.00020407, 0.00241491, -0.01336275]], dtype=float32)>: ['bert_model/pooler_transform/kernel']

kamalkraj commented 4 years ago

@dafs1 @ric1910 Could you share the link to weights to the Multi-cased model?

ricsharma commented 4 years ago

I download model from here https://github.com/google-research/bert bert-base-cased and named that folder as bert-base-cased

kamalkraj commented 4 years ago

@ric1910 Download the weights from here

fadybaly commented 4 years ago

I have pretrained my own bert on tf1. I tried converting it to tf2 using google's converter but it didn't work. Do you have a way to convert checkpoints to the format that works with your script?

niranjan8129 commented 4 years ago

@fadybaly folow the colab steps https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/guide/upgrade.ipynb#scrollTo=wfHOhbkgvrKr

fadybaly commented 4 years ago

@niranjan8129 this updates the script to tf2, not the checkpoint. it took a lot of resources to train the bert model from scratch. not sure i can do it again. do u have any way to convert the checkpoint from tf1 to tf2?

kamalkraj commented 4 years ago

You can refer to https://github.com/kamalkraj/ALBERT-TF2.0 . Here I converted tf hub weights to TF2 weights.

ibrahimishag commented 4 years ago

You can refer to https://github.com/kamalkraj/ALBERT-TF2.0 . Here I converted tf hub weights to TF2 weights.

Dear @kamalkraj , I am wondering about a tool to convert TF 1.x weights(not TF-Hub) to TF2.

Thanks in advance.

ibrahimishag commented 4 years ago

@niranjan8129 this updates the script to tf2, not the checkpoint. it took a lot of resources to train the bert model from scratch. not sure i can do it again. do u have any way to convert the checkpoint from tf1 to tf2?

Dear @fadybaly , Were you able to convert the TF1.x's checkpoints to TF2.x's pre-trained weights ? If you so, please share the method you've used!.

Thanks in advance.