search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
14.69k stars 2.93k forks source link

[BERT-Squad][opset-10][models/20191107.zip] Expected values mismatch #6429

Open mbencer opened 3 years ago

mbencer commented 3 years ago

Describe the bug I've tried to run BERT-Squad (opset10) model (https://onnxruntimetestdata.blob.core.windows.net/models/20191107.zip) using test data. Unfortunately, I see a big mismatch (and incorrect order) between expected values from sample data and values produced by ONNX Runtime.

System information

To Reproduce

import onnx
import onnx.numpy_helper
import onnxruntime as rt
import numpy as np

base_path = "/home/mbencer/workspace/ort_bert/"
model_path = base_path + "bertsquad10.onnx"

sess = rt.InferenceSession(model_path)

input_names = [sess.get_inputs()[i].name for i in range(len(sess.get_inputs()))]
# order from bertsquad10.onnx
output_names = ["unique_ids:0","unstack:0", "unstack:1"]

input_data = []
for i in range(len(sess.get_inputs())):
    proto_bytes = open(base_path+"test_data_set_1/input_{0}.pb".format(i), 'rb').read()
    tensor_proto = onnx.TensorProto.FromString(proto_bytes)
    data_array = onnx.numpy_helper.to_array(tensor_proto)
    input_data.append(data_array)

inputs_map = dict(zip(input_names, input_data))
pred = sess.run(output_names, inputs_map)

or_result = {}
for idx, val in enumerate(output_names):
    or_result[val] = pred[idx]

for i, name in enumerate(output_names):
    proto_bytes = open(base_path+'test_data_set_1/output_{0}.pb'.format(i), 'rb').read()
    tensor_proto = onnx.TensorProto.FromString(proto_bytes)
    data_array = onnx.numpy_helper.to_array(tensor_proto)
    print("Result for output: " + name)
    print("Expected:")
    print(data_array)
    print("Produced by ONNX Runtime:")
    print(or_result[name])
    print("------------------------------------------------")

Current behavior

Result for output: unique_ids:0
Expected:
[[-7.7425637  -8.218501   -8.677254   -8.338841   -8.604311   -8.84387
  -8.490502   -8.915914   -8.343249   -8.624639   -6.689491   -8.629656
  -8.673133   -9.534565   -8.750434   -8.369956   -3.915541   -3.450283
  -8.575512   -6.2463503  -8.379851   -7.744997   -5.7037973  -8.414856
  -9.100826   -8.28799    -8.883646   -8.922359   -8.498017   -7.8939986
  -7.936869   -8.765843   -3.9077907  -2.378959   -8.08765    -8.550091
  -6.107754   -7.205675   -5.743803   -8.490921   -8.041068   -7.794489
  -7.4701595  -4.21051    -7.9159603  -8.322495   -7.760231   -8.875621
  -8.76782    -7.4637313  -6.838551   -6.980595   -7.7347927  -8.886888
  -8.3515835  -5.776036   -6.698425   -7.9757466  -5.939224   -6.98086
  -8.832632   -8.761497   -6.2840576  -8.651577   -0.38624665 -1.1095788
  -7.1796007  -8.154757   -5.484854   -7.3328424  -8.151014   -5.427039
  -6.2172637  -8.540557   -8.012108   -6.160336   -6.859819   -3.2133396
  -7.7328434  -4.021851   -4.46886     6.1962748   6.225349    1.1641924
  -5.0020266  -7.611444   -3.6928244  -8.6663065  -5.5468516  -6.397411
  -4.604154   -4.7050023  -5.5484204  -2.868484   -7.6685176  -7.2183456
  -9.700549   -8.397554   -4.849909   -8.610162   -9.048771   -9.320874
  -8.818169   -8.470439   -8.063076   -8.486222   -9.143743   -7.794301
  -7.4291115  -8.724903   -4.091894   -7.7702527  -7.9544425  -6.5347624
  -5.9644747  -3.9790697  -8.345175   -8.902779   -7.18646    -8.923271
  -8.736635   -9.007066   -9.152918   -8.927531   -8.61433    -8.459272
  -7.8229465  -9.22188    -9.427916   -8.81047    -8.621281   -8.321307
  -8.197027   -8.795549   -9.288285   -9.201286   -8.678082   -8.133057
  -8.762048   -8.61658    -8.701047   -8.50378    -8.572202   -8.803587
  -8.511148   -8.414716   -7.721463   -8.961927   -9.03054    -8.137675
  -9.261776   -9.075058   -8.061392   -8.903873   -9.1128025  -9.064353
  -7.894063   -8.762052   -9.405123   -9.220515   -8.823745   -8.334905
  -9.068795   -8.862873   -9.147915   -8.148087   -6.733486   -8.882837
  -8.7377205  -8.782916   -8.399412   -8.793529   -6.292643   -4.8278584
  -8.471405   -8.061337   -9.112238   -8.7103815  -8.6182575  -8.249628
  -8.718195   -7.626752   -8.812523   -9.339518   -8.821876   -6.765984
  -5.5538993  -7.815057   -9.302865   -8.807534   -8.766554   -8.396334
  -8.866269   -7.5073767  -8.850694   -9.272594   -8.997729   -8.350142
  -8.735088   -8.730509   -8.728538   -8.725588   -8.735129   -8.715272
  -8.722743   -8.733848   -8.718388   -8.703801   -8.734895   -8.720818
  -8.72258    -8.730037   -8.707461   -8.699065   -8.695888   -8.701864
  -8.712221   -8.702159   -8.703327   -8.720149   -8.720413   -8.726896
  -8.746185   -8.7527     -8.758233   -8.747547   -8.747094   -8.733433
  -8.738569   -8.742947   -8.733186   -8.7264185  -8.743622   -8.747217
  -8.741836   -8.756318   -8.752414   -8.768163   -8.77962    -8.785813
  -8.779377   -8.771755   -8.777394   -8.78261    -8.762546   -8.756755
  -8.760555   -8.7539425  -8.73433    -8.741919   -8.749946   -8.740309
  -8.758248   -8.766118   -8.754494   -8.772917  ]]
Produced by ONNX Runtime:
[1000012005]
------------------------------------------------
Result for output: unstack:0
Expected:
[[-5.8333063  -8.241654   -8.372095   -8.753174   -8.433429   -8.201076
  -8.4077215  -8.083273   -8.702093   -8.395958   -8.982073   -8.249475
  -8.257881   -6.838529   -8.404837   -6.6982617  -7.502687   -4.339758
  -7.6118245  -1.7457944  -8.419111   -8.876224   -5.92101    -6.882468
  -7.4276752  -8.448018   -7.9946823  -7.3726315  -8.473654   -9.010646
  -6.57988    -7.56125    -8.077414   -5.8430114  -5.8359594  -7.737444
  -1.4576873  -8.440942   -7.8824005  -5.6969256  -4.2151737  -8.463533
  -8.798064   -7.497704   -7.7127166  -7.9414034  -3.9817936  -5.391129
  -6.718227   -8.851458   -9.066548   -8.245631   -8.887319   -7.781917
  -8.440966   -7.956026   -5.2279973  -8.360157   -7.861549   -8.212669
  -6.549996   -8.132203   -4.490472   -6.4799995  -6.6693316  -5.8599453
  -4.844711   -7.201618    0.30820063 -8.257333   -7.817514   -8.91809
  -6.443946   -6.5059896  -8.511733   -7.5533133  -6.7756867  -6.4713206
  -8.767701   -2.6538937  -3.8820038  -2.6245267  -0.3698883   7.435213
  -3.1957757  -8.400886   -4.0937705  -8.012276   -7.3372784  -1.4405768
   3.6587121  -2.166864   -7.3477755  -2.4662204  -8.363328   -8.425817
  -6.1464415  -7.5566025  -6.267593   -7.0573287  -7.593878   -6.8491173
  -7.7484117  -1.8512489  -7.849536   -7.6438093  -7.5387273  -8.607834
  -7.6081147  -8.074454   -5.5638185  -7.559317   -2.479043   -1.072226
  -8.579926   -6.7233253  -6.265842   -7.7682304  -2.695749   -8.002887
  -8.074192   -7.9649544  -7.727623   -8.015128   -8.434099   -8.400776
  -6.710034   -6.63205    -6.910135   -8.103239   -8.385738   -8.549153
  -8.669522   -8.126712   -6.868259   -7.038781   -8.141724   -8.625942
  -8.247161   -8.383083   -8.048038   -8.470919   -8.413105   -8.04793
  -8.472281   -8.44648    -8.093593   -7.004929   -7.9282923  -8.429489
  -7.063168   -7.8639836  -8.012509   -7.598219   -6.801132   -7.880735
  -8.27971    -8.243292   -7.097034   -7.3409224  -8.200708   -8.326926
  -7.6698513  -6.643312   -7.086793   -8.693572   -8.666896   -6.1487703
  -7.21102    -8.205438   -6.6873527  -8.247451   -7.974248   -6.715651
  -7.5861974  -4.773685   -7.627546   -8.361283   -8.47842    -8.761732
  -8.34337    -8.396667   -6.901589   -7.1074615  -8.122676   -8.283627
  -6.983921   -4.2953987  -7.1135063  -8.221367   -8.321928   -8.604809
  -8.184777   -8.322093   -8.076208   -5.777829   -6.229177   -6.744565
  -8.435942   -8.440732   -8.439906   -8.441335   -8.433141   -8.449609
  -8.44306    -8.435972   -8.4504595  -8.459752   -8.439243   -8.449947
  -8.449005   -8.442657   -8.459524   -8.465898   -8.466421   -8.462572
  -8.452573   -8.458817   -8.457622   -8.441249   -8.44466    -8.439574
  -8.421152   -8.415532   -8.409076   -8.421948   -8.420881   -8.431703
  -8.427192   -8.423786   -8.434029   -8.440068   -8.420844   -8.41942
  -8.423738   -8.409506   -8.41262    -8.398481   -8.386405   -8.38201
  -8.3904     -8.397616   -8.392051   -8.388344   -8.410917   -8.4168
  -8.415849   -8.424467   -8.4403     -8.434374   -8.430928   -8.437987
  -8.424631   -8.420991   -8.431541   -8.416358  ]]
Produced by ONNX Runtime:
[[-0.9759766 -1.450721  -1.5763172 -1.5890969 -1.5351176 -1.5592333
  -1.4762229 -1.5235243 -1.5418369 -1.5519742 -1.5261447 -1.5356185
  -1.5496906 -1.5601956 -1.5637292 -1.5732459 -1.5009362 -1.5412742
  -1.5717137 -1.5537629 -1.5410321 -1.5377656 -1.5427722 -1.5282533
  -1.505671  -1.4524024 -1.546615  -1.515481  -1.5195574 -1.5088006
  -1.542704  -1.5814463 -1.6066822 -1.6429515 -1.6299806 -1.6232748
  -1.6548588 -1.7063512 -1.6978551 -1.6717566 -1.6935661 -1.7054031
  -1.7081916 -1.7241248 -1.6873819 -1.7056538 -1.6975089 -1.6892244
  -1.6723325 -1.6575607 -1.6184343 -1.6553632 -1.6192279 -1.6135063
  -1.6260265 -1.510229  -1.5394856 -1.583099  -1.5614667 -1.6019158
  -1.6194118 -1.6312293 -1.6577559 -1.623504  -1.6477847 -1.5991151
  -1.5939354 -1.622714  -1.6142517 -1.6058099 -1.5889214 -1.5808182
  -1.6010368 -1.6174303 -1.5824285 -1.5857712 -1.5999596 -1.5231024
  -1.5306231 -1.4671273 -1.4481268 -1.4969685 -1.494214  -1.452083
  -1.419344  -1.4229387 -1.4419068 -1.4853362 -1.4379315 -1.460971
  -1.471414  -1.4884704 -1.5203454 -1.4884084 -1.4794726 -1.519544
  -1.4958186 -1.5243486 -1.5139706 -1.5214247 -1.5702736 -1.6216162
  -1.6112769 -1.597801  -1.6337198 -1.5961083 -1.6229031 -1.6053306
  -1.6470095 -1.6540616 -1.6639413 -1.6726943 -1.6871973 -1.7179561
  -1.6748053 -1.717884  -1.7326006 -1.7490648 -1.766892  -1.7890314
  -1.8105596 -1.8182937 -1.8389627 -1.8328342 -1.845332  -1.8743113
  -1.8682914 -1.9282228 -1.921805  -1.9535471 -1.8306996 -1.9036961
  -1.9576998 -1.9329656 -1.9585017 -1.8955457 -1.891672  -1.9164375
  -1.9217926 -1.8955537 -1.9567021 -1.9678723 -2.007722  -1.9475445
  -1.9046175 -1.8934498 -1.9082874 -2.0030818 -2.1020648 -2.2066185
  -2.2834296 -2.2772322 -2.3245912 -2.3516495 -2.401715  -2.3750682
  -2.363668  -2.2740357 -2.3502886 -2.358609  -2.3020158 -2.225013
  -2.1204824 -2.1380496 -2.0943851 -1.9676083 -1.8211714 -1.6732813
  -1.6079019 -1.5448402 -1.4950548 -1.5528611 -1.486073  -1.4470421
  -1.3817142 -1.358766  -1.4102038 -1.4637333 -1.4379079 -1.5060511
  -1.5656558 -1.5718687 -1.5788442 -1.6153969 -1.6167845 -1.6238128
  -1.6404494 -1.6754898 -1.6709024 -1.6641167 -1.6194288 -1.6955898
  -1.6419479 -1.6337758 -1.5610831 -1.561306  -1.5620652 -1.6307055
  -1.6214684 -1.611353  -1.6211519 -1.6056811 -1.5862633 -1.5827228
  -1.5923761 -1.5674363 -1.6037403 -1.6359917 -1.5845059 -1.5680711
  -1.5853049 -1.5818526 -1.5813358 -1.6153027 -1.6256508 -1.5922209
  -1.5925366 -1.5854892 -1.6331828 -1.6142281 -1.6257747 -1.6343166
  -1.6140649 -1.6141883 -1.6488514 -1.6074402 -1.6743914 -1.6575767
  -1.6568912 -1.6608633 -1.6574987 -1.6324492 -1.6760123 -1.6561714
  -1.646965  -1.5736759 -1.6181663 -1.6341717 -1.6154071 -1.6514673
  -1.6447797 -1.6249369 -1.6595393 -1.6948471 -1.6426021 -1.6565257
  -1.6380554 -1.5651519 -1.5675589 -1.5920293 -1.569626  -1.5117399
  -1.5338209 -1.5180663 -1.527133  -1.5474983]]
------------------------------------------------
Result for output: unstack:1
Expected:
[1000012005]
Produced by ONNX Runtime:
[[-0.36428243 -0.65445775 -0.7659082  -0.7935119  -0.7481695  -0.73068815
  -0.64523095 -0.714809   -0.7164711  -0.7569726  -0.69840974 -0.6807602
  -0.69519323 -0.66507024 -0.6614157  -0.7172157  -0.6891945  -0.73177105
  -0.783917   -0.77680343 -0.75702685 -0.77518374 -0.81282    -0.81438345
  -0.8173483  -0.8142962  -0.92513543 -0.8664235  -0.8801006  -0.8546123
  -0.88459843 -0.8760842  -0.87997514 -0.8856838  -0.8748742  -0.8643026
  -0.89406127 -0.9046325  -0.8789348  -0.8796331  -0.8760676  -0.91483873
  -0.9337774  -0.9381452  -0.89810103 -0.9257491  -0.9301317  -0.94334227
  -0.91457886 -0.9016335  -0.89747757 -0.96615165 -0.91779935 -0.9619789
  -0.97980803 -0.85285455 -0.9222514  -0.9170558  -0.8792221  -0.9122589
  -0.88681346 -0.8920817  -0.91148764 -0.8444577  -0.9053207  -0.82081515
  -0.83988184 -0.844111   -0.84142584 -0.8350988  -0.8298654  -0.82823354
  -0.8506276  -0.8521582  -0.8183802  -0.8404047  -0.84346074 -0.76155615
  -0.80646294 -0.7237704  -0.73872024 -0.7826956  -0.7856826  -0.73501295
  -0.6743624  -0.6629616  -0.69897825 -0.71145093 -0.6775457  -0.66068465
  -0.68422896 -0.70213056 -0.7219971  -0.68632966 -0.6916229  -0.7325507
  -0.6758872  -0.68715125 -0.6739555  -0.6742949  -0.7636932  -0.81091374
  -0.8065447  -0.81203073 -0.8366683  -0.8142419  -0.8574132  -0.8626985
  -0.8983435  -0.93302786 -0.9367614  -0.97500706 -0.94844294 -0.978641
  -0.92729205 -0.9706332  -1.013283   -0.98836976 -1.0358481  -1.0683397
  -1.0934719  -1.0724169  -1.0934477  -1.1017811  -1.0787352  -1.0728683
  -1.064396   -1.0915014  -1.1345319  -1.1842524  -1.0513996  -1.0913066
  -1.1894689  -1.2638508  -1.2681626  -1.2085812  -1.1911026  -1.2693659
  -1.2642025  -1.2585833  -1.3077561  -1.3607422  -1.3963063  -1.3727025
  -1.3088276  -1.3356591  -1.3088322  -1.4363979  -1.5367892  -1.649587
  -1.729499   -1.7360331  -1.7725393  -1.8379817  -1.8995994  -1.8225228
  -1.8131849  -1.66373    -1.7389001  -1.7709086  -1.6824465  -1.6060101
  -1.4205089  -1.4690455  -1.4939958  -1.3814293  -1.2618252  -1.0531955
  -0.9599772  -0.90438634 -0.8996236  -0.82332665 -0.8073879  -0.85076517
  -0.79153174 -0.74035364 -0.8074298  -0.84986067 -0.7879202  -0.8287427
  -0.8491667  -0.7791769  -0.8426145  -0.8681217  -0.87314886 -0.894617
  -0.8891974  -0.88294464 -0.8533407  -0.8506084  -0.82696795 -0.8301706
  -0.8159041  -0.82503897 -0.81812054 -0.8234837  -0.8398509  -0.8713147
  -0.87848383 -0.83959705 -0.820674   -0.82350075 -0.8099441  -0.80115086
  -0.87185234 -0.8268748  -0.86762613 -0.8733602  -0.8247039  -0.8189853
  -0.90798235 -0.8466014  -0.81285375 -0.8357361  -0.8836693  -0.8403074
  -0.7970475  -0.7714967  -0.8262318  -0.82647735 -0.8079489  -0.81319267
  -0.82377744 -0.8310015  -0.8963962  -0.84627336 -0.8869917  -0.8347685
  -0.8409597  -0.8453347  -0.850218   -0.7829371  -0.8289574  -0.80998844
  -0.82171065 -0.70609695 -0.78698176 -0.80823076 -0.75761443 -0.7967404
  -0.8183175  -0.78146166 -0.84380966 -0.86980635 -0.8117575  -0.8530488
  -0.83755016 -0.7678513  -0.7547589  -0.78611785 -0.7281521  -0.71463484
  -0.7471989  -0.7406808  -0.8008632  -0.7555141 ]]
------------------------------------------------

Even if output order is not guaranteed (based on data shapes we can assume that expected outputs from .pb files are saved in order unique_ids:0, unstack:0, unstack:1) the mismatch is very huge. When I run it on other framework I got results the same as produced by ONNX Runtime which suggest problem with expected test data. The other problem can be missing "preprocessing" or "postprocessing" step which I am not aware of.

What is important if I run this model with expected data using ./onnx_test_runner app I see that everything succeeded.

snnn commented 3 years ago

I believe the model was from https://github.com/onnx/models/tree/master/text/machine_comprehension/bert-squad

If it is a test data issue, please help us report it to https://github.com/onnx/models/issues

Thank you!

mbencer commented 3 years ago

@snnn Thanks for answer Reporting issue to model zoo was my first step some time ago: https://github.com/onnx/models/issues/415 My PR with fix expected data to model zoo was also merged: https://github.com/onnx/models/pull/418

But still very strange for me is that ./onnx_test_runner PASS only for test data before my change. So there are some differences between onnx_test_runner execution and my test script.

Have you maybe plans to update https://onnxruntimetestdata.blob.core.windows.net/models/20191107.zip package?

snnn commented 3 years ago

I will help take a look.

mbencer commented 3 years ago

@snnn Great, thanks