Is BERT actually good for semantic similarity.. check the example

[ghost]

As per score below , second case is more semantic similar than other one. But in actual it is just the opposite.

>>> model.predict([("he is an indian", "he has indian citizenship")])
array([3.2054904], dtype=float32)
>>> model.predict([("he is an indian", "he is  not an indian")])
array([3.590286], dtype=float32)

[AndriyMulyar]

The model implemented is identical to the model in the BERT paper reporting .89 annotator correlation.

[ghost]

Hi, The implementation may be correct but does it worth if results are wrong. I belive bert embedding is not made for this purpose. Why to put a code in public which will not give intended results as the name suggests or the Readme page promises. Please correct me if I am wrong on BERT usage for this purpose

On Wed, 24 Jul, 2019, 3:48 AM Andriy Mulyar, notifications@github.com wrote:

The model implemented is identical to the model in the BERT paper reporting .89 annotator correlation.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/AndriyMulyar/semantic-text-similarity/issues/2?email_source=notifications&email_token=AHRBKIEEVIARLVOKM35SC2DQA57UJA5CNFSM4IGHWIXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2UTCBQ#issuecomment-514404614, or mute the thread https://github.com/notifications/unsubscribe-auth/AHRBKIAFMN6633QKUKIOPRDQA57UJANCNFSM4IGHWIXA .

[AndriyMulyar]

Sandeep,

That is an interesting question. Like any model trained over a dataset, generalization may be difficult due to dataset specific biases that the model can pick up. If you are interested in these kind of issues, this recent ACL paper may be a good place to start.

Best, Andriy

On Tue, Jul 23, 2019, 11:53 PM Sandeep notifications@github.com wrote:

Hi, The implementation may be correct but does it worth if results are wrong. I belive bert embedding is not made for this purpose. Why to put a code in public which will not give intended results as the name suggests or the Readme page promises. Please correct me if I am wrong on BERT usage for this purpose

On Wed, 24 Jul, 2019, 3:48 AM Andriy Mulyar, notifications@github.com wrote:

The model implemented is identical to the model in the BERT paper reporting .89 annotator correlation.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/AndriyMulyar/semantic-text-similarity/issues/2?email_source=notifications&email_token=AHRBKIEEVIARLVOKM35SC2DQA57UJA5CNFSM4IGHWIXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2UTCBQ#issuecomment-514404614 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AHRBKIAFMN6633QKUKIOPRDQA57UJANCNFSM4IGHWIXA

.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/AndriyMulyar/semantic-text-similarity/issues/2?email_source=notifications&email_token=ADJ4TBVWE6JUKZ36YAOSPT3QA7G4VA5CNFSM4IGHWIXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2VCZLY#issuecomment-514469039, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJ4TBVRN7D6P3UWNUULZL3QA7G4VANCNFSM4IGHWIXA .

[sabirdvd]

Interesting paper. However, according to its main authors, it is not suited for Semantic Textual Similarity (STS) task, since it does not generate a meaningful vector to compute the cosine distance or semantic similarity.

[timpal0l]

@sabirdvd What would be a better approach than using the sentence representation from bert for Semantic Similarity...?