linkedin / detext

DeText: A Deep Neural Text Understanding Framework for Ranking and Classification Tasks
BSD 2-Clause "Simplified" License
1.26k stars 133 forks source link

Randomness in the demo #47

Closed jakiejj closed 3 years ago

jakiejj commented 4 years ago

Sometimes running the demo would result in precision@1 = 0.5 instead of the expected precision@1 = 1

e.g. https://github.com/linkedin/detext/runs/1001643393

  def test_demo():
        from subprocess import run, PIPE
        from pathlib import Path
        completed_process = run(['sh', 'run_detext.sh'], stderr=PIPE, cwd=f'{Path(__file__).parent}/resources')
        assert completed_process.returncode == 0
>       assert completed_process.stderr.endswith(b'metric/precision@1 = 1.0\n')
E       assert False
E        +  where False = <built-in method endswith of bytes object at 0x558411bbd750>(b'metric/precision@1 = 1.0\n')
E        +    where <built-in method endswith of bytes object at 0x558411bbd750> = b"WARNING:tensorflow:\nThe TensorFlow contrib module will not be included in TensorFlow 2.0.\nFor more information, pl...FO:tensorflow:metric/precision@1 = 0.5\nI0819 06:13:35.714258 140186665367360 logger.py:27] metric/precision@1 = 0.5\n".endswith
E        +      where b"WARNING:tensorflow:\nThe TensorFlow contrib module will not be included in TensorFlow 2.0.\nFor more information, pl...FO:tensorflow:metric/precision@1 = 0.5\nI0819 06:13:35.714258 140186665367360 logger.py:27] metric/precision@1 = 0.5\n" = CompletedProcess(args=['sh', 'run_detext.sh'], returncode=0, stderr=b"WARNING:tensorflow:\nThe TensorFlow contrib modu...O:tensorflow:metric/precision@1 = 0.5\nI0819 06:13:35.714258 140186665367360 logger.py:27] metric/precision@1 = 0.5\n").stderr
StarWang commented 4 years ago

Due to randomness in training, perhaps it's better not to do precision check as a hard check

jakiejj commented 3 years ago

fixed by #49