search

InternLM / lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
https://lmdeploy.readthedocs.io/en/latest/
Apache License 2.0
3.15k stars 281 forks source link

fix SamplingDecodeTest and SamplingDecodeTest2 unittest failure #1874

Closed zhyncs closed 2 days ago

zhyncs commented 6 days ago

Motivation

fix SamplingDecodeTest and SamplingDecodeTest2 unittest failure

mkdir bld && cd bld && bash ../debug.sh && ninja -j48 && ./bin/unittest

Hi @lzhangzz @irexyc @lvhan028 May you help review this pr? Thanks.

Modification

as titled

Checklist

  1. Pre-commit or other linting tools are used to fix the potential lint issues.
  2. The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
  3. If the modification has a dependency on downstream projects of a newer version, this PR should be tested with all supported versions of downstream projects.
  4. The documentation has been modified accordingly, like docstring or example tutorials.
zhyncs commented 6 days ago

before the fix

[  FAILED  ] 28 tests, listed below:
[  FAILED  ] SamplingDecodeTest/0.TopK, where TypeParam = float
[  FAILED  ] SamplingDecodeTest/0.BatchTopK, where TypeParam = float
[  FAILED  ] SamplingDecodeTest/0.TopP, where TypeParam = float
[  FAILED  ] SamplingDecodeTest/0.BatchTopP, where TypeParam = float
[  FAILED  ] SamplingDecodeTest/0.TopKTopP, where TypeParam = float
[  FAILED  ] SamplingDecodeTest/0.BatchTopKTopP, where TypeParam = float
[  FAILED  ] SamplingDecodeTest/0.TopKBatchTopP, where TypeParam = float
[  FAILED  ] SamplingDecodeTest/0.BatchTopKBatchTopP, where TypeParam = float
[  FAILED  ] SamplingDecodeTest/0.InvalidArgsZeroTopK, where TypeParam = float
[  FAILED  ] SamplingDecodeTest/0.InvalidArgsZeroTopP, where TypeParam = float
[  FAILED  ] SamplingDecodeTest/0.InvalidArgsZeroTopKTopP, where TypeParam = float
[  FAILED  ] SamplingDecodeTest/0.InvalidArgsZeroBatchTopKTopP, where TypeParam = float
[  FAILED  ] SamplingDecodeTest/0.InvalidArgsZeroTopKBatchTopP, where TypeParam = float
[  FAILED  ] SamplingDecodeTest/0.InvalidArgsBatchTopKContainZero, where TypeParam = float
[  FAILED  ] SamplingDecodeTest/0.InvalidArgsBatchTopPContainZero, where TypeParam = float
[  FAILED  ] SamplingDecodeTest/0.InvalidArgsBatchTopKTopPContainZero, where TypeParam = float
[  FAILED  ] SamplingDecodeTest/0.InvalidArgsTopKBatchTopPContainZero, where TypeParam = float
[  FAILED  ] SamplingDecodeTest/0.InvalidArgsBatchTopKBatchTopPContainZero, where TypeParam = float
[  FAILED  ] SamplingDecodeTest/0.LocalBatchBatchTopP, where TypeParam = float
[  FAILED  ] SamplingDecodeTest/0.LocalBatchBatchTopKBatchTopP, where TypeParam = float
[  FAILED  ] SamplingDecodeTest2/0.CorrectnessSingleRandTopK, where TypeParam = float
[  FAILED  ] SamplingDecodeTest2/0.CorrectnessSingleRandTopP, where TypeParam = float
[  FAILED  ] SamplingDecodeTest2/0.CorrectnessBatchRandTopK, where TypeParam = float
[  FAILED  ] SamplingDecodeTest2/0.CorrectnessBatchRandTopP, where TypeParam = float
[  FAILED  ] SamplingDecodeTest2/0.CorrectnessBatchRandTopKLocalBatch, where TypeParam = float
[  FAILED  ] SamplingDecodeTest2/0.CorrectnessBatchRandTopPLocalBatch, where TypeParam = float
[  FAILED  ] SamplingDecodeTest2/0.CorrectnessCumLogProbTopK, where TypeParam = float
[  FAILED  ] SamplingDecodeTest2/0.CorrectnessCumLogProbTopP, where TypeParam = float

after

[==========] 111 tests from 14 test suites ran. (32080 ms total)
[  PASSED  ] 111 tests.
irexyc commented 5 days ago

https://github.com/InternLM/lmdeploy/blob/12e86091b1d376c14a2a8b023caa7942d3e79f47/tests/csrc/unittests/test_sampling_layer.cu#L1035-L1041

Different with fastertransformer, we initialize d_curand_state there. So in unittests, we should call invokeCurandBatchInitialize manually.

zhyncs commented 5 days ago

we should call invokeCurandBatchInitialize manually

ok

zhyncs commented 5 days ago

So in unittests, we should call invokeCurandBatchInitialize manually.

Hi @irexyc Could you please review the latest code? Thanks.