search

Improbable-AI / dexenv

Code for Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapes (Science Robotics)
https://taochenshh.github.io/projects/visual-dexterity
MIT License
86 stars 11 forks source link

"free(): invalid pointer" when using "begin_aggregate()" #1

Open sysu19351115 opened 7 months ago

sysu19351115 commented 7 months ago

Hi, I'm trying to train a new policy. Unfortunately, whether I train teacher or student, aborted occurs when creating the environment.

Take "mlp.py" as an example, specifically, the error occurs on line 95 in "dclaw_multiobjs,py" : "self.gym.begin_aggregate(env_ptr, max_agg_bodies, max_agg_shapes, True)". The specific error is as follows:

Importing module 'gym_38' (/workspace/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_38.so)
Setting GYM_USD_PLUG_INFO_PATH to /workspace/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
mlp.py:17: UserWarning: 
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
  @hydra.main(config_path=dexenv.PROJECT_ROOT.joinpath('conf').as_posix(), config_name="debug_dclaw")
/usr/local/lib/python3.8/dist-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
  ret = run_job(
PyTorch version 2.1.0+cu118
Device count 2
/workspace/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /root/.cache/torch_extensions/py38_cu118 as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/py38_cu118/gymtorch/build.ninja...
Building extension module gymtorch...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module gymtorch...
/workspace/isaacgym/python/isaacgym/torch_utils.py:135: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  def get_axis_params(value, axis_idx, x_value=0., dtype=np.float, n_dims=3):
2024-05-05 04:18:06.870 | INFO     | dexenv.utils.create_task_env:create_task_env:19 - Creating environment DclawMultiObjs
2024-05-05 04:18:06.871 | WARNING  | MODDclawMultiObjs:parse_obj_dataset:168 - Dataset path:/workspace/dexenv/dexenv/assets/miscnet/train
2024-05-05 04:18:06.877 | INFO     | MODDclawMultiObjs:__init__:23 - Object urdf root path:/workspace/dexenv/dexenv/assets/miscnet/train.
2024-05-05 04:18:06.877 | INFO     | MODDclawMultiObjs:__init__:24 - Number of available objects:150.
2024-05-05 04:18:06.877 | WARNING  | dexenv.envs.dclaw_base:__init__:29 - Domain randomization is enabled!
Obs type: full
2024-05-05 04:18:06.879 | INFO     | dexenv.envs.base.vec_task:__init__:89 - RL device:cuda
2024-05-05 04:18:06.879 | INFO     | dexenv.envs.base.vec_task:__init__:90 - Simulation device:cuda:0
2024-05-05 04:18:06.879 | INFO     | dexenv.envs.base.vec_task:__init__:91 - Graphics device:-1
/usr/local/lib/python3.8/dist-packages/gym/spaces/box.py:84: UserWarning: WARN: Box bound precision lowered by casting to float32
  logger.warn(f"Box bound precision lowered by casting to {self.dtype}")
[Warning] [carb.gym.plugin] useGpuPipeline is set, forcing GPU PhysX
[Warning] [carb.gym.plugin] useGpu is set, forcing single scene (0 subscenes)
Not connected to PVD
+++ Using GPU PhysX
Physics Engine: PhysX
Physics Device: cuda:0
GPU Pipeline: enabled
2024-05-05 04:18:07.985 | INFO     | dexenv.envs.dclaw_base:get_dclaw_asset:455 - VHACD:True
/workspace/dexenv/dexenv/envs/dclaw_base.py:461: DeprecationWarning: an integer is required (got type isaacgym._bindings.linux-x86_64.gym_38.DofDriveMode).  Implicit conversion to integers using __int__ is deprecated, and may be removed in a future version of Python.
  asset_options.default_dof_drive_mode = gymapi.DOF_MODE_POS
Dclaw asset root:/workspace/dexenv/dexenv/assets/dclaw_4f robot name:dclaw_4f
D-Claw:
     Number of bodies: 21
     Number of shapes: 105
     Number of dofs: 12
2024-05-05 04:18:10.423 | INFO     | dexenv.envs.dclaw_base:get_dclaw_asset:481 - Joint names:dict_keys(['four1_jnt', 'four2_jnt', 'four3_jnt', 'one1_jnt', 'one2_jnt', 'one3_jnt', 'three1_jnt', 'three2_jnt', 'three3_jnt', 'two1_jnt', 'two2_jnt', 'two3_jnt'])
Number of fingertips:4  Fingertips:['four_tip_link', 'one_tip_link', 'three_tip_link', 'two_tip_link']
Actuator   ---  DoF Index
     four1_jnt   0
     four2_jnt   1
     four3_jnt   2
     one1_jnt   3
     one2_jnt   4
     one3_jnt   5
     three1_jnt   6
     three2_jnt   7
     three3_jnt   8
     two1_jnt   9
     two2_jnt   10
     two3_jnt   11
Setting DOF velocity limit to:[6.55172413793103, 7.758620689655173, 7.7586206896551]
Setting DOF effort limit to:2.6
Setting stiffness to:[2.7724128689655174, 3.558619503448275, 3.5586195034482757]
Setting damping to:[0.273946924137931, 0.382384248275862, 0.382384248275862]
2024-05-05 04:18:10.460 | INFO     | MODDclawMultiObjs:load_object_asset:207 - Loading object IDs from 0 to 150.
Loading Asset: 100%|███████████████████████████████████| 150/150 [01:18<00:00,  1.92it/s]
free(): invalid pointer
Aborted (core dumped)

This error occurs very randomly, when the number of environments is set differently, it will occur in the aggregation of different objects. Also, I tried it on different computers and the same thing happened all the time, so I don't think it was a problem with my computer.

When I try to cancel "begin_aggregate", I get no error when I create the environment, but then I get a new error: "RuntimeError: CUDA error: an illegal memory access was encountered" when training. Even though I reduced the number of environments to 1000, it also happens. A smaller number of environments would lead to a longer training time, and I don't want that. I wonder if the cancellation of aggregation caused this problem?

The computer I'm using has 250GB of RAM, an RTX A6000 GPU, and 48GB of video memory

In general, I wanted to solve the "free(): invalid pointer" problem caused by aggregation, but I didn't succeed. Does anyone have any idea about this situation? If more detailed information is needed, please let me know. Thanks for any suggestion.

sysu19351115 commented 7 months ago

When canceling the "begin_aggregate", the following error occurs: 

0it [00:00, ?it/s]2024-05-05 04:48:03.024 | INFO     | dexenv.utils.torch_utils:save_model:356 - Saving checkpoint, step: 16000.
125it [21:18, 18.96s/it]/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2155) : internal error : memcpy failed fail! 2  700 

/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2165) : internal error : GPU cudaMainGjkEpa or prepareLostFoundPairs kernel fail!
  700

/buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h (54) : internal error : SynchronizeStreams cuEventRecord failed

/buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h (60) : internal error : SynchronizeStreams cuStreamWaitEvent failed

/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2408) : internal error : memcpy failed fail!
  700

/buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h (54) : internal error : SynchronizeStreams cuEventRecord failed

/buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h (60) : internal error : SynchronizeStreams cuStreamWaitEvent failed

/buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h (54) : internal error : SynchronizeStreams cuEventRecord failed

/buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h (60) : internal error : SynchronizeStreams cuStreamWaitEvent failed

/buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h (80) : internal error : SynchronizeStreams failed

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1129) : internal error : GPU contactConstraintPrepare fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1200) : internal error : GPU artiContactConstraintPrepare fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h (80) : internal error : SynchronizeStreams failed

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1492) : internal error : GPU solveContactParallel fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1529) : internal error : GPU solveContactParallel fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1697) : internal error : GPU compute solver bodies average velocities fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (859) : internal error : GPU solveSelfConstraints fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (879) : internal error : GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (907) : internal error : GPU solveInternalConstraints fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1762) : internal error : GPU solveStaticBlock fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1772) : internal error : GPU solveStaticBlock fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1784) : internal error : GPU propagate solver bodies average velocities fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1697) : internal error : GPU compute solver bodies average velocities fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (859) : internal error : GPU solveSelfConstraints fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (879) : internal error : GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (907) : internal error : GPU solveInternalConstraints fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1762) : internal error : GPU solveStaticBlock fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1772) : internal error : GPU solveStaticBlock fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1784) : internal error : GPU propagate solver bodies average velocities fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1697) : internal error : GPU compute solver bodies average velocities fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (859) : internal error : GPU solveSelfConstraints fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (879) : internal error : GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (907) : internal error : GPU solveInternalConstraints fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1762) : internal error : GPU solveStaticBlock fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1772) : internal error : GPU solveStaticBlock fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1784) : internal error : GPU propagate solver bodies average velocities fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1697) : internal error : GPU compute solver bodies average velocities fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (859) : internal error : GPU solveSelfConstraints fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (879) : internal error : GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (907) : internal error : GPU solveInternalConstraints fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1762) : internal error : GPU solveStaticBlock fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1772) : internal error : GPU solveStaticBlock fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1784) : internal error : GPU propagate solver bodies average velocities fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1697) : internal error : GPU compute solver bodies average velocities fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (859) : internal error : GPU solveSelfConstraints fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (879) : internal error : GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (907) : internal error : GPU solveInternalConstraints fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1762) : internal error : GPU solveStaticBlock fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1772) : internal error : GPU solveStaticBlock fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1784) : internal error : GPU propagate solver bodies average velocities fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1697) : internal error : GPU compute solver bodies average velocities fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (859) : internal error : GPU solveSelfConstraints fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (879) : internal error : GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (907) : internal error : GPU solveInternalConstraints fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1762) : internal error : GPU solveStaticBlock fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1772) : internal error : GPU solveStaticBlock fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1784) : internal error : GPU propagate solver bodies average velocities fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1697) : internal error : GPU compute solver bodies average velocities fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (859) : internal error : GPU solveSelfConstraints fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (879) : internal error : GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (907) : internal error : GPU solveInternalConstraints fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1762) : internal error : GPU solveStaticBlock fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1772) : internal error : GPU solveStaticBlock fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1784) : internal error : GPU propagate solver bodies average velocities fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1064) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (1074) : internal error : GPU averageDeltaV fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1612) : internal error : GPU mArtiSolvePartitionFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1697) : internal error : GPU compute solver bodies average velocities fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (859) : internal error : GPU solveSelfConstraints fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (879) : internal error : GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp (907) : internal error : GPU solveInternalConstraints fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1762) : internal error : GPU solveStaticBlock fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1772) : internal error : GPU solveStaticBlock fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1784) : internal error : GPU propagate solver bodies average velocities fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1814) : internal error : GPU concludeBlockFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (1858) : internal error : GPU solveContactParallel fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (2116) : internal error : GPU mWritebackBlockFunction fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h (80) : internal error : SynchronizeStreams failed

/buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp (2406) : internal error : GPU integrateCoreParallel fail to launch kernel!!

/buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h (80) : internal error : SynchronizeStreams failed

/buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h (80) : internal error : SynchronizeStreams failed

/buildAgent/work/99bede84aa0a52c2/source/physx/src/NpScene.cpp (3509) : internal error : PhysX Internal CUDA error. Simulation can not continue!

[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 3480
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 3535
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 3454
/buildAgent/work/99bede84aa0a52c2/source/physx/src/NpScene.cpp (3189) : internal error : PhysX Internal CUDA error. Simulation can not continue!

/buildAgent/work/99bede84aa0a52c2/source/physx/src/NpScene.cpp (3496) : invalid operation : PxScene::fetchResults: fetchResults() called illegally! It must be called after advance() or simulate()

[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 3480
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 3454
/buildAgent/work/99bede84aa0a52c2/source/physx/src/NpScene.cpp (3189) : internal error : PhysX Internal CUDA error. Simulation can not continue!

/buildAgent/work/99bede84aa0a52c2/source/physx/src/NpScene.cpp (3496) : invalid operation : PxScene::fetchResults: fetchResults() called illegally! It must be called after advance() or simulate()

[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 3480
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 3535
125it [21:31, 10.33s/it]
Error executing job with overrides: ['alg.num_envs=2000', 'task.headless=True']
Traceback (most recent call last):
  File "mlp.py", line 33, in main
    engine.train()
  File "/workspace/dexenv/dexenv/engine/ppo_engine.py", line 24, in train
    traj, rollout_time = self.rollout_once(sample=True,
  File "/workspace/dexenv/dexenv/engine/base_engine.py", line 93, in rollout_once
    traj = self.runner(**kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/workspace/dexenv/dexenv/runner/nstep_runner.py", line 52, in __call__
    next_ob, reward, done, info = env.step(action)
  File "/workspace/dexenv/dexenv/envs/base/vec_task.py", line 336, in step
    self.timeout_buf = torch.where(self.progress_buf >= self.max_episode_length - 1, torch.ones_like(self.timeout_buf),
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
kiwiwan commented 5 months ago

I found a way to fix this bug.image @sysu19351115 ，you could try it. It came from hora