What happened + What you expected to happen

I am using CRR model with custom environment, I am getting this error

q_vals = q_model(model_out)[rows, actions].unsqueeze(-1) IndexError: tensors used as indices must be long, byte or bool tensors.

So I tried CRR with Cart-Pole environment, It's working fine !!.

Can you please help me on this ??

Thank you

Versions / Dependencies

Python 3.8.8 Ray rllib 2.1.0 pytorch 1.13.0

Reproduction script

model = crr.CRR(

our environment

env = "pagRobot_5D", config = { "framework": "torch", "env_config": { "curr_dir": "data_pxp\FMG", "isTrain": True }, "model": {

"custom_model": "my_model",

    "conv_filters": [[32, [8,8], 4], [64, [4,4], 2], [64, [4,4], 3], [512, [52,8], 1]],
    "conv_activation": "relu",
    "vf_share_layers": True,
   },

   #CRR algorithm specific config
   "weight_type" : "bin",
   "temperature" :1.0,
    "max_weight" : 20.0,
    "advantage_type" : "mean",
    "n_action_sample" :4,

    # target_network_update_freq by default is 100 * train_batch_size
    # if target_network_update_freq is not set. See self.setup for code.
    #Q fucntion update settings
    "target_network_update_freq" : 1,
    "tau": 0.0005,
    "twin_q" : True,

    #normal 
    "input": "expertdata",
    #"postprocess_inputs": True,
    "num_workers": 1,
    "train_batch_size": 32,

logger_creator=custom_log_creator( custom_path=os.path.expanduser("results_ray/"), custom_str='CRR' ) )

Issue Severity

Medium: It is a significant difficulty but I can work around it.

ray-project / ray

Critic Regularized Regression (CRR) model is getting error with Custom Environment (Offline RL) #30411