Could not override 'config.task_specific_params.pooling_method' when lauching wandb agent

[Rhett-Ying]

Hi I am trying to train models on MIMIC-IV via following tutorial. But failed. please help on it. thanks.

Here're the commands I ran:

1. clone ESGPT and MIMIC related repo.
2. built dataset and task dataframes.
3. under MIMICIV_FMs_public ./scripts/pretrain.sh +cohort_name=try240521_0927
4. ./scripts/launch_hyperparameter_tuning.sh cohort_name=try240521_0927. ERROR https://github.com/mmcdermott/EventStreamGPT/blob/main/scripts/launch_wandb_hp_sweep.py does not exist any more. So I replace it with launch_pretraining_wandb_hp_sweep.py.
5. ./scripts/launch_wandb_agent.sh --project MIMIC_FMs_public --entity xxxx cdby50ub

Step 5 failed with below error:

Could not override 'config.task_specific_params.pooling_method'.
To append to your config use +config.task_specific_params.pooling_method=max
Key 'pooling_method' is not in struct
    full_key: config.task_specific_params.pooling_method
    object_type=dict

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
2024-05-24 07:44:17,407 - wandb.wandb_agent - INFO - Running runs: ['cdby50ub']
2024-05-24 07:44:17,408 - wandb.wandb_agent - ERROR - Detected 3 failed runs in the first 60 seconds, shutting down.

[mmcdermott]

Thanks @Rhett-Ying -- apologies for the delay in response. I'll look into this soon -- don't hesitate to re-ping here as needed.

[Rhett-Ying]

@mmcdermott do you have any findings for this issue? And one more question, how much GPU RAM is required to train or fine-tune the GPT model on my own? what kind of GPU and how much time it takes? A100?