bic4907
commented
2 months ago Sorry for the late response, here's the repsonses for your question.
Issue 1: UnityEnvironment worker: send failed. Error
(mlagents) D:\RaidEnv-v1\RaidEnv\Build\Win>mlagents-learn pcg_winRate-0.5-1.0.yaml --env MMORPG.exe
┐ ╖
╓╖╬│╡ ││╬╖╖
╓╖╬│││││┘ ╬│││││╬╖
╖╬│││││╬╜ ╙╬│││││╖╖ ╗╗╗
╬╬╬╬╖││╦╖ ╖╬││╗╣╣╣╬ ╟╣╣╬ ╟╣╣╣ ╜╜╜ ╟╣╣
╬╬╬╬╬╬╬╬╖│╬╖╖╓╬╪│╓╣╣╣╣╣╣╣╬ ╟╣╣╬ ╟╣╣╣ ╒╣╣╖╗╣╣╣╗ ╣╣╣ ╣╣╣╣╣╣ ╟╣╣╖ ╣╣╣
╬╬╬╬┐ ╙╬╬╬╬│╓╣╣╣╝╜ ╫╣╣╣╬ ╟╣╣╬ ╟╣╣╣ ╟╣╣╣╙ ╙╣╣╣ ╣╣╣ ╙╟╣╣╜╙ ╫╣╣ ╟╣╣
╬╬╬╬┐ ╙╬╬╣╣ ╫╣╣╣╬ ╟╣╣╬ ╟╣╣╣ ╟╣╣╬ ╣╣╣ ╣╣╣ ╟╣╣ ╣╣╣┌╣╣╜
╬╬╬╜ ╬╬╣╣ ╙╝╣╣╬ ╙╣╣╣╗╖╓╗╣╣╣╜ ╟╣╣╬ ╣╣╣ ╣╣╣ ╟╣╣╦╓ ╣╣╣╣╣
╙ ╓╦╖ ╬╬╣╣ ╓╗╗╖ ╙╝╣╣╣╣╝╜ ╘╝╝╜ ╝╝╝ ╝╝╝ ╙╣╣╣ ╟╣╣╣
╩╬╬╬╬╬╬╦╦╬╬╣╣╗╣╣╣╣╣╣╣╝ ╫╣╣╣╣
╙╬╬╬╬╬╬╬╣╣╣╣╣╣╝╜
╙╬╬╬╣╣╣╜
╙
Version information:
ml-agents: 0.29.0,
ml-agents-envs: 0.29.0,
Communicator API: 1.5.0,
PyTorch: 1.11.0+cu113
[INFO] Connected to Unity environment with package version 2.2.1-exp.1 and communication version 1.5.0The 'Broken pipe' error occurs when the python client failed to connect with Unity game client. If the game successfully connected with the game client, you can find the below message from the console.
[INFO] Connected to Unity environment with package version 2.2.1-exp.1 and communication version 1.5.0 ddd
Please make sure you have mount the correct build path ($(pwd)/../../../RaidEnv/Build/Linux:/game) to the /game internal path. In my case, the game build was located in RaidEnv/Build/Linux/MMORPG.x86_64.
Or, please edit this part of $(pwd)/../../../RaidEnv/Build/Linux:/game in the run_experiment.sh file.
docker run --rm -t --gpus all --name $exp_name$exp_seed -v $(pwd):/config -v $(pwd)/../../../RaidEnv/Build/Linux:/game -v /mnt/nas/MMORPG/PCG/PCGRL:/workspace/results inchang/ct_game /bin/bash -c "chmod -R 755 /game && CUDA_VISIBLE_DEVICES=$allocated_gpu mlagents-learn /config/$_file --env /game/MMORPG.x86_64 --num-envs 16 --no-graphics --run-id $exp_name$exp_seed" > ./exp_logs/$exp_name$exp_seed.log 2>&1 &Please try my suggestion and leave your further feedback.
Issue 2: PCGRL Implementation
The PCGRL implementation is coded in the below CSharp file path (PCGAgent.cs).
RaidEnv\Assets\ML-Agents\Examples\MMORPG\Scripts\Agent\PCGAgent\PCGAgent.cs
In this version of RaidEnv, we used range, cool time, cast time, and damage parameter and the index is 7, 8, 9, and 15, respectively.
State
The CollectObservations function determines the observation of the PCG agent. This version of RaidEnv using only four values (cause of the homogeneous agent settings.)
Append the line of codes on the below to add skill parameters.
sensor.AddObservation(GetMinMaxScaledValue(skill[7], thresHold.range[0], thresHold.range[1]));
sensor.AddObservation(GetMinMaxScaledValue(skill[8], thresHold.cooltime[0], thresHold.cooltime[1]));
sensor.AddObservation(GetMinMaxScaledValue(skill[9], thresHold.casttime[0], thresHold.casttime[1]));
sensor.AddObservation(GetMinMaxScaledValue(skill[15], thresHold.value[0], thresHold.value[1]));Action
The action is decrease/increase of the skill parameters; the value is moved with the tickScale (1/6 of the each range).
For example, the action value 4 for 'cool time' induces 1/6 increasement of a tick of 'cool time' (i.e., (max-min) / 6)
The index of the _lastGeneratedSkill represents the each parameter (e.g., range, cool time, cast time, and damage) of the skills.
public void GenerateSkill(ActionSegment<int> _act)
{
<omitted>
// Convert action to float
float[] act = new float[_act.Length];
for(int i = 0; i < _act.Length; i++)
{
switch(_act[i])
{
case 0:
act[i] = -1.0f;
break;
case 1:
act[i] = -0.5f;
break;
case 2:
act[i] = 0;
break;
case 3:
act[i] = +0.5f;
break;
case 4:
act[i] = +1.0f;
break;
}
}
SkillGenerator.MinMaxThreshold thresHold = new SkillGenerator.MinMaxThreshold();
if(changeParameterDirectly == false)
{
float tickScale = 6;
// Get range value from MineMaxThreshold in SkillGenerator
float rangeTick = (thresHold.range[1] - thresHold.range[0]) / tickScale;
float coolTick = (thresHold.cooltime[1] - thresHold.cooltime[0]) / tickScale;
float castTimeTick = (thresHold.casttime[1] - thresHold.casttime[0]) / tickScale;
float valueTick = (thresHold.value[1] - thresHold.value[0]) /tickScale;
_lastGeneratedSkill[7] += rangeTick * act[0];
_lastGeneratedSkill[8] += coolTick * act[1];
_lastGeneratedSkill[9] += castTimeTick * act[2];
_lastGeneratedSkill[15] += valueTick * act[3];
_lastGeneratedSkill[7] = ActionLimiting(_lastGeneratedSkill[7], thresHold.range[0], thresHold.range[1]);
_lastGeneratedSkill[8] = ActionLimiting(_lastGeneratedSkill[8], thresHold.cooltime[0], thresHold.cooltime[1]);
_lastGeneratedSkill[9] = ActionLimiting(_lastGeneratedSkill[9], thresHold.casttime[0], thresHold.casttime[1]);
_lastGeneratedSkill[15] = ActionLimiting(_lastGeneratedSkill[15], thresHold.value[0], thresHold.value[1]);
}
I believe you can append the generated parameter of PCG agent by noting below information which present the index-value pair.
L#173 of RaidEnv\Assets\ML-Agents\Examples\MMORPG\Scripts\Generator\SkillGenerator.cs
The index of the parameter of '-1' value of the annotatation. (e.g., '8 - 1' => index of the 'range')
Add the codes into the GenerateSkill function with the desired index of the skill paramter. You may regulate the threshold value to approciatte problem complexity.
Append (increase) action space of 'PCGAgent' in the Unity inspector.
generatedSkill.Add((float)(int)triggerType); // 1
generatedSkill.Add((float)(int)magicSchool); // 2
generatedSkill.Add((float)(int)hitType); // 3
generatedSkill.Add((float)(int)targetType); // 4
generatedSkill.Add((float)projectileSpeed); // 5
generatedSkill.Add((canUseToAlly == true ? 1.0f : 0.0f)); // 6
generatedSkill.Add((canUseToEnemy == true ? 1.0f : 0.0f)); // 7
generatedSkill.Add(range); // 8
generatedSkill.Add(cooltime); // 9
generatedSkill.Add(casttime); // 10
generatedSkill.Add((float)cost); // 11
generatedSkill.Add((float)maximumCharge); // Do not touch this // 12
generatedSkill.Add((float)maximumCharge); // 13
generatedSkill.Add((canCastWhileCasting == true ? 1.0f : 0.0f)); // 14
generatedSkill.Add((canCastWhileChanneling == true ? 1.0f : 0.0f)); // 15
generatedSkill.Add(value); // 16
generatedSkill.Add(0f); // projectileFX.type // 17
generatedSkill.Add(0f); // projectileFX.size // 18
generatedSkill.Add(1f); // hitCount // 19Reward
Note the CalculateReward function. The function gets the previous and current winrates and returns a reward value, which decreasement on the L1 error.
protected float CalculateReward(float newValue, float oldValue, float targetValue)
{
float oldError = Mathf.Abs(targetValue - oldValue);
float newError = Mathf.Abs(targetValue - newValue);
// Return the difference between the new error and the old error
return oldError - newError;
}Best wishes!
jiabinfan
If I run your PCGRL/run_experiment.sh, I get errors like this.
Version information: ml-agents: 0.29.0, ml-agents-envs: 0.29.0, Communicator API: 1.5.0, PyTorch: 1.7.1 [INFO] Learning was interrupted. Please wait while the graph is generated. Traceback (most recent call last): File "/opt/conda/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 92, in send self.conn.send(req) File "/opt/conda/lib/python3.7/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/opt/conda/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes self._send(header + buf) File "/opt/conda/lib/python3.7/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/opt/conda/bin/mlagents-learn", line 8, in
sys.exit(main())
File "/opt/conda/lib/python3.7/site-packages/mlagents/trainers/learn.py", line 260, in main
run_cli(parse_command_line())
File "/opt/conda/lib/python3.7/site-packages/mlagents/trainers/learn.py", line 256, in run_cli
run_training(run_seed, options, num_areas)
File "/opt/conda/lib/python3.7/site-packages/mlagents/trainers/learn.py", line 132, in run_training
tc.start_learning(env_manager)
File "/opt/mlagents_envs/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, *kwargs)
File "/opt/conda/lib/python3.7/site-packages/mlagents/trainers/trainer_controller.py", line 198, in start_learning
raise ex
File "/opt/conda/lib/python3.7/site-packages/mlagents/trainers/trainer_controller.py", line 173, in start_learning
self._reset_env(env_manager)
File "/opt/mlagents_envs/mlagents_envs/timers.py", line 305, in wrapped
return func(args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/mlagents/trainers/trainer_controller.py", line 105, in _reset_env
env_manager.reset(config=new_config)
File "/opt/conda/lib/python3.7/site-packages/mlagents/trainers/env_manager.py", line 68, in reset
self.first_step_infos = self._reset_env(config)
File "/opt/conda/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 440, in _reset_env
self.set_env_parameters(config)
File "/opt/conda/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 457, in set_env_parameters
ew.send(EnvironmentCommand.ENVIRONMENT_PARAMETERS, config)
File "/opt/conda/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 94, in send
raise UnityCommunicationException("UnityEnvironment worker: send failed.")
mlagents_envs.exception.UnityCommunicationException: UnityEnvironment worker: send failed.
Besides, I am very confused about your PCGRL implementation. It seems all you have here is "run_experiments.sh", and you pass your training parameters to mlagent in run_experments.sh by a .yaml. Where are codes? For example, where do you define actions, state representations, rewards, and how to introduce more skill into state features? if I want to introduce more skills or use different RL settings for actions and rewards, which part of the code should I take a look?