search

RLGC-Project / RLGC

An open-source platform for applying Reinforcement Learning for Grid Control (RLGC)
Other
108 stars 30 forks source link

Method __getstate__([]) does not exist & Training can't stop #2

Closed wuyuenhs closed 4 years ago

wuyuenhs commented 4 years ago

Hi, I encountered some problems that could not be solved during the recurrence process. I hope you can give me some help.

In the process of running the code, I got an error:

py4j.protocol.Py4JError: An error occurred while calling t.__getstate__. Trace:
py4j.Py4JException: Method __getstate__([]) does not exist
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
    at py4j.Gateway.invoke(Gateway.java:274)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:214)
    at java.lang.Thread.run(Thread.java:744)

And the Traceback:

Traceback (most recent call last):
  File "trainIEEE39LoadSheddingAgent.py", line 145, in <module>
    train(ll, env, model_path)
  File "trainIEEE39LoadSheddingAgent.py", line 105, in train
    act.save(savedModel + "/" + model_name + "_lr_%s_100w.pkl" % (str(learning_rate)))
  File "/root/anaconda3/envs/py3ml/lib/python3.6/site-packages/baselines/deepq/simple.py", line 55, in save
    dill.dump((model_data, self._act_params), f)
  File "/root/anaconda3/envs/py3ml/lib/python3.6/site-packages/dill/dill.py", line 274, in dump
    pik.dump(obj)
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 409, in dump
    self.save(obj)
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 736, in save_tuple
    save(element)
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/anaconda3/envs/py3ml/lib/python3.6/site-packages/dill/dill.py", line 871, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/anaconda3/envs/py3ml/lib/python3.6/site-packages/dill/dill.py", line 1355, in save_function
    obj.__dict__), obj=obj)
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 610, in save_reduce
    save(args)
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 736, in save_tuple
    save(element)
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/anaconda3/envs/py3ml/lib/python3.6/site-packages/dill/dill.py", line 1098, in save_cell
    pickler.save_reduce(_create_cell, (f,), obj=obj)
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 610, in save_reduce
    save(args)
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 736, in save_tuple
    save(element)
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 634, in save_reduce
    save(state)
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/anaconda3/envs/py3ml/lib/python3.6/site-packages/dill/dill.py", line 871, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/root/anaconda3/envs/py3ml/lib/python3.6/pickle.py", line 496, in save
    rv = reduce(self.proto)
  File "/root/anaconda3/envs/py3ml/lib/python3.6/site-packages/py4j/java_gateway.py", line 1160, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/root/anaconda3/envs/py3ml/lib/python3.6/site-packages/py4j/protocol.py", line 324, in get_return_value
    format(target_id, ".", name, value))

Episodes have been increasing until they are shut down by the system. % time spent exploring always show 2 in the above situation.

qhuang-pnl commented 4 years ago

@wuyuenhs The codes have been updated to make it compatible with the latest OpenAI Gym. Please try to install it in a new virtual environment (Don't use the yml file to create it as it is outdated). Please run the testPowerDynSimEnv_v3.py to make sure the environment is properly set up. Please report here if you have further issue.

wuyuenhs commented 4 years ago

Thanks for helping. I follow the above to create a new env and install packages as needed. Then run test_PowerDynSimEnv_v3.py for testing. As shown below, it indicates that the environment is properly set up.

test completed
Connection with Ipss Server is closed

However, running trainIEEE39LoadSheddingAgent.py has the following error:

Traceback (most recent call last):
  File "trainIEEE39LoadSheddingAgent.py", line 144, in <module>
    train(ll, env, model_path)
  File "trainIEEE39LoadSheddingAgent.py", line 87, in train
    model = deepq.models.mlp([256,256])
AttributeError: module 'baselines.deepq.models' has no attribute 'mlp'

Baselines v0.1.6 removed def mlp. So I switched to baselines v0.1.4. Then error:

Traceback (most recent call last):
  File "trainIEEE39LoadSheddingAgent.py", line 144, in <module>
    train(ll, env, model_path)
  File "trainIEEE39LoadSheddingAgent.py", line 101, in train
    trained_model = model_path
TypeError: learn() got an unexpected keyword argument 'trained_model'

Changing the code of trainIEEE39LoadSheddingAgent.py can cause a series of uncontrollable errors. I don't know if these errors are caused by updates to the environment package(such as baselines). Of course, the latest version of OpenAI Gym is no problem. If I return the other environment to the old version, there will be problems when I first asking questions in this issue. So I don't know how to do it now.

qhuang-pnl commented 4 years ago

It turned out Baselines made some changes to the implementation. We fixed the issues with this update.

Our internal test environments are: OpenAI gym (0.15.3) and baselines (0.1.5)

Please pull the latest RLGC and try it again. Thanks again for your feedback, we really appreciate it.

wuyuenhs commented 4 years ago

Thanks to the update, the program is able to get the final Finished. Unfortunately, it seems that there are some problems in the training process. In addition to FutureWarning, there are the following displays:

/root/anaconda3/envs/new/lib/python3.6/site-packages/numpy/core/fromnumeric.py:3257: RuntimeWarning: Mean of empty slice.
  out=out, **kwargs)
/root/anaconda3/envs/new/lib/python3.6/site-packages/numpy/core/_methods.py:161: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
Saving final model to power_model_multistep_581_585_lr_5e-05.pkl
total running time is 5.142378330230713
Finished!!

I think the total running time is too short, I don't know if this is normal. At the same time, the terminal that launch the Java server displays:

Observed states:
[voltageMag_Bus4, voltageMag_Bus7, voltageMag_Bus8, voltageMag_Bus18, voltageMag_Bus504, voltageMag_Bus507, voltageMag_Bus508, voltageMag_Bus518]
Initial values of the observed states:
[0.98712668745297, 0.9748013683313521, 0.9731456890757114, 1.0089329691108317, 0.9813836367392125, 0.9712347933931141, 0.9652967656365352, 1.005561904995773]
Bus, initLoad, shedLoadFraction = Bus504, 5.0, -0.2
Bus, initLoad, shedLoadFraction = Bus507, 2.338, -0.2
Bus, initLoad, shedLoadFraction = Bus518, 1.6320000000000001, -0.2
Bus, initLoad, shedLoadFraction = Bus507, 2.338, -0.2
Bus, initLoad, shedLoadFraction = Bus518, 1.6320000000000001, -0.2
Bus, initLoad, shedLoadFraction = Bus507, 2.338, -0.2
Bus, initLoad, shedLoadFraction = Bus518, 1.6320000000000001, -0.2
Bus, initLoad, shedLoadFraction = Bus507, 2.338, -0.2
Bus, initLoad, shedLoadFraction = Bus518, 1.6320000000000001, -0.2
Bus, initLoad, shedLoadFraction = Bus507, 2.338, -0.2
Bus, initLoad, shedLoadFraction = Bus518, 1.6320000000000001, -0.2
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : -5.551115123125783E-17
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : -5.551115123125783E-17
Bus, initLoad, shedLoadFraction = Bus507, 2.338, -5.551115123125783E-17
Bus, initLoad, shedLoadFraction = Bus518, 1.6320000000000001, -5.551115123125783E-17
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus507 is large than remaining fraction, it has been changed to : 0.0
Oct 31, 2019 7:37:47 PM org.pnnl.gov.rl.action.LoadChangeActionProcessor applyAction
SEVERE: change fraction for bus =Bus518 is large than remaining fraction, it has been changed to : 0.0
qhuang-pnl commented 4 years ago

Thanks for the confirmation. You can increase the total_timesteps in the deepq.learn() inside the train() function to increase the training, for example 500,000 or 1 Million steps.

The msgs in the java sever side are expected as the agent is mostly working under the exploration mode at the initial stage. The severe msgs mean the agent continues to try to shed the loads even there is no remaining load to shed.

wuyuenhs commented 4 years ago

Thank you for your tireless guidance. Now I can run the program successfully. The RLGC-Project has taught me a lot. Thanks again！