Problem with loading trained model

[hn2]

I am trying to load a trained model with CQL.load_model(..full model [path). I first got fname is missing I tried fname=..full_model_path I then got self is missing I added self It still doesn't load the model. no attribute 'impl' ...

[hn2]

from d3rlpy.algos import DiscreteCQL from d3rlpy.datasets import get_cartpole from d3rlpy.metrics.scorer import evaluate_on_environment

obtain dataset

dataset, env = get_cartpole()

setup algorithm

DiscreteCQL = DiscreteCQL(n_epochs=1)

'''

train

DiscreteCQL.fit(dataset.episodes, n_epochs=1)

DiscreteCQL.save_model('DiscreteCQLModel') ''' DiscreteCQL.load_model(fname='DiscreteCQLModel')

evaluate trained algorithm

evaluate_on_environment(env, render=True)(DiscreteCQL)

When trying to load this trained model ('DiscreteCQLModel') I get:

d3rlpy\base.py", line 244, in load_model self.impl.load_model(fname)

AttributeError: 'NoneType' object has no attribute 'load_model'

Problem is here:

def load_model(self, fname): """ Load neural network parameters.

    .. code-block:: python

        algo.load_model('model.pt')

    Args:
        fname (str): source file path.

    """
    self.impl.load_model(fname)

[takuseno]

@hn2 Hello, thank you for reporting the issue and sorry for the inconvenience. I confirmed that there is a bug when loading the model in your way. Please do work around as follows for a moment.

cql = DiscreteCQL(<give parameters>)

cql.create_impl(dataset.get_observation_shape(), dataset.get_action_size())

cql.load_model('<path-to-your-model>')

I'll prepare more sophisticated ways to do this in the next release. Thank you.

[takuseno]

Or please do this.

cql = DiscreteCQL.from_json('<path-to-params.json>')
cql.load('<path-to-model>')

Please replace <path-to-params.json> with the JSON file named 'params.json' that d3rlpy saves in d3rlpy_logs/<experiment directory>. This is the cleanest way so far.

[hn2]

I did not save the model as json.

Do I need to retrained it and save as json?

From: Takuma Seno [mailto:notifications@github.com] Sent: Sunday, 29 November 2020 16:30 To: takuseno/d3rlpy d3rlpy@noreply.github.com Cc: hn2 hannan2@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuseno/d3rlpy] Problem with loading trained model (#17)

Or please do this.

cql = DiscreteCQL.from_json('') cql.load('')

Please replace with the JSON file named 'params.json' that d3rlpy saves in d3rlpy_logs/. This is the cleanest way so far.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuseno/d3rlpy/issues/17#issuecomment-735403399 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZTKSR5N7IYQ6PN6FGNZA3SSJLGFANCNFSM4UGOZ2FA . https://github.com/notifications/beacon/ABZTKSRAEQDKN4GKB3VVKT3SSJLGFA5CNFSM4UGOZ2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFPKV3BY.gif

-- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

[takuseno]

The json file is automatically saved in the log directory. The purpose of this file is to save the configuration of the algorithm (not the model parameters such as weights and bias). You can use the json file to initialize the algorithm object as it was.

[hn2]

Is there a way to change the folder to which the json is saved to?

I tried 4 different models and I have 4 directories under log and I do not now to which run params it is refered to.

From: Takuma Seno [mailto:notifications@github.com] Sent: Sunday, 29 November 2020 17:37 To: takuseno/d3rlpy d3rlpy@noreply.github.com Cc: hn2 hannan2@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuseno/d3rlpy] Problem with loading trained model (#17)

The json file is automatically saved in the log directory. The purpose of this file is to save the configuration of the algorithm (not the model parameters such as weights and bias). You can use the json file to initialize the algorithm object as it was.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuseno/d3rlpy/issues/17#issuecomment-735412487 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZTKSUKHXZSJPCXR5J5XOLSSJTBBANCNFSM4UGOZ2FA . https://github.com/notifications/beacon/ABZTKSTLKL33LAFSME6CMW3SSJTBBA5CNFSM4UGOZ2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFPKYCBY.gif

-- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

[takuseno]

You can change that via logdir and experiment_name. Please see details at here.

[hn2]

I tried this:

cql = CQL()

cql.from_json('E:\QuantConnect\my-algos\drl-forex-offline\d3rlpy_logs\CQL_20201129130908\params.json')

cql.load(v_offline_model_file_name)

I get 'CQL' object has no attribute 'load'

From: Takuma Seno [mailto:notifications@github.com] Sent: Sunday, 29 November 2020 17:48 To: takuseno/d3rlpy d3rlpy@noreply.github.com Cc: hn2 hannan2@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuseno/d3rlpy] Problem with loading trained model (#17)

You can change that logdir and experiment_name. Please see details at here https://d3rlpy.readthedocs.io/en/v0.40/references/generated/d3rlpy.algos.DiscreteCQL.html#d3rlpy.algos.DiscreteCQL.fit .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuseno/d3rlpy/issues/17#issuecomment-735413961 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZTKSUMXK7HPF3U6AR23A3SSJUJLANCNFSM4UGOZ2FA . https://github.com/notifications/beacon/ABZTKSRZJZPEVDDZSAHL7ULSSJUJLA5CNFSM4UGOZ2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFPKYNSI.gif

-- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

[takuseno]

@hn2 I'm afraid to say this, but I'd like you to learn Python a bit more because I cannot support how to code Python. Please write as below.

cql = CQL.from_json('E:\QuantConnect\my-algos\drl-forex-offline\d3rlpy_logs\CQL_20201129130908\params.json')
cql.load(v_offline_model_file_name)

[takuseno]

And this will be useful to figure out how to save and load models.

[hn2]

I did that but you confirmed that there is a bug?

From: Takuma Seno [mailto:notifications@github.com] Sent: Sunday, 29 November 2020 17:57 To: takuseno/d3rlpy d3rlpy@noreply.github.com Cc: hn2 hannan2@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuseno/d3rlpy] Problem with loading trained model (#17)

And this will be useful to figure out how to save and load models here https://d3rlpy.readthedocs.io/en/v0.40/references/save_and_load.html .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuseno/d3rlpy/issues/17#issuecomment-735415392 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZTKSQ7EQFYOKNWNYVZSEDSSJVNRANCNFSM4UGOZ2FA . https://github.com/notifications/beacon/ABZTKSU4RT5444FJSG2KCBTSSJVNRA5CNFSM4UGOZ2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFPKYYYA.gif

-- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

[takuseno]

There is a bug, but you can load your model with work arounds.

option 1

Initialize PyTorch models manually.

cql = CQL()
cql.create_impl(dataset.get_observation_shape(), dataset.get_action_size())
cql.load_model(<path-to-model>)

option 2

initialize PyTorch's models via JSON.

cql = CQL.from_json(<path-to-params.json>)
cql.load_model(<path-to-model>)

[hn2]

Yes trying that.

When do you believe you will have a fix to the bug?

From: Takuma Seno [mailto:notifications@github.com] Sent: Sunday, 29 November 2020 18:02 To: takuseno/d3rlpy d3rlpy@noreply.github.com Cc: hn2 hannan2@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuseno/d3rlpy] Problem with loading trained model (#17)

There is a bug, but you can load your model with work arounds.

option 1

Initialize PyTorch models manually.

cql = CQL() cql.create_impl(dataset.get_observation_shape(), dataset.get_action_size()) cql.load_model()

option 2

initialize PyTorch's models via JSON.

cql = CQL.from_json() cql.load_model()

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuseno/d3rlpy/issues/17#issuecomment-735416065 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZTKSREWKLIM4AJJVYWG33SSJV6DANCNFSM4UGOZ2FA . https://github.com/notifications/beacon/ABZTKSQI4YSMVYCMVTYQPCTSSJV6DA5CNFSM4UGOZ2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFPKY6AI.gif

-- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

[takuseno]

I'm not working on this project as a part of my full-time project. But, I'll release the next version in 2 weeks.

[hn2]

Ok. Thanks.

From: Takuma Seno [mailto:notifications@github.com] Sent: Sunday, 29 November 2020 18:04 To: takuseno/d3rlpy d3rlpy@noreply.github.com Cc: hn2 hannan2@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuseno/d3rlpy] Problem with loading trained model (#17)

I'm not working on this project as a part of my full-time project. But, I'll release the next version in 2 weeks.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuseno/d3rlpy/issues/17#issuecomment-735416414 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZTKSQVAJSR53FXMUDJ6ODSSJWHXANCNFSM4UGOZ2FA . https://github.com/notifications/beacon/ABZTKSUAOFKWSH73X7SAKL3SSJWHXA5CNFSM4UGOZ2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFPKZAXQ.gif

-- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

[takuseno]

The next release includes new methods build_with_dataset and build_with_env to solve your problem, which is already available in master branch. In your case, you'll be able to do as follows:

# create algorithm object
cql = CQL()

# build neural networks with shapes of your dataset
cql.build_with_dataset(dataset)

# or build with your environment that must have observation_space and action_space as OpenAI does
# cql.build_with_env(env)

# load the model
cql.load_model(<path-to-your-model>)

I'll let you know when it's released.

[hn2]

Ok. Thanks.

I tried this:

train_episodes, test_episodes = train_test_split(dataset, test_size=0.2)

v_offline_model.fit(train_episodes,

                    eval_episodes=test_episodes,

                    n_epochs=n_epoches,

                    experiment_name='experiment1', 

                    logdir=v_offline_model_dir,

                    scorers={

                        'environment': evaluate_on_environment(v_env),

                        'advantage': discounted_sum_of_advantage_scorer

                    })

On my custom env and I get this error:

File "E:\QuantConnect\my-algos\drl-forex-offline\train - new.py", line 197, in train_offline_model

'advantage': discounted_sum_of_advantage_scorer

File "C:\Users\hanna\Anaconda3\lib\site-packages\d3rlpy\base.py", line 360, in fit

self._evaluate(eval_episodes, scorers, logger)

File "C:\Users\hanna\Anaconda3\lib\site-packages\d3rlpy\base.py", line 433, in _evaluate

test_score = scorer(self, episodes)

File "C:\Users\hanna\Anaconda3\lib\site-packages\d3rlpy\metrics\scorer.py", line 405, in scorer

action = algo.predict([observation])[0]

File "C:\Users\hanna\Anaconda3\lib\site-packages\d3rlpy\algos\base.py", line 80, in predict

return self.impl.predict_best_action(x)

File "C:\Users\hanna\Anaconda3\lib\site-packages\d3rlpy\algos\torch\utility.py", line 147, in wrapper

return f(self, *args, **kwargs)

File "C:\Users\hanna\Anaconda3\lib\site-packages\d3rlpy\algos\torch\utility.py", line 137, in wrapper

return f(self, *tensors, **kwargs)

File "C:\Users\hanna\Anaconda3\lib\site-packages\d3rlpy\algos\torch\base.py", line 23, in predict_best_action

return self._predict_best_action(x).cpu().detach().numpy()

File "C:\Users\hanna\Anaconda3\lib\site-packages\d3rlpy\algos\torch\ddpg_impl.py", line 145, in _predict_best_action

return self.policy.best_action(x)

File "C:\Users\hanna\Anaconda3\lib\site-packages\d3rlpy\models\torch\policies.py", line 164, in best_action

return self.forward(x, deterministic=True, with_log_prob=False)

File "C:\Users\hanna\Anaconda3\lib\site-packages\d3rlpy\models\torch\policies.py", line 133, in forward

action = self.mu(self.encoder(x))

File "C:\Users\hanna\AppData\Roaming\Python\Python36\site-packages\torch\nn\modules\module.py", line 722, in _call_impl

result = self.forward(*input, **kwargs)

File "C:\Users\hanna\Anaconda3\lib\site-packages\d3rlpy\models\torch\encoders.py", line 133, in forward

h = self.activation(self.fcs[i](h))

File "C:\Users\hanna\AppData\Roaming\Python\Python36\site-packages\torch\nn\modules\module.py", line 722, in _call_impl

result = self.forward(*input, **kwargs)

File "C:\Users\hanna\AppData\Roaming\Python\Python36\site-packages\torch\nn\modules\linear.py", line 91, in forward

return F.linear(input, self.weight, self.bias)

File "C:\Users\hanna\AppData\Roaming\Python\Python36\site-packages\torch\nn\functional.py", line 1674, in linear

ret = torch.addmm(bias, input, weight.t())

RuntimeError: size mismatch, m1: [1 x 1806], m2: [1092 x 256] at ..\aten\src\TH/generic/THTensorMath.cpp:41

What could be the reason?

From: Takuma Seno [mailto:notifications@github.com] Sent: Wednesday, 2 December 2020 16:48 To: takuseno/d3rlpy d3rlpy@noreply.github.com Cc: hn2 hannan2@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuseno/d3rlpy] Problem with loading trained model (#17)

The next release includes new methods build_with_dataset and build_with_env to solve your problem, which is already available in master branch. In your case, you'll be able to do as follows:

create algorithm object

cql = CQL()

build neural networks with shapes of your dataset

cql.build_with_dataset(dataset)

or build with your environment that must have observation_space and action_space as OpenAI does

cql.build_with_env(env)

load the model

cql.load_model()

I'll let you know when it's released.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuseno/d3rlpy/issues/17#issuecomment-737275655 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZTKSXLQLMGZAYQEOXDGDLSSZHQHANCNFSM4UGOZ2FA . https://github.com/notifications/beacon/ABZTKSRW2IMJ4CLUHVX5JBLSSZHQHA5CNFSM4UGOZ2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFPY66BY.gif

-- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

[takuseno]

It looks like the observation shape of your environment is not valid.

[hn2]

Is it better to train(fit) on env with scorer or just fit?

From: Takuma Seno [mailto:notifications@github.com] Sent: Thursday, 3 December 2020 16:00 To: takuseno/d3rlpy d3rlpy@noreply.github.com Cc: hn2 hannan2@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuseno/d3rlpy] Problem with loading trained model (#17)

It looks like the observation shape of your environment is not valid.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuseno/d3rlpy/issues/17#issuecomment-738011171 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZTKSTPMXOBFVZLP5VRH7DSS6KURANCNFSM4UGOZ2FA . https://github.com/notifications/beacon/ABZTKSVTEVGSIHP4FJHBWVDSS6KURA5CNFSM4UGOZ2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFP6SQIY.gif

-- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

[hn2]

Hi,

Did you push a fix to this problem to git?

From: Takuma Seno [mailto:notifications@github.com] Sent: Wednesday, 2 December 2020 16:48 To: takuseno/d3rlpy d3rlpy@noreply.github.com Cc: hn2 hannan2@gmail.com; Mention mention@noreply.github.com Subject: Re: [takuseno/d3rlpy] Problem with loading trained model (#17)

The next release includes new methods build_with_dataset and build_with_env to solve your problem, which is already available in master branch. In your case, you'll be able to do as follows:

create algorithm object

cql = CQL()

build neural networks with shapes of your dataset

cql.build_with_dataset(dataset)

or build with your environment that must have observation_space and action_space as OpenAI does

cql.build_with_env(env)

load the model

cql.load_model()

I'll let you know when it's released.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/takuseno/d3rlpy/issues/17#issuecomment-737275655 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZTKSXLQLMGZAYQEOXDGDLSSZHQHANCNFSM4UGOZ2FA . https://github.com/notifications/beacon/ABZTKSRW2IMJ4CLUHVX5JBLSSZHQHA5CNFSM4UGOZ2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFPY66BY.gif

-- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

[takuseno]

Closes this since the v0.41 supports build_with_dataset and build_with_env.