continue fitting - Githubissues

zfan2016 commented 1 year ago

I tried to continue fitting as below:

pacemaker input.yaml -p ../interim_potential_best_cycle.yaml

But I found that it always started to train from scratch.

From log.txt, I found that 2023/05/31 17:07:04 I - pacemaker/pyace version: 0.2.7+95.g37808e2 2023/05/31 17:07:04 I - ace_evaluator version: 2022.6.27 2023/05/31 17:07:04 I - Loading input.yaml... 2023/05/31 17:07:04 I - Potential settings is overwritten from arguments: ../interim_potential_best_cycle.yaml 2023/05/31 17:07:04 I - Set numpy random seed to 42 2023/05/31 17:07:04 I - Target potential loaded from file '../interim_potential_best_cycle.yaml', it contains 618 functions 2023/05/31 17:07:04 I - Initial potential is NOT provided, starting from empty potential 2023/05/31 17:07:04 I - Ladder-scheme fitting is ON 2023/05/31 17:07:04 I - Ladder_type: power_order is selected

Could you give some suggestions to resolve this issue?

yury-lysogorskiy commented 1 year ago

if you have ladder_step option in input.yaml, then it will assume the ladder-scheme. Try to comment out "ladder_step" and "ladder_type" in input.yaml

zfan2016 commented 1 year ago

Yes, I indeed have ladder_step option. I tried to comment out "ladder_step" and "ladder_type" but another error appeared. However, when changing to using -ip instead of -p, it seems to work.

yury-lysogorskiy commented 1 year ago

which error appeared ?

zfan2016 commented 1 year ago

2023/06/01 09:57:58 I - Fitting done 2023/06/01 09:57:58 D - Update metadata: MapStringString{_loss: 0.0001399597625878143, ace_evaluator_version: 2022.6.27, intermediate_time: 2023-05-31 04:53:01.102509, pacemaker_version: 0.2.7+95.g37808e2, starttime: 2023-06-01 09:57:56.073934, tensorpot_version: 0+untagged.17.g8b6fcd3, user: zfan} 2023/06/01 09:57:58 I - Final potential is saved to output_potential.yaml 2023/06/01 09:57:58 I - Making predictions 2023/06/01 09:57:58 I - For train data Traceback (most recent call last): File "/global/homes/z/zfan/.conda/envs/ace/bin/pacemaker", line 438, in main(sys.argv[1:]) File "/global/homes/z/zfan/.conda/envs/ace/bin/pacemaker", line 299, in main pred_data = predict_and_save(general_fit, target_bbasisconfig, general_fit.fitting_data, File "/global/homes/z/zfan/.conda/envs/ace/bin/pacemaker", line 411, in predict_and_save pred_data = general_fit.predict(structures_dataframe=structures_dataframe, File "/global/homes/z/zfan/.conda/envs/ace/lib/python3.8/site-packages/pyace/generalfit.py", line 605, in predict return self.fit_backend.predict(structures_dataframe=structures_dataframe, bbasisconfig=bbasisconfig) File "/global/homes/z/zfan/.conda/envs/ace/lib/python3.8/site-packages/pyace/fitadapter.py", line 312, in predict self.setup_backend_for_predict(bbasisconfig) File "/global/homes/z/zfan/.conda/envs/ace/lib/python3.8/site-packages/pyace/fitadapter.py", line 295, in setup_backend_for_predict raise ValueError("bbasisconfig couldn't be None for FitAdapter.setup_backend_for_predict") ValueError: bbasisconfig couldn't be None for FitAdapter.setup_backend_for_predict

celinehalkali commented 11 months ago

i have tried to replicate your error messages, and by implementing the following changes, they did not occur anymore

first error message the correct command for ladder upfitting from an existing potential would be pacemaker input.yaml -ip ipbc.yaml in "normal" upfitting, -p sets the input potential in ladder upfitting, -ip sets the input potential while -p sets the shape of the target potential

(assumed follow-up error message) if you use interim_potential_best_cycle.yaml, you will probably get an error as this file has an extra line fit_cycles. remove this line, or use another potential file (e.g. interim_potential_x.yaml, output_potential.yaml)

second error message number_of_basis_functions_per_element of your input potential and your target potential shape are probably the same. they should be different, as in ladder fitting you are continuously expanding the number of basis functions. to change the target potential shape, you can

manually set the number_of_basis_functions_per_element to a number larger than the one you used in your input potential
use the flag -p and provide any target_potential_shape.yaml which was trained with a different (larger) number_of_basis_functions_per_element. only the shape of the potential will be used, so you could also use target_potential.yaml from another training run

zfan2016 commented 11 months ago

Thanks for the detailed answers. Yes, last month I found "-ip" works for ladder upfitting.

ICAMS / python-ace

continue fitting #39