Is BO really working?

[qzhu2017]

@yanxon I just finished the scripting. Now it should be very easy to run the code. However, I am suspecting that the code does not really work as it is. Below are two example outputs.

Running random structure search 2... 
       Eng_true    Eng_estimation
   0  0.00000 <- -0.00000 Ibam         
   1 -1.46946 <- -0.00000 P6/mmm       
   2 -1.55275 <- -0.00000 P-3m1        
   3  0.00000 <- -0.00000 Im-3m        
   4 -4.24083 <- -0.00000 P1           
   5 -1.69075 <- -0.00000 P6/mmm       
   6 -4.16196 <- -0.00000 P6_3/mmc     
   7 -1.46946 <- -0.00000 P4/mmm       
   8 -3.90044 <- -0.00000 Amm2         
   9 -4.64995 <- -0.00000 Fd-3m        
Ground State is found at trial 10

20.39 s in 10 BO trials

Running random structure search 3... 

   0 -4.07406 <- -0.00001 P-1          
   1  0.00000 <- -0.00000 I-43m        
   2 -2.82267 <- -0.00000 Pmm2         
   3 -3.20896 <- -0.00000 Pmmm         
   4 -1.46946 <- -0.00000 P3m1         
   5 -1.94836 <- -1.94833 Pm-3m        
   6 -3.59280 <- -0.00030 P1           
   7 -4.27766 <- -0.00004 Cmcm         
   8 -3.58843 <- -0.00000 P4/mmm       
   9 -3.96814 <- -0.00000 P4_332       
  10 -3.48526 <- -0.00000 Cmmm         
  11 -1.69075 <- -0.00000 P6/mmm       
  12 -3.41348 <- -0.11915 P1           
  13 -1.46946 <- -0.00000 P6_3/mmc     
  14 -1.46946 <- -0.00000 Cmmm         
  15 -1.46946 <- -0.00000 I4/mmm       
  16 -3.36349 <- -0.00002 C2           
  17  0.00000 <- -0.00000 Im-3m        
  18 -2.47355 <- -0.00000 Pmm2         
  19 -4.17469 <- -0.00019 C222_1       
  20 -2.93893 <- -0.00000 Pmm2         
  21 -1.94836 <- -1.94836 Pm-3m        
  22 -4.01783 <- -0.00002 Pmc2_1       
  23 -1.76847 <- -0.00000 I4/mmm       
  24 -2.40468 <- -0.00000 P4mm         
  25 -1.94837 <- -1.94837 Pm-3m        
  26 -1.94836 <- -1.94836 Pm-3m        
  27 -1.46946 <- -0.00000 P3m1         
  28 -3.97440 <- -0.00000 I4_1/amd     
  29 -1.46946 <- -0.00000 Immm         
  30 -1.46946 <- -0.00000 P3m1         
  31 -4.07749 <- -0.00018 P2/c         
  32 -4.18982 <- -0.00006 C222         
  33  0.00000 <- -0.00000 P4/nmm       
  34 -1.46946 <- -0.00000 I4/mmm       
  35 -3.48526 <- -3.48526 Cmmm         
  36 -3.95877 <- -0.00000 Cmmm         
  37 -3.96735 <- -0.00000 P4/nmm       
  38 -2.99680 <- -0.00072 P2           
  39 -4.00846 <- -0.00000 Cmcm         
  40 -3.96814 <- -3.96814 P4_332       
  41  0.00000 <- -0.00000 Fm-3m        
  42 -1.46946 <- -0.00000 Cmmm         
  43 -3.96735 <- -0.00005 P2_1/m       
  44 -1.76847 <- -0.00000 P4/mmm       
  45 -4.00964 <- -0.00000 I4_132       
  46 -1.94837 <- -1.94836 Pm-3m        
  47 -3.27134 <- -0.00000 P-3m1        
  48 -3.97440 <- -1.68882 I4_1/amd     
  49 -3.95877 <- -0.00000 Cmmm         
  50 -4.01763 <- -0.00373 P2           
  51 -3.97100 <- -0.00295 P2           
  52 -3.05510 <- -0.00000 Im-3m        
  53 -3.95155 <- -0.00000 P4/nmm       
  54 -1.46946 <- -0.00000 P4/mmm       
  55 -0.58451 <- -0.00000 I4/mmm       
  56 -4.00626 <- -0.00000 Fmmm         
  57 -4.25244 <- -0.00000 P4_122       
  58 -1.46946 <- -0.00000 P4/nmm       
  59 -4.19503 <- -0.00000 P4/nmm       
  60 -2.69483 <- -0.00001 Amm2         
  61 -4.38102 <- -0.00020 C2/m         
  62 -4.07661 <- -0.00000 P4_2/mcm     
  63 -1.46946 <- -0.00000 P4/nmm       
  64 -4.16196 <- -0.00000 P6_3/mmc     
  65 -1.55275 <- -0.00000 P-3m1        
  66 -1.66128 <- -0.00000 P6/mmm       
  67 -4.07626 <- -1.01316 C2           
  68 -1.94837 <- -1.94836 Pm-3m        
  69 -4.02465 <- -0.00000 P2_1/m       
  70 -1.46946 <- -0.00000 P4/mmm       
  71 -1.46946 <- -0.00000 Pmmm         
  72 -2.80651 <- -0.00000 P-62m        
  73 -1.46946 <- -0.00000 Pmmm         
  74 -1.89138 <- -0.00000 P4/mmm       
  75 -1.46946 <- -0.00000 P4/nmm       
  76 -4.43317 <- -0.00000 I4/mmm       
  77 -4.03463 <- -0.00000 Pbcm         
  78  0.00000 <- -0.00000 Fm-3m        
  79  0.00000 <- -0.00000 Fm-3m        
  80 -3.96431 <- -0.00000 P2_1/m       
  81 -4.00964 <- -0.00000 P6/mmm       
  82 -3.39013 <- -0.00000 P-6m2        
  83 -4.00964 <- -0.00000 P6/mmm       
  84 -1.46946 <- -0.00000 P2/m         
  85 -1.94837 <- -1.94837 Pm-3m        
  86 -4.07661 <- -0.00000 P4_2/mmc     
  87 -3.05504 <- -0.00000 I4/mmm       
  88 -4.64995 <- -0.00000 P6_3/mmc     
  89 -3.97336 <- -0.00000 Pmma         
  90 -1.55275 <- -0.00000 P4/mmm       
  91 -0.58451 <- -0.00000 I4/mmm       
  92 -2.93893 <- -0.00001 C2/c         
  93 -1.51688 <- -0.00000 P4/mmm       
  94 -1.94837 <- -1.94837 Pm-3m        
  95  5.65945 <- -0.00000 P6/mmm       
  96 -3.96814 <- -3.96813 P4_132       
  97 -3.96814 <- -3.96814 P4_132       
  98 -3.45934 <- -0.00000 P6/mmm       
  99 -3.47361 <- -0.00000 Pm           
 100 -4.64995 <- -0.00030 Fd-3m        
Ground State is found at trial 101

[qzhu2017]

@yanxon I just figured out this is because of the kernel function used. If I changed it to other kernel function, the results become reasonable

The reference structure is -4.64995 eV/atom in Fd-3m

Running random structure search 0... 
                   Estimated ->  True -> Diff
   0 P4/mmm       -0.58451 -> -0.45147 -0.13304 
   1 Fm-3m         0.00000 -> -1.65491  1.65491 
   2 Cmme         -3.74855 -> -2.65103 -1.09752 
   3 P4/mmm       -1.46946 -> -1.91076  0.44129 
   4 Fm-3m         0.00000 -> -1.68554  1.68554 
   5 P2_1/m       -4.00964 -> -4.07454  0.06490 
   6 P1           -4.04385 -> -4.07908  0.03523 
   7 P4_2/m       -1.46946 -> -2.40085  0.93139 
   8 P2/m         -4.07899 -> -3.47308 -0.60591 
   9 I4_1/amd     -1.46946 -> -2.40547  0.93601 
  10 Fd-3m        -3.74201 -> -2.35560 -1.38641 
  11 P6/mmm       -1.46946 -> -1.99406  0.52460 
  12 Im-3m         0.00000 -> -1.64641  1.64641 
  13 I4_1/amd     -4.00964 -> -3.50248 -0.50717 
  14 P4/mmm       -0.58451 -> -1.31837  0.73385 
  15 P222_1       -4.01761 -> -4.12267  0.10506 
  16 Im-3m         0.00000 -> -1.68637  1.68637 
  17 R-3m         -1.69075 -> -3.21798  1.52723 
  18 P4/mmm       -1.76847 -> -1.59711 -0.17136 
  19 P-1          -4.38844 -> -4.36493 -0.02351 
  20 R3           -4.16660 -> -3.99242 -0.17418 
  21 Fm-3m         0.00000 -> -0.42698  0.42698 
  22 P6/mmm       -1.46946 -> -1.85305  0.38359 
  23 Im-3m         0.00000 -> -0.52074  0.52074 
  24 Im-3m         0.00000 -> -0.61585  0.61585 
  25 P4_132       -4.00958 -> -4.04161  0.03203 
  26 Pm-3m         0.00000 -> -1.16438  1.16438 
  27 Fm-3m        -2.80727 -> -1.25289 -1.55438 
  28 P6_3/mmc     -1.46946 -> -1.35962 -0.10984 
  29 P-1          -4.07661 -> -3.17989 -0.89672 
  30 I4/mmm       -2.76484 -> -2.77870  0.01385 
  31 P6_3/mmc     -4.64995 -> -4.15869 -0.49127 
  32 P4_322       -1.76847 -> -2.41859  0.65012 
  33 P4/mmm       -0.58451 -> -1.61249  1.02798 
  34 Fm-3m         0.00000 -> -0.53504  0.53504 
  35 Cmmm         -1.46946 -> -1.59268  0.12322 
  36 Pm-3m         0.00000 -> -1.16614  1.16614 
  37 Pm-3m         0.00000 -> -0.06983  0.06983 
  38 Fm-3m         0.00000 -> -0.73889  0.73889 
  39 P6_3/mmc     -4.16196 -> -2.24513 -1.91682 
  40 P6/mmm       -1.46946 -> -1.86814  0.39867 
  41 P2_1/m       -2.93891 -> -2.28249 -0.65642 
  42 P6_3/mmc     -4.64995 -> -4.64995 -0.00001 
  43 Im-3m         0.00000 -> -0.62168  0.62168 
  44 I4/mmm       -1.46946 -> -1.11960 -0.34986 
  45 P4_2/mmc     -1.46946 -> -1.86875  0.39929 
  46 P-62m        -1.46946 -> -2.18620  0.71673 
  47 Fd-3m        -3.74201 -> -3.74202  0.00001 
  48 P4_12_12     -1.46946 -> -2.13308  0.66362 
  49 Fm-3m         0.00000 -> -1.19611  1.19611 
  50 Im-3m        -3.05510 -> -1.26878 -1.78632 
  51 P6_3/mmc     -1.46946 -> -1.92688  0.45742 
  52 P2_1/m       -3.60434 -> -4.15360  0.54926 
  53 P6_3/mmc     -1.46946 -> -1.85065  0.38119 
  54 Amm2         -1.55275 -> -2.17116  0.61841 
  55 Amm2         -1.55275 -> -1.41122 -0.14152 
  56 Cmcm         -4.00846 -> -3.51128 -0.49718 
  57 P6/mmm       -2.83941 -> -1.97168 -0.86773 
  58 Fm-3m         0.00000 -> -0.01938  0.01938 
  59 P6/mmm       -1.46946 -> -1.22421 -0.24526 
  60 Fmmm         -1.46946 -> -1.68932  0.21986 
  61 P4/nmm       -1.46946 -> -2.31114  0.84167 
  62 Pm-3m        -1.94837 -> -1.94836 -0.00000 
  63 P6/mmm        5.65945 -> -0.09641  5.75586 
  64 Amm2         -1.71605 -> -4.24962  2.53357 
  65 Im-3m         0.00000 -> -0.77874  0.77874 
  66 R3m          -3.39185 -> -2.72299 -0.66886 
  67 P4/mmm       -1.76847 -> -2.13827  0.36981 
  68 P-43m         0.00000 -> -1.61958  1.61958 
  69 Pm           -3.14873 -> -2.93231 -0.21642 
  70 Pm-3m         0.00000 -> -0.60175  0.60175 
  71 R3m          -2.69904 -> -2.94531  0.24627 
  72 I4/mmm       -1.76847 -> -1.83718  0.06871 
  73 P6/mmm       -1.46946 -> -1.70653  0.23706 
  74 Pmm2         -2.12139 -> -2.93095  0.80956 
  75 P4/nmm       -1.46946 -> -1.67555  0.20609 
  76 Cmme         -2.93890 -> -1.49001 -1.44889 
  77 C2/c         -2.93893 -> -2.67002 -0.26891 
  78 Pmm2         -0.77637 -> -2.84035  2.06397 
  79 P4/nmm       -1.46946 -> -1.68632  0.21686 
  80 Im-3m         0.00000 -> -0.56590  0.56590 
  81 P4/nmm       -1.46946 -> -2.27192  0.80246 
  82 P2_1         -2.72004 -> -4.33915  1.61911 
  83 Im-3m         0.00000 -> -0.02413  0.02413 
  84 Pmmm         -4.07661 -> -2.36772 -1.70889 
  85 Cmcm         -4.07661 -> -3.19941 -0.87720 
  86 P6/mmm       -1.46946 -> -1.98967  0.52021 
  87 P-1          -3.62300 -> -3.59010 -0.03290 
  88 I4/mmm       -1.46946 -> -1.85945  0.38998 
  89 P1           -4.12657 -> -4.13611  0.00954 
  90 I-43m         0.00000 -> -1.02743  1.02743 
  91 Fd-3m        -3.74201 -> -3.74201 -0.00000 
  92 P1           -3.44201 -> -3.72848  0.28646 
  93 Pmma         -2.03867 -> -2.08921  0.05054 
  94 P4/mmm       -1.46946 -> -1.52159  0.05213 
  95 Cmmm         -2.22311 -> -2.70734  0.48423 
  96 P1           -2.93892 -> -1.68470 -1.25423 
  97 Fd-3m        -4.64995 -> -4.64996  0.00000 
Ground State is found at trial 98

[qzhu2017]

@yanxon Shouldn't the following line be np.argmin ? https://github.com/yanxon/CSP_BO/blob/fd140268813c40a0e6a1b399e5f1fb0c869db6d6/bo_run.py#L47

[qzhu2017]

@yanxon I just checked this video. It seems that your Thompson Sampling is correct. However, it should be np.argmin.

Have you finished the GP for energy estimation?

[yanxon]

@qzhu2017

I have not. I couldn't figure it out what's wrong with your code. Although I found several mistake to make the results better. I need to figure this out myself because it's important for me. Please let me take care of this today. If not, we can discuss about it tomorrow in the meeting.

The current result:

Previous result:

It seems like the std at the data point is still not behaving as it should.

[qzhu2017]

@yanxon How about your previous GPR code in Pyxtal_FF? Is that code not working?

[yanxon]

@qzhu2017

I believe it was working before and I showed it to you previously. However, I found out that there is a typo last night and I fix it in FF-project. GP may show a reasonable good fit (as you can see in the results above), but there might be bugs in the code.

Therefore, I believe it is crucial to get your GPR.ipynb working. Although it's a simple toy model, it should work properly before we move on to bigger code. At least we have something to verify it with.

[yanxon]

@qzhu2017

I think I figure it out. I use the exact training set to make the prediction with the trained GP. This should be what we expected