Missing mentioned paper

[HafizRiaz]

Dear Author, thanks for providing the base code for training volt/var. Actually the paper you have mentioned in the repo could not found by me after searching a lot from google scholar. Could you please share the said paper if published. I will appreciate your positive response. Another query is that, can we plot the voltage profile from the training data?

[jtscs]

Hello, yes, sorry, the paper hasn't been officially published yet, so for the moment I added it to the git repo where you can find it as the 9B_2_WISO23_187_paper_Türk_Jurek.pdf. I am not sure what you mean with plotting the voltage profiles from the training data, since the training data consist of a distribution grid model and load and generation power profiles (given in LoadProfile.csv and RESProfile.csv). The voltage profiles of a training/evaluation episode can be plotted. Currently only the evaluation episode saves data about the average voltage deviation in a eval_log.db file during training. Sadly these log files get some 100MB big and are to large to add to github but if you start the training you should see a database file get created after some time. If you want I can share with you an example output database of a training on a different platform. I also added some more stuff to the repo like plotting and evaluation scripts and example outputs. I will try to find some time the next days to document it a little. Feel free to ask if you have any further questions otherwise I hope the added material helps you :) Best regards, Jurek

[HafizRiaz]

Thanks for updating the repo and sharing the required material. By voltage plots, i meant to get the pIots similar to Figure 9-11 of your paper. Now I will try to run your repo to reproduce the results of paper. I am very thankful to you for this help. Stay blessed dear.

[jtscs]

Sure, I hope the update helps you with your work. Okay, I fear I forgot the script that creates figures 9-11, but it is also a rather simple one that is based on the other evaulation scripts. As the work with the software is not really described in the paper, the main workflow was to run a training, copy the trained model to the evaluation folder, run a evaluation over a full year period. The evaluation scripts create a big sql database with a lot more detail data like voltages which can then be easily plotted. I hope you can get it to run, it should only require a few python libs. A training run usually took my computer about a day, just so you know it runs for quiet some time :D

[HafizRiaz]

Ok sure i will try my best to get the results of Figure 9-11. however, if i could not produce the results from training and evaluation, will seek your help. Thanks for your value able time & Consideration.

[HafizRiaz]

Dear author, does this repository requires Linux operating system to run? As I have tried a lot to install a package " reverb " but gives error during installation. When I searched it , I get to know that it's not compatible with tensor flow in windows, rather I need to use Linux operating system.

Get Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: jtscs @.> Sent: Tuesday, October 17, 2023 1:24:34 PM To: jtscs/DRL_volt-var_control @.> Cc: Mehboob Riaz @.>; Author @.> Subject: Re: [jtscs/DRL_volt-var_control] Missing mentioned paper (Issue #1)

Sure, I hope the update helps you with your work. Okay, I fear I forgot the script that creates figures 9-11, but it is also a rather simple one that is based on the other evaulation scripts. As the work with the software is not really described in the paper, the main workflow was to run a training, copy the trained model to the evaluation folder, run a evaluation over a full year period. The evaluation scripts create a big sql database with a lot more detail data like voltages which can then be easily plotted. I hope you can get it to run, it should only require a few python libs. A training run usually took my computer about a day, just so you know it runs for quiet some time :D

— Reply to this email directly, view it on GitHubhttps://github.com/jtscs/DRL_volt-var_control/issues/1#issuecomment-1765918476, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BAK6YL5H7NTDPLKK2MOSUKDX7Y6EFAVCNFSM6AAAAAA56AARW6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRVHEYTQNBXGY. You are receiving this because you authored the thread.Message ID: @.***>

[jtscs]

Hello, I have only used the repo on ubuntu. I fear the scripts use the reverb python library ( should be this one https://pypi.org/project/dm-reverb/) which is only available for ubuntu, sorry. So yes, I think the repo can only be run under ubuntu at the moment, I will mention it in the describtion. You might be able to replace the type of replay_buffer but it would probably take quite a while to get that to work.

[HafizRiaz]

Ok go it. Do I need to change the replay buffer as well for running the repo on ubantu??

Get Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: jtscs @.> Sent: Thursday, October 19, 2023 12:20:33 PM To: jtscs/DRL_volt-var_control @.> Cc: Mehboob Riaz @.>; Author @.> Subject: Re: [jtscs/DRL_volt-var_control] Missing mentioned paper (Issue #1)

Hello, I have only used the repo on ubuntu. I fear the scripts use the reverb python library ( should be this one https://pypi.org/project/dm-reverb/) which is only available for ubuntu, sorry. So yes, I think the repo can only be run under ubuntu at the moment, I will mention it in the describtion. You might be able to replace the type of replay_buffer but it would probably take quite a while to get that to work.

— Reply to this email directly, view it on GitHubhttps://github.com/jtscs/DRL_volt-var_control/issues/1#issuecomment-1770209913, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BAK6YL3TUK622B5BF6X7RODYADIEDAVCNFSM6AAAAAA56AARW6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZQGIYDSOJRGM. You are receiving this because you authored the thread.Message ID: @.***>

[jtscs]

No, the training scripts should work as they come in the repo. For the evaluation scripts you probably need to adapt the path to the approprite database file. The path should always be found in the top of the file. I will write some documentation on the usage

[HafizRiaz]

That would be a great help for users. Thanks

Get Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: jtscs @.> Sent: Thursday, October 19, 2023 7:11:41 PM To: jtscs/DRL_volt-var_control @.> Cc: Mehboob Riaz @.>; Author @.> Subject: Re: [jtscs/DRL_volt-var_control] Missing mentioned paper (Issue #1)

No, the training scripts should work as they come in the repo. For the evaluation scripts you probably need to adapt the path to the approprite database file. The path should always be found in the top of the file. I will write some documentation on the usage

— Reply to this email directly, view it on GitHubhttps://github.com/jtscs/DRL_volt-var_control/issues/1#issuecomment-1771076311, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BAK6YL2YEY5FLAZGRTFU7FTYAEYJ3AVCNFSM6AAAAAA56AARW6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZRGA3TMMZRGE. You are receiving this because you authored the thread.Message ID: @.***>

[jtscs]

Hi, sorry it took me a while to find some time to update the README and the evaluation script folder but they should be easier to use now. Feel free to ask if you encounter any issues.

[HafizRiaz]

Thanks a lot, to spare sometime to update the required things. Actually i tried this on Google colab as I didn't have enough time to install Linux os for running the repository. But in tracing the error I got to know that the " package reverb" was written in python 2 and have not been updated since 2011 ( documentation). Is it possible that, you may use some alternative package so that your repository may be runa and used with convenience.

I will appreciate if you may try this. Actually, I am PhD scholar in a university and really we don't have enough resources in terms of required high specs system having GPUs for running deep learning based project like the one created by you although paper is well written and easy to understand. Overall, I found you very very cooperative, helping to others and courteous. Allah bless you with healthier and prosperous life. Stay blessed dear and I would be waiting for your kind response.

Get Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: jtscs @.> Sent: Tuesday, November 14, 2023 1:30:46 am To: jtscs/DRL_volt-var_control @.> Cc: Mehboob Riaz @.>; Author @.> Subject: Re: [jtscs/DRL_volt-var_control] Missing mentioned paper (Issue #1)

Hi, sorry it took me a while to find some time to update the README and the evaluation script folder but they should be easier to use now. Feel free to ask if you encounter any issues.

— Reply to this email directly, view it on GitHubhttps://github.com/jtscs/DRL_volt-var_control/issues/1#issuecomment-1809041465, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BAK6YL4JHC2E5R3NUQJ5ZPDYEJ7PFAVCNFSM6AAAAAA56AARW6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBZGA2DCNBWGU. You are receiving this because you authored the thread.Message ID: @.***>

[jtscs]

Hi, okay thanks for the information. I ran this repository also on a laptop without a gpu but having a custom python installation on ubuntu is necessary I fear. I will try to have a look if can replace reverb and/or make it work on google colab. Thanks for the sweet feedback :)

[HafizRiaz]

Dear you can see the problem here

[jtscs]

Hi, yes that looks like one of the reverb errors I encountered when I worked on the install_commands script. Since you mentioned that you work with google colab I put the code into a jupyter notebook and tested it. It still uses reverb but uses the install commands from the install_commands script. The Notebook is in the Jupyter_Notebook_Version Folder. I hope that helps you to get it to run. It worked for me but I did not run it for a full training and I fear that takes more in the range of 30 hours which you won't get with free google colab. I only read that 12h should be the maximum for the free version which should still give you good results since the learning curve drops pretty quickly towards the optimum.

[HafizRiaz]

I really appreciate your efforts. At this time i am using the free version of colab. Can I run this project for a shorter period of time (e.g. 1 or 2 or 3 months) instead of the whole year so that it may be trained in a shorter period of time? But i fear that it will not give the accurate results if the agent trains for lesser data. is it so ?

---

Engr. Hafiz Mehboob Riaz Lecturer, Electrical Engineering Department (E.E.D) Secretary, PLO-CQI Committee University of Engineering & Technology (U.E.T) Taxila, Pakistan Mobile # 0092-302-7259033

---

From: jtscs @.> Sent: Monday, November 20, 2023 2:42 AM To: jtscs/DRL_volt-var_control @.> Cc: Mehboob Riaz @.>; Author @.> Subject: Re: [jtscs/DRL_volt-var_control] Missing mentioned paper (Issue #1)

Hi, yes that looks like one of the reverb errors I encountered when I worked on the install_commands script. Since you mentioned that you work with google colab I put the code into a jupyter notebook and tested it. It still uses reverb but uses the install commands from the install_commands script. The Notebook is in the Jupyter_Notebook_Versionhttps://github.com/jtscs/DRL_volt-var_control/tree/main/Jupyter_Notebook_Version Folder. I hope that helps you to get it to run. It worked for me but I did not run it for a full training and I fear that takes more in the range of 30 hours which you won't get with free google colab. I only read that 12h should be the maximum for the free version which should still give you good results since learning curve drops pretty quickly towards the optimum.

— Reply to this email directly, view it on GitHubhttps://github.com/jtscs/DRL_volt-var_control/issues/1#issuecomment-1817986584, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BAK6YL6R7YSKW5P2X2N2GLDYFJ4MXAVCNFSM6AAAAAA56AARW6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJXHE4DMNJYGQ. You are receiving this because you authored the thread.Message ID: @.***>

[jtscs]

The majority of the learning (the improvement on the evaluation dataset) happens in the first quarter of the training. You should be able to set num_iterations to something like 300000-400000 which should also reduce the training time to 7-8 hours. You can also increase train_episodes_per_eval_episode to something in the range 3-6 I think with probably still getting good results, just a lot quicker.

[HafizRiaz]

Sure. I will do it and will share with you. The jupyter notebook added by you may also be run on jupyter notebook application available in anaconda?

Get Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: jtscs @.> Sent: Monday, November 20, 2023 1:41:02 PM To: jtscs/DRL_volt-var_control @.> Cc: Mehboob Riaz @.>; Author @.> Subject: Re: [jtscs/DRL_volt-var_control] Missing mentioned paper (Issue #1)

The majority of the learning (the improvement on the evaluation dataset) happens in the first quarter of the training. You should be able to set num_iterations to something like 300000-400000 which should also reduce the training time to 7-8 hours. You can also increase train_episodes_per_eval_episode to something in the range 3-6 I think with probably still getting good results, just a lot quicker.

— Reply to this email directly, view it on GitHubhttps://github.com/jtscs/DRL_volt-var_control/issues/1#issuecomment-1818463880, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BAK6YLYRPZ6VEKOY732PZCLYFMJR5AVCNFSM6AAAAAA56AARW6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJYGQ3DGOBYGA. You are receiving this because you authored the thread.Message ID: @.***>

[jtscs]

I haven't tried that and I am not to familiar with jupyter notebooks so I don't know... But I would think so if it can load the file.

[HafizRiaz]

Thanks for your time. I will try to run it on google colab and hopefully it will give result. As you have already tested it.

---

Engr. Hafiz Mehboob Riaz Lecturer, Electrical Engineering Department (E.E.D) Secretary, PLO-CQI Committee University of Engineering & Technology (U.E.T) Taxila, Pakistan Mobile # 0092-302-7259033

---

From: jtscs @.> Sent: Monday, November 20, 2023 1:51 PM To: jtscs/DRL_volt-var_control @.> Cc: Mehboob Riaz @.>; Author @.> Subject: Re: [jtscs/DRL_volt-var_control] Missing mentioned paper (Issue #1)

I haven't tried that and I am not to familiar with jupyter notebooks so I don't know... But I would think so if it can load the file.

— Reply to this email directly, view it on GitHubhttps://github.com/jtscs/DRL_volt-var_control/issues/1#issuecomment-1818478915, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BAK6YL3J72H7T2QFBF7M2V3YFMKYRAVCNFSM6AAAAAA56AARW6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJYGQ3TQOJRGU. You are receiving this because you authored the thread.Message ID: @.***>

[jtscs]

Hi, if you are still interested in a version that can run under windows, it is available in this branch windows-version_without-reverb.

[HafizRiaz]

Thank you for your time. I really appreciate your efforts. Although I installed ubantu but there were some issues which were not being handled by me. Now I will try it on windows... hopefully it will work there

Get Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: jtscs @.> Sent: Friday, December 22, 2023 12:14:25 AM To: jtscs/DRL_volt-var_control @.> Cc: Mehboob Riaz @.>; Author @.> Subject: Re: [jtscs/DRL_volt-var_control] Missing mentioned paper (Issue #1)

Hi, if you are still interested in a version that can run under windows, it is available in this branch windows-version_without-reverbhttps://github.com/jtscs/DRL_volt-var_control/tree/windows-version_without-reverb.

— Reply to this email directly, view it on GitHubhttps://github.com/jtscs/DRL_volt-var_control/issues/1#issuecomment-1866809606, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BAK6YLY4VNYYWYOG6OH4Q5DYKSDBDAVCNFSM6AAAAAA56AARW6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRWHAYDSNRQGY. You are receiving this because you authored the thread.Message ID: @.***>

[HafizRiaz]

Hi, if you are still interested in a version that can run under windows, it is available in this branch windows-version_without-reverb.

One important point is which specific version of python and editor (Anaconda, pycharm) should i installed on windows for running your updated version for windows? as in windows OS, python version does matter

[HafizRiaz]

Hi dear, i am trying to run the centralized training code on windows, although it started training but after waiting for almost 12 hours, it is still in progress. is there any specific indication (like message display of "training has been completed") for the completion of training process? And on completing the training, would i use "plot_training_progress.py" file (provided in repo) to reproduce the results of Fig. 9 of paper, or we have to write separate code for window to do this? Due to your kind response i have learnt a lot in my PhD research. Your positive response would be highly appreciated. May you have healthier and prosperous life. Really remember you in my prayers. thanks

[HafizRiaz]

I have compared the training session with the one you have uploaded in jupyter notebook version. you can see that my training limit has reached eval episode is at step 1024840 , while Google colab training history shared by you ended at "eval episode at step 58648". now i am worried how to stop or terminate this training session to plot the results??

[jtscs]

Hi, the google colab session was ended prematurely, not sure if I manually stopped it or if it reached a limit of my fere version. The standard training script runs for 1.5 million training steps which a lot more than it needs to reach a good value. It should have a pretty good value after about 100-200 thousand training steps. The full run of 1.5 million steps took my computer a bit more than a day, so yes it runs for quite some time. plotting_scripts/plot_training_progress.py should give you something like figure 7 which shows that there was a learning improvement and that the a value close to the optimum was reached. I just chose to train it very long to make sure the was no further improvement. The training can be stopped prematurely in most cases without a problem since the latest best actor net gets saved during training. To get a plot like figure 9 you need to copy your actor_policy folder into evaluation_scripts/centralized_training/ and then run pp_cigre_dist_pv-wind_with_load_profiles_q-characteristic_actor.py. This will create a database with a full year run with your trained actor net. Then you can use the single_day_eval.py script to create plots like figure 9. Before you start it, you will likely have to change which databases the script uses which is badly defined in the middle of the script around lines 387-389.

[HafizRiaz]

Thanks for your kind response. It means we may reduce the training from 1.5 million to have trained agent earlier with good training data for saving the time

Get Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: jtscs @.> Sent: Monday, January 15, 2024 10:38:26 PM To: jtscs/DRL_volt-var_control @.> Cc: Mehboob Riaz @.>; Author @.> Subject: Re: [jtscs/DRL_volt-var_control] Missing mentioned paper (Issue #1)

Hi, the google colab session was ended prematurely, not sure if I manually stopped it or if it reached a limit of my fere version. The standard training script runs for 1.5 million training steps which a lot more than it needs to reach a good value. It should have a pretty good value after about 100-200 thousand training steps. The full run of 1.5 million steps took my computer a bit more than a day, so yes it runs for quite some time. plotting_scripts/plot_training_progress.py should give you something like figure 7 which shows that there was a learning improvement and that the a value close to the optimum was reached. I just chose to train it very long to make sure the was no further improvement. The training can be stopped prematurely in most cases without a problem since the latest best actor net gets saved during training.

— Reply to this email directly, view it on GitHubhttps://github.com/jtscs/DRL_volt-var_control/issues/1#issuecomment-1892568749, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BAK6YL4JGF7TVO5KWGXKSJLYOVSRFAVCNFSM6AAAAAA56AARW6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJSGU3DQNZUHE. You are receiving this because you authored the thread.Message ID: @.***>

[HafizRiaz]

Hi there, i want to read the reference 19 of the paper. but i am unable to get it. Would you please share the link of that paper? moreover, i have trained the model using window version provided by you for centralized training. What would i have to do to get the voltage plots of evaluations script?

[jtscs]

Hi, sorry, forgot a bit to answer. Sadly reference 19 is not available for free and I can't share it with you. You can buy it here (https://www.vde-verlag.de/normen/0100735/vde-ar-n-4110-anwendungsregel-2023-09.html) but it is in German so I am not sure if will help you. I can try to answer specific questions you have about it though. For the training process plot you adjust the path in plotting_scripts/plot_training_progress.py and simply run the script. For the evaluation_scripts you copy the actor_policy folder (or folders in decentralized_training) into the evaluation folder where the scripts are. Then you run the pp_cigre_dist_pv-wind_with_load_profiles_q-characteristic_actor.py script. This will create a sql database with the evaluation data. You need to reference the path/change the path in evaluation_scripts/centralized_training/database_evaluation.py and evaluation_scripts/centralized_training/single_day_eval.py to get all evaluation data and plots.

[HafizRiaz]

Hi, i have obtained the voltage deviation plot after running the evaluation file as directed by you. i was wondering that voltage curve for decentralized training and all the methods have been plotted even i just trained the centralized agent (bcz of window version availability of just centralized method). i think this the main reason behind that voltage curve deviation from 1.04 reaching about 1.05 whuch is obvious from attached file Am i right? Moreover, can i use the same code for a different test system like ieee 33, 69 or 123 bus in place of Cigre medium network by just changing the pandapower network command or i have to change the loads assocuated with newer system as well?

[jtscs]

Hi, sorry for the late answer. Glad you were able to run the project. Sadly the single day eval script is quite messy. In line 387-389 the eval_env_log.db files are referenced but the standard does not fit to the names. In the folder is also a centralized_eval_env_log.db and qu_eval_env_log.db file from me that are referenced also as decentralized because the names are wrong. The main idea of the script is to load three db file called actor, ref and ref_2 and plot the voltage plots for them. I will try to find some time soon to clean the script up a bit but for now you will have to check the loaded db files and how they are called in the plot in detail. When you change the pandapower network you should also check if the way the generator powers and loads are changed. As both are accessible as lists or pandas DataFrames, I am not sure, the way they are changed can stay the same but if their count changes the load and generation power setting needs to be adapted. Does that answer your question?

[HafizRiaz]

thanks for your continuous response of my queries. Regarding your last reply, the second answer related to pandapower makes sense but i was really unable to understand the answer regarding "eval_env_log.db files". what names are wrong, and what code lines in what files should i change to obtain the standard results. As my underestimating, you provided the window version of centralized training script, but in the plots, all the 6 methods have been plotted simultaneously for comparison though just centralized trained network is run by me in window version. ? would you please explain the first reply in easy to understand language?

[jtscs]

Hello, I reworked the script in the hopes that it will be more understandable. The script loads three databases that hold the evaluation data from one year with a volt var strategy, These databases can be created with the following scripts pp_cigre_dist_pv-wind_with_load_profiles_q-characteristic_actor.py or pp_cigre_dist_pv-wind_with_load_profiles_q-characteristic_Q(U)_env.py. The databases and the according names are mentioned in the first lines of the script where you can adjust which databases are used and which names they have. currently two example databases from me are used. I hope that helps :)

[HafizRiaz]

Hi there, i appreciate your efforts and valuable time spent for the repo. I will go through it.