First level parallelisation of scripts running Ansys FEM simulator

nikosavola commented 1 year ago

Ansys simulation export could be extended to support second level of parallelisation (assuming the users' license configuration allows this) using the script in https://github.com/iqm-finland/KQCircuits/blob/main/klayout_package/python/scripts/simulations/elmer/scripts/simple_workload_manager.py as is done with Elmer.

The export_ansys API would ideally accept an n_workers argument, which would then export a .sh or .bat file that actually uses the simple workload manager for calling Ansys.

qpavsmi commented 1 year ago

Prerequisites

KQCircuits, GUI and standalone installations
Having Ansys-EDT (https://www.ansys.com/products/electronics) installed is useful for debugging, but technically this issue can be worked on without it. Note that the software is proprietary.

Issue description

One of the features of KQCircuits is to export simulation input data, which can be picked up by an external FEM tool in order to perform the simulations and to report results. We often need to perform multiple simulations to plot the results as a function of some feature in the geometry, for example we need to plot qubit shape's capacitance as a function of coupler width. Since we need to repeat the simulation process for multiple shapes, it makes sense to perform these simulations in parallel.

For Elmer we have this implemented already, but for Ansys-EDT we are missing first level parallelisation*. The implementation of the first level parallelisation for Elmer resides in simple_workload_manager.py where we use apply_async to run each Elmer call in parallel. The goal is to have the same workload manager to also run Ansys calls in parallel.

*) By first level parallelisation we effectively mean parallelisation outside of the execution of the FEM tool, usually for distinct simulations that don't depend on each other. Second level is parallelisation implemented within the FEM tool execution.

Explanation of the simulation export workflow in KQCircuits through example

In the klayout_package/python/scripts/simulations folder we keep simulation scripts that export files to be picked up by an external FEM tool. The scripts export many files, including geometry files in GDS format, some configuration files in JSON format, some python scripts copied from the KQCircuits repository and other files. Among the exported files is also a simulation script file, which is the main entry point for managing the sequence of python script and FEM tool invokations. The simulation file is a bash script (.sh) when exported for Elmer and a Windows batch script (.bat) for Ansys.

To see the exported files for yourself, go to the KQCircuits source code folder and run: python klayout_package/python/scripts/simulations/double_pads_sim.py This should take no longer than a minute, and will cause a KLayout window to pop up, the content of which might be informative but otherwise not useful for fixing this issue. More importantly, the python script will generate the tmp subfolder if not present already, with three folders inside:

tmp/double_pads_sim_output_eigenmode
tmp/double_pads_sim_output_q3d
tmp/double_pads_sim_output_elmer

So it exports one simulation configuration to be used by Elmer and two configurations for Ansys.

Let's review the contents of the double_pads_sim.py script we just ran. The relevant highlights are:

We iterate over sim_tools = ['elmer', 'eigenmode', 'q3d'], causing the script to export three simulation configurations
At the bottom, export_ansys and export_elmer are the key methods to export the files for different configurations. Notice the export_parameters_elmer and export_parameters_ansys dicts and how they are formed.
In export_parameters_elmer the n_worker entry is defined as 4. We don't have such entry in export_parameters_ansys. You can check how the simulation.sh is different if such entry is missing from Elmer parameters as well.
intermediate_processing_command entry in export_parameters_ansys is set if sim_tool is 'eigenmode' but not if it's 'q3d'.

Next, let's have a look at the contents of the Elmer configuration in tmp/double_pads_sim_output_elmer/simulation.sh. Seems to be a long one-liner: python scripts/simple_workload_manager.py 4 "./double_pads_island_dist_70_coupler_width_50.sh" "./double_pads_island_dist_70_coupler_width_88.sh" ...

The exported python script is a copy of simple_workload_manager.py. simulation.sh runs that script, setting the n_workers argument to 4 and the simulations argument to contain every .sh script in the tmp/double_pads_sim_output_elmer folder (except simulation.sh). Reading through simple_workload_manager.py you will find that it will allocate 4 worker threads to asynchronously execute each script listed in the simulations argument. Notice that if changes are made to simple_workload_manager.py, these changes will be available in the tmp/double_pads_sim_output_elmer/scripts/simple_workload_manager.py file after exporting the files with python klayout_package/python/scripts/simulations/double_pads_sim.py.

Finally let's review the contents of tmp/double_pads_sim_output_eigenmode/simulation.sh and tmp/double_pads_sim_output_q3d/simulation.sh. Before the echo Simulation 1/42 - ... command there are some powershell commands involving the blocking_pids.xml file. This ensures that multiple instances of the simulation.bat script are run one at a time. Let's not change this, since we will do the parallelisation within one single instance of a batch script. Then there are copy-pasted lines of echo Simulation 1/42 - ... and "%PROGRAMFILES%\AnsysEM\v222\Win64\ansysedt.exe" -scriptargs "xxx.json" -RunScriptAndExit "scripts\import_and_simulate.py", which invoke the Ansys-EDT executable to run a simulation. In tmp/double_pads_sim_output_eigenmode/simulation.sh we also have extra lines python "scripts/t1_estimate.py" "xxx.json", which were added due to the intermediate_processing_command entry in export_parameters_ansys. Then there is a final command "%PROGRAMFILES%\AnsysEM\v222\Win64\ansysedt.exe" -RunScriptAndExit "scripts\export_batch_results.py" to compile the results. The goal is to have the simulation.bat contain the big python scripts/simple_workload_manager.py 4 ... one liner that we saw in the Elmer case.

What needs to be done

simple_workload_manager.py python script should be moved to klayout_package/python/scripts/simulations/simple_workload_manager.py since we don't want this to be Elmer specific anymore
During Ansys simulation export, also export bash scripts like ./double_pads_island_dist_70_coupler_width_50.sh similar to what is exported for Elmer configuration, containing:
- progress print out echo Simulation 1/42 - xxx.json
- Ansys call "%PROGRAMFILES%\AnsysEM\v222\Win64\ansysedt.exe" -scriptargs "xxx.json" -RunScriptAndExit "scripts\import_and_simulate.py" -waitforlicense
- additional calls defined in intermediate_processing_command, if any.
All Ansys invokations should have -waitforlicense option. Ansys users are limited by the number of licenses they posess in order to run multiple Ansys processes simultaneously.

Modify ansys_export.py and make sure that the exported simulation.bat file has the following content when n_workers is set:

@echo off
powershell -Command "Get-Process | Where-Object {$_.MainWindowTitle -like \"Run Simulations*\"} | Select -ExpandProperty Id | Export-Clixml -path blocking_pids.xml"
title Run Simulations
powershell -Command "$sim_pids = Import-Clixml -Path blocking_pids.xml; if ($sim_pids) { echo \"Waiting for $sim_pids\"; Wait-Process $sim_pids -ErrorAction SilentlyContinue }; Remove-Item blocking_pids.xml"
python scripts/simple_workload_manager.py 4 "./double_pads_island_dist_70_coupler_width_50.sh" "./double_pads_island_dist_70_coupler_width_88.sh" <... other scripts in the folder>
"%PROGRAMFILES%\AnsysEM\v222\Win64\ansysedt.exe" -RunScriptAndExit "scripts\export_batch_results.py"

It's possible that simple_workload_manager.py requires no changes in code, but at least edit API docs to not mention "Elmer" specifically
Update documentation pages, probably needs changes in ansys_export.rst. No need to spend too much time on this, we will tell what to change during the review round.
No unit tests needed

How to verify that the feature works

Set some value to n_workers in export_parameters_ansys in double_pads_sim.py. Run the python script. Focus on tmp/double_pads_sim_output_eigenmode/ and tmp/double_pads_sim_output_q3d/.

If you have Ansys installed, running simulation.bat should work without failures. Note that it would require access to multiple licenses from Ansys license server to actually run the simulations in parallel, but in any case it should be able to run without failures.

If you don't have Ansys installed, make sure that simulation.bat has the required content. Also make sure that the scripts like ./double_pads_island_dist_70_coupler_width_50.sh get exported. Verify that simulation.bat would work if there were a working Ansys installation in the system by replacing the ansysedt.exe calls with some test script of your own that, for example, creates a file containing some text or a timestamp of creation. If n_workers is not set, the contents of simulation.bat should be the same as if run from KQCircuits main branch.

AbdullahKazi500 commented 1 year ago

Can I get assigned to this issue

qpavsmi commented 1 year ago

Can I get assigned to this issue

Yes and thank you for your interest! Once you have a solution ready please make a pull request and we will review the code quality and try out your solution on our Ansys environments. Don't hesitate to make a pull request earlier rather than later so that we could have some time before the deadline to reiterate your solution from our feedback if needed.

qpavsmi commented 1 year ago

Just wanted to make it clear: I would prefer to mark this issue to be assigned to someone after the pull request is created and I have reviewed and approved it. So feel free to work on the issue and create a pull request once you're ready to share it with us.

iqm-finland / KQCircuits