Proposed modifications to CfdOF to utilize a remote computer for meshing and simulation tasks.

[linuxguy123]

I'm thinking of modifying CfdOF so that the user can run meshes and simulations locally or on a remote computer. I'm looking for input before I do anything.

Generating large meshes and simulating them can take considerable time on even the fastest desktop CPU. In some cases hours or even days. Meanwhile the workstation machine is tied up and often not very responsive because the CPU core loads are high and the memory bandwidth is almost entirely consumed. The storage device will also be accessed frequently, further slowing down the local machine.

A better way to handle large meshing and simulation tasks is to off load them to a second (remote) computer, leaving the workstation almost unhindered while they run. Furthermore the remote computer can be optimized to run mesh and simulation loads much faster than a desktop machine might be.

I presently use a remote computer to generate meshes and run OpenFOAM simulations. I do this by editing the mesh and OF case files on my workstation, then copying them to the remote machine, logging into the remote machine with ssh and then running the tasks in the ssh session. I'm doing most of this without CfdOF, manually.

While this approach works, it is cumbersome. One is always moving case files back and forth between the workstation and the remote machine as well as the results from the remote machine back to the workstation. More than once I thought I was looking at a mesh or simulation result that just ran when, in fact, it wasn't the current output. Ugghhh.

I'd like to streamline the use of a remote computer in CfdOF, so that it happens automagically from the CfdOF GUI.

I envision doing this as follows:

1) Add a Remote tab to the CfdOF Preferences GUI for the remote connection parameters and set up. The Remote tab would have a checkbox to allow the use of a remote computer and fields for the remote hostname and username on the remote computer as well as a way to test it the connection. Ie can one successfully invoke an ssh session with the remote computer ?

The Remote tab will also allow setting up OpenFOAM, gmsh, cfmesh and Hisa on the remote machine, just as how it happens on the local computer right now.

I want the remote computer to be set up exactly as the local computer is set up as far as the tools go. That way anything that can be done in a worker thread on the local machine could also be done in a worker thread on the remote machine in a ssh session. With a little tweaking, of course.

For now I will not handle setting up Docker on the remote machine, just because I am not terribly familiar with Docker. I have used virtual environments, but not Docker. I may ask for assistance on this part of the task or leave it unimplemented.

2) In the CFD Mesh GUI I will change the "Write mesh case" button to "Write mesh case locally" and add a "Write mesh case remotely" button. The former will do as it does now. The later will write the mesh case to the working directory on the remote computer.

Likewise the Check Mesh button will also have a local and remote version.

I will also change the "Run mesher" button to "Run mesher locally" and add a "Run mesher remotely" button. The former will do as it does now. The later will mesh the mesh case that has been written on the remote machine and display the progress results in the Status box, just as it does now.

Having local and remote versions of these buttons will allow the user to use CfdOF exactly as it is presently used or with a remote computer as the user decides when working on the project. If the remote machine is busy running a mesh or simulation for another FreeCAD instance, the user may want to run meshes or simulations locally. Having both versions of the buttons allows the user the option of doing either.

Upon completion of the remote meshing task, the results will automatically be copied back to the local machine so they can be viewed (Load surface mesh) just as if they were meshed locally.

3) In the CFD Solver GUI the Write button will have a local and remote version, just as the Mesh GUI did.

I haven't figured out how to eloquently handle Editing the remote files. My best solution so far is to write the local files, allow the user to edit them locally and then have a "Copy to remote" button that will copy the entire (edited) case to the remote machine.

The Run button will have a local and remote version. The results of the remote run will display in the Status box and the Reporting Functions just as they do now.

Upon completion of a remote run the results will be copied back to the local machine, just as if it ran locally. Thus the Paraview button will work exactly as it does now, by using Paraview locally on locally stored results.

The availability of all the remote buttons will be controlled by the "Allow remote computations" checkbox on the Remote tab. If the "Allow remote computations" CB isn't checked, all the remote buttons will be disabled, leaving the user only the local buttons. Thus if the "Allow remote computations" button isn't checked, CfdOF will behave exactly as it does now.

SSH will require a password every time a new session is invoked. I'll ask the local user for the remote user's password the first time a remote operation (via ssh) is needed, but not store it. The ssh session should stay available forever unless something happens to one of the computers, ie gets powered down or goes to sleep (timeout). I'll check that the ssh session is still alive every time before using it. If the ssh session has ended, I'll invoke a new one, in which case I'll have to ask the user for the password again.

I also want to (eventually) handle the case of the workstation going to sleep, or being shut down or the user quitting FreeCAD while a remote task is running. I think I can do this by opening a new ssh connection when CfdOF is restarted and reattaching to the existing task. I'm not sure how I'll handle the data that will be missing while the connection is lost. I may have to buffer it on the remote side and redisplay it on the local side when the connection is resumed.

It seems like a lot of work to allow the workstation FreeCAD session to disconnect and resume, but it allows the user the flexibility to reboot the workstation, shut it off overnight, etc., while a task is running on the remote machine.

I am not calling the local machine a host and the remote machine a server because this isn't a true client/server relationship in the traditional sense. Right now I'm using the terminology of local and remote. I'm open to suggestions on the terminology aspect of it.

Theoretically the remote machine will allow multiple FreeCAD "clients" to connect to it, as long as each FreeCAD connection uses a different directory to store the results in. By allocating each client a subset of the total number of cores, each may have a decent experience, ie realtime but slower compared to having exclusive access to the remote machine. For now the allocation of cores will be done manually by the local user, not automatically.

The highest incarnation of using a remote computer for CfdOF processing would be the use of an AWS server or similar. I'm not saying that the work I would do would achieve this. But I do think it would be a step in the right direction. Right now it is quite cumbersome to use an AWS server for OpenFOAM work, for the same reasons I experienced and mentioned. If the remote functionality of the proposed enhancement were done right it could really streamline working with an AWS server.

Not everyone can afford a dedicated EPYC server beside their desk to run their OpenFOAM tasks. But just about everyone can afford to buy some time on an AWS server or make use of a virtual instance granted to them on a server at their local university. Adding remote processing capabilities to CfdOF would really steamline this workflow.

One of the advantages of using an ssh session to do the work on the remote computer is that just about every OS available already has ssh on it. While we could write a special server to run on the remote computer to interface with CfdOF, it would require special setup on the remote machine and I don't see that it would add a lot of value. If one uses ssh sessions, all one needs to do is ensure that ssh is set up on the remote computer.

I'm hoping to add my code to the existing worker thread routines, adding cases for the remote functionality. The alternative would be to duplicate the existing worker thread routines and create local and remote versions of each. I think there will be a lot of code commonality and having duplicates of each would increase maintenance effort in the future. However, I understand that there is a case for leaving the present routines untouched and developing stand alone remote routines. I await your input on this.

At this point I'm doing this just for me to test the concept. These changes may never be published if I find it too cumbersome in use. I'm asking for input now so that if remote processing does make it to the public that what I'm doing has a good chance of being accepted without making a lot of changes. I'm also looking for better ideas on how it might be done.

Thanks for your attention. Thoughts ? Ideas ? Advice ? Feedback ?

[luzpaz]

Is there a way to reliably reload a workbench in FreeCAD without restarting FreeCAD?

When I make a change to my code, I'm currently having to restart FreeCAD to load the changed code and test it. Is there a way to reload the workbench such that I don't have to restart FreeCAD ?

@linuxguy123 check out https://github.com/FreeCAD/FreeCAD-addons/issues/159

[linuxguy123]

@linuxguy123 check out FreeCAD/FreeCAD-addons#159

What is the library name to reload CfdOF ?

[luzpaz]

@oliveroxtoby do you mind weighing-in ?

[linuxguy123]

@linuxguy123 check out FreeCAD/FreeCAD-addons#159

Is there a (re)load(workbench) that one could use to reload CfdOF without restarting FreeCAD ?

[howetuft]

Is there a way to reliably reload a workbench in FreeCAD without restarting FreeCAD?

When I make a change to my code, I'm currently having to restart FreeCAD to load the changed code and test it. Is there a way to reload the workbench such that I don't have to restart FreeCAD ?

If that can help: For Render workbench, facing the same problem, I've chosen to write a function that reloads the workbench. The function is inside the workbench, the code is here: https://github.com/FreeCAD/FreeCAD-render/blob/master/Render/utils.py#L101 It is exposed in the workbench API, see here: https://github.com/FreeCAD/FreeCAD-render/blob/master/Render/__init__.py#L64

Therefore, now, to reload the workbench, I just enter Render.reload() in the console. Actually, with autocompletion, I just type 2 or 3 letters. You could even imagine to have it embedded in a button on the toolbar or anything that saves you some time. To be complete, beforehand, I have to import the workbench once (import Render), I do it once and for all at the beginning of my session.

[oliveroxtoby]

@linuxguy123 check out FreeCAD/FreeCAD-addons#159

What is the library name to reload CfdOF ?

You need to reload the specific python module or modules that you have changed. There are a few of the importlib commands scattered in CfdOF to do this which I put in as I needed them, but it's not comprehensive. @howetuft's solution is the best way to go, but would still need to be implemented in CfdOF.

[luzpaz]

@howetuft's solution is the best way to go, but would still need to be implemented in CfdOF.

Should we open a new feature request for this ?

[linuxguy123]

Update I'm still working on this project. Just busy this week. I'll be sharing an update in the near future.

[adrianinsaval]

assuming this isn't dead, will it support using https://cfd.direct/cloud/?

[linuxguy123]

assuming this isn't dead, will it support using https://cfd.direct/cloud/?

Not sure yet. I'd have to have a closer look.

There is an issue with the CfdOF Preference pages not fitting well on 1920x1080 displays. See attached.

I'm not going to address this now.

[linuxguy123]

What command can I use to get an OpenFOAM version without triggering an error because I don't have any input files ? Something like $simpleFoam -version. If I run $simpleFoam I get a string I can parse the version out of, but it returns an error because there are no input files.

Update The project is coming along really well, now that I am back working on it.

It turns out that RunFoamCommand works with ssh commands. Kudos to the developers for writing a function that I can use and extend without too much trouble.

Hardest part of the whole project was setting up the remote preferences, which is mostly complete, for a Linux remote computer anyway. I'll leave Windows and Docker stuff to other people, though it will be basically the same for a remote computer as for the local machine, so 95% of the existing code can be reused.

This will make sense when I share my code and you see what I've done.

These modifications should make it very practical to mesh and run OpenFOAM on remote computers. When I started the project I was worried the "linkage" would be finicky and not robust. But it's actually quite robust.

[linuxguy123]

[linuxguy123]

The mesh case can be written and run locally or remotely.

All display/visualization is done locally, by copying the completed mesh back from the remote machine.

[linuxguy123]

Ignore the number of threads parameter. It isn't used anymore. It's a relic of past testing and I don't know how to get it out of the data. The number of threads per process is setable for local and remote individually, in case the users wants to do something different on each machine.

[linuxguy123]

I've run into an issue - ./Allmesh runs fine locally, but the exact same mesh case fails on the remote machine. Why ?

The error on the remote run is:

--> FOAM FATAL IO ERROR: (openfoam-2206 patch=221104)
Unknown objectRefinement type insideSurface

Valid objectRefinement types :
[default: objectRefinement]5(box cone hollowCone line sphere)

file: system/meshDict.objectRefinements.MeshRefinement at line 26 to 28.

    From static Foam::autoPtr<Foam::Module::objectRefinement> Foam::Module::objectRefinement::New(const Foam::word&, const Foam::dictionary&)
    in file utilities/octrees/meshOctree/refinementControls/objectRefinement/newObjectRefinement.C at line 56.

FOAM exiting

My code builds the mesh case on the local machine and then copies it to the remote machine. The only difference in the build cases is the number of threads and processors is set differently for each.

Both machines are running OpenFOAM-2206, installed from the same source. Why would ./Allmesh run differently on one machine than the other ?

Is there anything besides the meshCase folder that needs to be copied to the remote machine to have ./Allmesh run properly ?

See attached session files below.

runMeshRemote.txt runMeshLocal.txt

Update

I found the problem. The machines have different versions of cfMesh installed. I'll update cfMesh on the remote machine and report back.

The local machine has cfMesh-CfdOF (cfMesh 1.1.2) 1.19 and it works.

I didn't implement the code to install cfMesh on the remote machine. I guess it is time to do that. It also appears that I should use more stringent cfMesh version checking. This is what happens when you eat your own dogfood !

[linuxguy123]

This project is coming along very well. Remote meshing and FOAMing should be operational with Linux local and remote computers shortly. From there "we" can add functionality for other local/remote OS combinations and the use of docker.

For me, remote processing is going to dramatically improve my CFD work flow. It will be a night and day difference. Moving files back and forth between my workstation and my CFD server was extremely tedious. It's fantastic to just press "Write mesh case on remote host" and it is over there, ready to run ! That is a game changer for me.

My CFD server is an AMD Epyc 7601 with 32 cores and 8 memory channels. It meshes about 6x faster than my present workstation (AMD 5900X) and runs FOAM about 3.5x faster. And my workstation doesn't get tied up while doing this work ! I might upgrade my CFD server to a dual EPYC 7002 configuration which would be even faster.

In the past I've only used the CFD server for large jobs because of the tedium of moving the files back and forth. I had scripts to do this, but it was still a hindrance. With remote processing built into CfdOF, I'll be using my CFD server all the time.

For me, the combination of FreeCAD, CfdOF and remote processing makes this a very high quality CFD tool. Up there with the likes of the commercial packages, not that I've used them very much.

I thought of another enhancement in CfdOF I would like to have - comparing result "sets". Whenever I'm doing CFD work, I'm trying to optimise geometries. I do this by tweaking the CAD object, running the CFD and checking the results, over and over. I'm thus doing lots of Save As... and then I usually have more than 1 project open at once so I can compare differences between geometries and results. It would be nice if this was streamlined somehow, to make it easier to do.

This isn't something I'm going to work on now or soon, but I'm interested in hearing people's thoughts on how it could be done better. Some sort of "diff" tool to compare the results of two or more simulations ?

[linuxguy123]

This is how I installed cfMesh on the remote host.

#**********************************************************************************************
       def remoteDownloadInstallCfMesh(self):

        #TODO: this routine doesn't clean up after itself if it fails
        # Nor does it check for a full or partial build
        # If the user reruns the build after fully or partially building previously
        # this routine will fail.

        # Get the username and hostname for the remote host
        prefs = CfdTools.getPreferencesLocation()
        remote_user = FreeCAD.ParamGet(prefs).GetString("RemoteUsername", "")
        remote_hostname =FreeCAD.ParamGet(prefs).GetString("RemoteHostname", "")
        ssh_prefix = "ssh -tt " + remote_user + "@" + remote_hostname + " "
        working_dir = remote_hostname =FreeCAD.ParamGet(prefs).GetString("RemoteOutputPath", "")

        self.consoleMessage("Installing cfMesh on the remote host...")

        # create a dir to download the CfMesh source files to
        # cfMesh is installed in the user's home directory, not the output directory

        command = "EOT\n"
        #command += "cd " + working_dir + "\n"
        command += "mkdir cfMesh" + "\n"
        command += "exit \n"
        command += "EOT"

        try:
            command_result = CfdTools.runFoamCommand(ssh_prefix + "<< " + command)[0]
            working_dir += "/cfMesh"

            print("Created cfMesh install dir: " + working_dir)
            self.consoleMessage("Created cfMesh install directory " + working_dir)
        except:
             self.consoleMessage("Could not create an install directory on the remote host.")
             return

        # download the cfMesh source files
        command = "EOT\n"
        command += "cd " + working_dir + "\n"
        command += "wget "  + self.form.le_cfmesh_url.text() + "\n"
        command += "exit \n"
        command += "EOT"

        try:
             command_result = CfdTools.runFoamCommand(ssh_prefix + " << " + command)[0]
             self.consoleMessage("Downloaded cfMesh source files")
        except:
             self.consoleMessage("Could not download cfMesh source files.")
             return

        # unzip the downloaded file
        # TODO: this assumes the downloaded archive is called "download".  Which might not always be the case.
        # Get name of the downloaded file from the URL and use that filename in this command
        command = "EOT\n"
        command += "cd " + working_dir + "\n"
        command += "unzip download \n"
        command += "exit \n"
        command += "EOT"

        try:
             command_result = CfdTools.runFoamCommand(ssh_prefix + " << " + command)[0]
             self.consoleMessage("Unzipped cfMesh source files")
        except:
             self.consoleMessage("Could not unzip cfMesh source files.")
             return

        # run the wmake process
        # TODO: add the number of processes to use on the build with export WM_NCOMPPROCS=`nproc`;
        # TODO: add logging wiht ./Allwmake  ->  log.Allwmake
        # TODO: delete the previous log before starting the new build
        # Executing: { rm -f log.Allwmake; export WM_NCOMPPROCS=`nproc`; ./Allwmake 1> >(tee -a log.Allwmake) 2> >(tee -a log.Allwmake >&2); } in $WM_PROJECT_USER_DIR/cfmesh-cfdof

        command = "EOT\n"
        command += "cd " + working_dir + "/cfmesh-cfdof \n"
        command += "./Allwmake \n"
        command += "exit \n"
        command += "EOT"

        self.consoleMessage("Starting cfMesh build on remote host.")
        self.consoleMessage("This takes a bit of time.  Please wait.")
        try:
             command_result = CfdTools.runFoamCommand(ssh_prefix + " << " + command)[0]
             self.consoleMessage("Successfully built and installed cfMesh.")
        except:
             self.consoleMessage("Could not build cfMesh.")
             return

[linuxguy123]

I can't get CfdConsoleProcess.waitForStarted() to be true after starting my meshing process in a proxy. But my meshing process works with CfdRunFoamCommand outside of a proxy.

Is this a race condition ?

Here is the code that works using CFDRunFoamCommand() outside of a proxy.

 # Get the username and hostname for the remote host
        prefs = CfdTools.getPreferencesLocation()
        remote_user = FreeCAD.ParamGet(prefs).GetString('RemoteUsername', '')
        remote_hostname =FreeCAD.ParamGet(prefs).GetString('RemoteHostname', '')

        # create the ssh connection command
        ssh_prefix = 'ssh -tt ' + remote_user + '@' + remote_hostname + ' '

        # Get the working directory for the mesh
        working_dir = remote_hostname =FreeCAD.ParamGet(prefs).GetString('RemoteOutputPath', '')

        # create the command to do the actual work 
        command = 'EOT \n'
        command += 'cd ' + working_dir + '/meshCase \n'
        command += './Allmesh \n'
        command += 'exit \n'
        command += 'EOT'
        command = ssh_prefix + ' << '  + command

        self.consoleMessage("Starting remote meshing...")
        try:
            CfdTools.runFoamCommand(command)
            print("Remote meshing is complete.")
            self.consoleMessage("Remote meshing is complete.")
        except Exception as error:
             self.consoleMessage("Error meshing on remote host: " + str(error))

Here is the code that uses CfdConsoleProcess and proxy that doesn't work:

# run on remote host
            else:
                 # Get the username and hostname for the remote host
                 FreeCADGui.doCommand("prefs = CfdTools.getPreferencesLocation() \n" +
                                       "remote_user = FreeCAD.ParamGet(prefs).GetString('RemoteUsername', '')\n" +
                                       "remote_hostname =FreeCAD.ParamGet(prefs).GetString('RemoteHostname', '')")

                 # create the ssh connection command
                 FreeCADGui.doCommand("ssh_prefix = 'ssh -tt ' + remote_user + '@' + remote_hostname + ' '")

                 # Get the working directory for the mesh
                 FreeCADGui.doCommand("working_dir = remote_hostname =FreeCAD.ParamGet(prefs).GetString('RemoteOutputPath', '')")

                 # create the command to do the actual work
                 FreeCADGui.doCommand("command = 'EOT \\n' \n" +
                                      "command += 'cd ' + working_dir + '/meshCase \\n' \n" +
                                      "command += './Allmesh \\n' \n" +
                                      "command += 'exit \\n' \n" +
                                      "command += 'EOT' \n")
                 FreeCADGui.doCommand("command = ssh_prefix + ' << '  + command + '\\n'")

                 FreeCADGui.doCommand("print(command)")

                 FreeCADGui.doCommand("proxy.running_from_macro = True")
                 self.mesh_obj.Proxy.running_from_macro = False

                 FreeCADGui.doCommand("if proxy.running_from_macro:\n" +
                                      "  mesh_process = CfdConsoleProcess.CfdConsoleProcess()\n" +
                                      "  mesh_process.start(command)\n" +
                                      "  mesh_process.waitForFinished()\n" +
                                      "else:\n" +
                                      "  proxy.mesh_process.start(command)")

            if self.mesh_obj.Proxy.mesh_process.waitForStarted():
                # prevent user form running a second instance
                self.form.pb_run_mesh.setEnabled(False)
                self.form.pb_run_remote_mesh.setEnabled(False)

                # enable/disable the correct buttons
                if not remotely:
                    self.form.pb_stop_mesh.setEnabled(True)
                    self.form.pb_write_mesh.setEnabled(False)
                    self.form.pb_check_mesh.setEnabled(False)
                    self.form.pb_paraview.setEnabled(False)
                    self.form.pb_load_mesh.setEnabled(False)
                else:
                    self.form.pb_stop_remote_mesh.setEnabled(True)
                    self.form.pb_write_remote_mesh.setEnabled(False)
                    self.form.pb_check_mesh.setEnabled(False)
                    self.form.pb_paraview.setEnabled(False)
                    self.form.pb_load_mesh.setEnabled(False)
                self.consoleMessage("Mesher started ...")
            else:
                self.consoleMessage("Error starting meshing process", 'Error')

Here is the code that is executing in the Python console:

>>> from CfdOF.Mesh import CfdMeshTools
>>> from CfdOF import CfdTools
>>> from CfdOF import CfdConsoleProcess
>>> cart_mesh = CfdMeshTools.CfdMeshTools(FreeCAD.ActiveDocument.Cut_Mesh)
>>> proxy = FreeCAD.ActiveDocument.Cut_Mesh.Proxy
>>> proxy.cart_mesh = cart_mesh
>>> cart_mesh.error = False
>>> prefs = CfdTools.getPreferencesLocation() 
>>> remote_user = FreeCAD.ParamGet(prefs).GetString('RemoteUsername', '')
>>> remote_hostname =FreeCAD.ParamGet(prefs).GetString('RemoteHostname', '')
>>> ssh_prefix = 'ssh -tt ' + remote_user + '@' + remote_hostname + ' '
>>> working_dir = remote_hostname =FreeCAD.ParamGet(prefs).GetString('RemoteOutputPath', '')
>>> command = 'EOT \n' 
>>> command += 'cd ' + working_dir + '/meshCase \n' 
>>> command += './Allmesh \n' 
>>> command += 'exit \n' 
>>> command += 'EOT' 
>>> 
>>> command = ssh_prefix + ' << '  + command + '\n'
>>> print(command)
>>> proxy.running_from_macro = True
>>> if proxy.running_from_macro:
>>>   mesh_process = CfdConsoleProcess.CfdConsoleProcess()
>>>   mesh_process.start(command)
>>>   mesh_process.waitForFinished()
>>> else:
>>>   proxy.mesh_process.start(command)

print(command) is returning this, which is correct. This same command is operating correctly with runFoamCommand outside of a proxy.

ssh -tt me@david  << EOT 
cd /tmp/meshCase 
./Allmesh 
exit 
EOT

The proxy code is failing on if self.mesh_obj.Proxy.mesh_process.waitForStarted():, which is returning false. When that happens it is executing: self.consoleMessage("Error starting meshing process", 'Error') and nothing else.

Why is waitForStarted() failing ?

How can I get a better error message as to what is going on ?

Update

I added time.sleep(2) before checking waitForStarted(). It made no difference.

            time.sleep(2)
            if self.mesh_obj.Proxy.mesh_process.waitForStarted():

[linuxguy123]

Where does proxy.mesh_process get set up and initialized ? I'm calling proxy.mesh_process.start(command) but haven't set up a mesh_process object ?

It gets set up in init:

self.mesh_obj.Proxy.mesh_process = CfdConsoleProcess(finished_hook=self.meshFinished,
                                                             stdout_hook=self.gotOutputLines,
                                                             stderr_hook=self.gotErrorLines)

The local code runs properly. It has 2 lines of code that the remote code doesn't have:

                 FreeCADGui.doCommand("cmd = CfdTools.makeRunCommand('./Allmesh', cart_mesh.meshCaseDir, source_env=False)")
                FreeCADGui.doCommand("env_vars = CfdTools.getRunEnvironment()")

I made my own command and I don't use env_vars. It runs with runFoamCommand without env_vars.

???? I'm stumped.

How do I read stdout and stderr from the CfdConsoleProcess ?

[linuxguy123]

[linuxguy123]

It's pretty quiet in here. I'm looking for comments and feedback as I go. If you see something you don't like, the sooner I hear about it the better.

[oliveroxtoby]

It's pretty quiet in here. I'm looking for comments and feedback as I go. If you see something you don't like, the sooner I hear about it the better.

Hi @linuxguy123, I'm the only other active developer at the moment as far as I know. This is a completely voluntary spare-time project for me and as such I don't have a lot of time to devote to it. I'll try to respond where I can but I'm afraid I can't do so with any degree of immediacy. Really appreciate your work though, it's looking good!!

[linuxguy123]

It's pretty quiet in here. I'm looking for comments and feedback as I go. If you see something you don't like, the sooner I hear about it the better.

Hi @linuxguy123, I'm the only other active developer at the moment as far as I know. This is a completely voluntary spare-time project for me and as such I don't have a lot of time to devote to it. I'll try to respond where I can but I'm afraid I can't do so with any degree of immediacy. Really appreciate your work though, it's looking good!!

So does that mean you'll let me merge it into CfdOF ?

What happened to the other devs ? They did a lot of work !

[oliveroxtoby]

So does that mean you'll let me merge it into CfdOF ?

That's the intention, if you could please do a pull request once it's polished up to your satisfaction.

[linuxguy123]

I'm adding the ability to have multiple remote hosts. Not just one. Because it is handy to have several hosts set up, ready to use.

CFD work involves a lot of A/B comparisons, comparing simulation 1 to simulation 2 after changing the geometry or flow speed, etc. With multiple remote hosts, I can have 2 different instances of FreeCAD open and send simulation 1 to remote host 1 and simulation 2 to remote host 2.

Likewise, a user can have an AWS host, a host at his university and a remote host in his office, all set up. If he can get off peak pricing on AWS, he can send the job there. If the university host is busy, he can run it on his office host. Multiple hosts offer lots of flexibility.

At some point I'll get and display the load percentages for the remote hosts on the pages that send jobs to the host. But for now I'll just allow different hosts to be used.

There has been talk of working with schedulers on the hosts. I'll leave that topic for another day. I'm aware of the need but for now, one step at a time.

[linuxguy123]

[linuxguy123]

Improved the layout. I hate doing layouts !

Added threads and processes.

Added appending the filename to the output directory.

[mmcker]

Looks great!

I think permanently hide the docker option. If a user has the knowledge to configure the remote machine to run openfoam, they should also be able to compile openfoam and set it up. Setting up docker to work remotely would be a fair amount of work and unfortunately I don't have time to do it.

I haven't had any user feedback on the docker implementation so it may be that noone is using it anyway.

On Mon, Feb 20, 2023 at 6:13 AM linuxguy123 @.***> wrote:

Improved the layout. I hate doing layouts !

Added threads and processes.

Added appending the filename to the output directory.

[image: New Remote Host Preferences] https://user-images.githubusercontent.com/1920754/219969913-dbd17eb1-6e11-4a8b-a380-4d02f6d0cc70.png

— Reply to this email directly, view it on GitHub https://github.com/jaheyns/CfdOF/issues/70#issuecomment-1436068785, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHZWUMI6MSOGP262YTAGGCLWYJWEFANCNFSM6AAAAAATNLY3IQ . You are receiving this because you commented.Message ID: @.***>

[linuxguy123]

Thanks. I'm still working on this, just got busy the last couple weeks.

The Preferences and Meshing page are all done for host profiles. There is still some cleanup to do and there are TODOs in the code, but they work pretty well. The Meshing page has an issue the mesh in a proxy, but works when run directly.

The OF process page works with a single host. I'll convert it to use host profiles shortly, maybe this weekend.

There isn't a lot of work left to do. I need this update for a work project so it will get done in the near future.

[luzpaz]

Maybe make a PR just in case so that it makes it in to the workbench ?

[linuxguy123]

[luzpaz]

@linuxguy123 any chance for a PR soon ?

[linuxguy123]

Let me finish up the analysis control page and then, yes.

[linuxguy123]

Status Update

Remote preferences is working with N host profiles.

I have writing the mesh case and the OF cases to the remote hosts working.

I just got remote meshing and solving to work within a proxy. This stumped me for quite a while.

I need to do some clean up on enabling controls for remote and local operations so that they make sense. And a bit of code cleanup. Then I'll do a PR.

The code was a bit straggly in places before I started. For example, there are several different ways to run a process in the code. The var naming convention is different between the meshing app and the solver app. That sort of thing.

My code is straggly too. Because I wrote this update twice - once for a single remote host and then again for N remote hosts. My code is riddled with comments and TODO items. I prefer to leave them in for now. At some point I'll go back and clean things up.

Nevertheless, the code seems pretty solid - nothing seems to hang if there is an error, for example.

This enhancement only works with Linux remote hosts running these tools directly, not within Docker. It wouldn't take too much to allow the use of Docker on the remote hosts.

I don't even want to think about trying to run this stuff on a Windows remote host. It might not be too bad if the Windows host is running WSL. The big issue with a Windows host is it having ssh and, of course, the file path differences.

I wrote some documentation back in January but I need to update it.

I'll try and get all this work done before the weekend. (March 18th.)

[linuxguy123]

I cloned CfdOF Master to start this project. Any ideas on how I should do the merge ? Should I create a branch or merge into Master ? Or just send my code to someone to look at/run before I do the merge ?

I'll be working on this all day today.

[oliveroxtoby]

Could you create a github pull request (go to 'pull requests' at the top of the page)? It will update automatically as you push new commits to your fork.

[linuxguy123]

OK. I'll push what I have at the moment so people can start looking at it/ playing with it while I continue to clean it up.

There are little things that need attention yet. No biggies. Things like stopping the solver doesn't kill the process on the server.

[linuxguy123]

I forked Master and added my changed files to it. We can remerge with jaheyns's Master once we are happy with my changes.

https://github.com/linuxguy123/CfdOF

I haven't tested if the new Master works. I'll leave that to you guys. Let me know here how you make out.

[linuxguy123]

Botched the upload. I'm using QtCreator instead of VsCode and it has an issue with its git tool. Standby for another upload.

[linuxguy123]

Uggg... QtCreator's git tool has a bug wherein it asks for a git password when it means to ask for the git username. Now when I do git push it is asking me for the password of an account that doesn't exit. And I can't figure out how to stop it from asking for that git account.

$git push mine
Password for 'https://wrong-git-user@github.com':

Any ideas how to fix this ?

[linuxguy123]

User doc thus far. How do I export this post as a markdown file ?

Background

Generating large meshes and simulating them in OpenFOAM can take considerable processing time on even the fastest desktop computer. In some cases hours or even days. Meanwhile the FreeCAD workstation machine is tied up and often not very responsive because the CPU core loads are high and the memory bandwidth is almost entirely consumed. The workstation storage device will also be accessed frequently, further slowing down the local machine.

A better way to handle large meshing and simulation tasks is to off load them to a second (remote) computer, leaving the FreeCAD workstation almost unhindered while they run. Furthermore the remote computer can be optimized to run mesh and solver loads much faster than a desktop machine might be.

The CfdOF workbench is now capable of allowing the user to use a remote computer to process meshes and run OpenFOAM solvers. It does this by using the Secure SHell (ssh) command to run the mesher and OpenFOAM on the remote computer.

A few things are required to accomplish this:

• The remote computer must be accessible to the local computer via a network connection.
• ssh must be installed on the remote computer.
• The FreeCAD user must have access to a user account on the remote computeran ssh key must be generated and stored on the remote computer such that the FreeCAD user can connect to the remote computer using ssh without using a password.
• all the relevant meshing and OpenFOAM software must be installed on the remote computer.
• the remote computer must support POSIX file system commands such as ssh, rsync, etc.

The purpose of the Remote Processing tab in CfdOF Preferences is to help the user set up and test each of the various components listed above.

Terminology

For the purpose of this discussion the FreeCAD workstation will be referred to as the local computer. The terms "workstation", "FreeCAD workstation" and local computer will be used interchangeably. Likewise the terms "server" and "remote host" refer to the non local computer that the remote processing will be done one.

Also, the terms computer and host will also be used interchangeably. Strictly speaking a computer is any computing device whereas a host is a computer connected to a network that can be reached by its IP address or hostname. In this discussion we assume that all computers are connected to a network and have an IP address and a hostname.

About ssh

"The Secure Shell Protocol (SSH) is a cryptographic [encrypted] network protocol for operating network services securely over an unsecured network." https://en.wikipedia.org/wiki/Secure_Shell

As such it will allow a FreeCAD user to run meshing and OpenFOAM applications on the remote computer from across the network, without actually being at the remote computer.The remote computer can be any computer capable of running ssh and the meshing and OpenFOAM applications that supports POSIX filesystem commands. Generally this is any computer running Linux, MacOS or Windows with WSL or MinGW installed. Hardware wise the remote computer could be almost anything - a laptop, a desktop PC, a server or a cloud computer such as an AWS instance.

However, at this time the CfdOF remote processing code has only been tested with Linux hosts.

Remote Computer Setup

In order to allow CfdOF to use the remote computer for remote processing, it must be set up as follows:

Step 1) Log into the remote computer and verify that ssh is installed and running.

On a linux computer one can do this with:

$ ps aux| grep sshd
root 1240 0.0 0.0 14024 8436 ? Ss 07:38 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups

Step 2) Log into the remote computer via ssh from the local computer.

On a linux computer one can do this from a console on the workstation with the following command:

$ssh <username>@<remote computer hostname or IP address>

Where username is the name of the user on the remote computer, not the FreeCAD workstation and IP address is the IP address of the remote computer. One can also use the hostname if the DNS server used on your network supports hostname resolution, ie dnsmasq is enabled.

For example, if the remote computer's hostname is goliath and goliath's IP address is 192.168.2.159, one can do either of the following:

$ssh me@192.168.2.159
$ssh me@goliath

If an ssh key is not set up for the user on the remote computer, ssh will ask for the password for the user account on the remote machine. This password is not to be confused with the password used on the FreeCAD workstation.

In either case the host computer should respond with a ssh shell in which you can issue commands to run on the remote computer.

me@goliath$ ls -l
total 0
drwxr-xr-x. 1 me me 20 Feb 14 2022 Desktop
drwxr-xr-x. 1 me me 0 Feb 14 2022 Documents
drwxr-xr-x. 1 me me 42 Feb 14 2022 Downloads
drwxr-xr-x. 1 me me 0 Feb 14 2022 Music
drwxr-xr-x. 1 me me 28 Feb 14 2022 openfoam
drwxr-xr-x. 1 me me 0 Feb 14 2022 Pictures
drwxr-xr-x. 1 me me 0 Feb 14 2022 Public
drwxr-xr-x. 1 me me 0 Feb 14 2022 Templates
drwxr-xr-x. 1 me me 0 Feb 14 2022 Videos

Notice that the command prompt changed to me@goliath$

To exit an ssh session, type exit:

$exit

This will bring you back so the shell on your local computer.

Step 3) Set up ssh keys.ssh

keys are used to allow the use of ssh without prompting the user for a password. Remote computing in CfdOF requires that ssh be used with keys. Each workstation will need its own key for each remote computer that it uses, if it uses more than one remote computer.

To generate an ssh key for the local computer, use the following command on a local console shell on the workstation computer: (Do not run this from within an ssh session !)

$ssh-keygen

Do not use a passphrase. TODO: explain this.

To copy the key over to the remote server, run this command on a local console on the workstation computer:

ssh-copy-id <username>@<remotecomputer>

It will prompt you for the password of your user account on the remote machine. It will then copy the ssh key you generated on the workstation over to the remote computer.

You should now be able to login to an ssh session on the remote computer without being prompted for a password.

ssh <username>@<remotecomputer>

If you are prompted for a password when you log in your ssh key is not set up. You'll need to remedy this before proceeding further.

Note: ssh requires a unique key for every host/remote computer pair. ssh identifies computers by their hostname and IP address. If you generate your ssh key using an IP address and the IP address of either the local machine or the host changes, you will have to generate a new ssh key for the new address pair. To avoid this problem, use fixed IP addresses whenever possible or use hostnames instead of IP addresses.

Likewise if you change the remote computer you use you'll have to copy the generated ssh key to the new remote computer with:

$ssh-copy-id <username>@<remotecomputer>

using the new computer name or IP address.

Once you can reliably log into an ssh session on the remote computer from the local computer using an ssh key the remote computer is ready to be used by CfdOF for remote processing.

CfdOF Remote Processing Setup

Once ssh is set up on the remote computer, one can setup the rest of remote computing from the Cfdof Remote Computing preferences page. (FreeCAD->Edit->Preferences->CfdOF->Remote Computing)

Step 1) Click "Enable Remote Computing

Step 2) Add a new host profile

A host profile is where all the details about a host are kept. The host profile is not the name of the host, but it can be. The actual name of the host is stored in the hostname field.

Step 3) Input the host name or IP address of the remote computer to be used for remote processing.

Step 4) Run "Ping Remote Host" to verify that the hostname or IP address is correct and that the local computer can reach the remote computer over the network. You should see " pinged successfully in the Output field below.

Step 5) Input the username of the user account to be used on the remote computer.

Step 6) Run "Test ssh" to verify that ssh has been set up correctly on the remote computer. You should see "ssh verified with " in the Output field below.

Step 7) Run remote host dependency checker.The output of this process will show the user what meshing and OpenFOAM applications have and haven't been installed on the remote computer and where they are if they have been installed.

Notes

As of this writing, CfdOF Remote Processing assumes that OpenFOAM will run on the remote computer from the command line without any other instructions from the workstation via an ssh session.

For example, to find out what version of foam is installed on the remote computer, run foamVersion in an ssh session from the workstation:

[me@david]$ foamVersion 
OpenFOAM-2206

To test if foam itself will run, use simpleFoam:

[me@david]$ simpleFoam -help 
Usage: simpleFoam [OPTIONS] 
Options:  
-case <dir>       Specify case directory to use (instead of cwd)  
-decomposeParDict <file>                    Use specified file for decomposePar dictionary  
-dry-run          Check case set-up only using a single time step  
-dry-run-write    Check case set-up and write only using a single time step  
-parallel         Run in parallel  
-postProcess      Execute functionObjects only  
-doc              Display documentation in browser  
-help             Display short help and exit  
-help-full        Display full help and exitSteady-state solver for incompressible, turbulent flows. 

Using: OpenFOAM-2206 (2206) - visit www.openfoam.com 
Build: _8993af73-20221106 (patch=221104) Arch:  LSB;label=32;scalar=64

OpenFOAM is a vast application. It is beyond the scope of this document to teach CfdOF users how to install and configure it. There are many good online resources for this purpose.

www.reddit.com/r/openfoam https://www.cfd-online.com/Forums/openfoam/ https://cfd.direct/openfoam/linux-guide/

One thing some users sometimes forget is that OpenFOAM must be initialized with a script prior to running it. This can be accomplished automatically by placing the following line (adjusted for the version of OpenFOAM being run) in the user's .bashrc file:

source /usr/lib/openfoam/openfoam2206/etc/bashrc

Note that this command will come into effect the next time a new bash session is started. Or you can run $source ~/.bashrc to reload it in the current session.

[linuxguy123]

I've still got a git issue. I'll push my code to my fork as soon as I get it resolved.

[linuxguy123]

I got the git permission issue worked out. But now I have merge issues:

$ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        modified:   CfdOF/CfdPreferencePage.py
        new file:   CfdOF/CfdRemotePreferencePage.py
        modified:   CfdOF/CfdTools.py
        modified:   CfdOF/Mesh/CfdMesh.py
        modified:   CfdOF/Mesh/CfdMeshTools.py
        modified:   CfdOF/Mesh/TaskPanelCfdMesh.py
        modified:   CfdOF/Solve/CfdCaseWriterFoam.py
        modified:   CfdOF/Solve/TaskPanelCfdSolverControl.py
        modified:   Gui/CfdPreferencePage.ui
        new file:   Gui/CfdRemotePreferencePage.ui
        modified:   Gui/TaskPanelCfdMesh.ui
        modified:   Gui/TaskPanelCfdSolverControl.ui
        modified:   InitGui.py

$ git push mine
To https://github.com/linuxguy123/CfdOF
 ! [rejected]        master -> master (non-fast-forward)
error: failed to push some refs to 'https://github.com/linuxguy123/CfdOF'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

I'm working on it.

[linuxguy123]

It was a mistake to fork before trying to merge. I should have created an empty repo and then pushed into it.

But it looks like the conflict is pretty small and all in one file.

$ git diff
diff --cc CfdOF/Mesh/TaskPanelCfdMesh.py
index 75afc58,4acfde2..0000000
--- a/CfdOF/Mesh/TaskPanelCfdMesh.py
+++ b/CfdOF/Mesh/TaskPanelCfdMesh.py
@@@ -470,22 -250,18 +470,30 @@@ class TaskPanelCfdMesh
              FreeCADGui.doCommand("cart_mesh.error = False")
              FreeCADGui.doCommand("cmd = CfdTools.makeRunCommand('checkMesh -meshQuality', cart_mesh.meshCaseDir)")
              FreeCADGui.doCommand("env_vars = CfdTools.getRunEnvironment()")
+             self.check_mesh_error = False
              FreeCADGui.doCommand("proxy.running_from_macro = True")
              self.mesh_obj.Proxy.running_from_macro = False
+             self.mesh_obj.Proxy.check_mesh_process = CfdConsoleProcess(
+                 stdout_hook=self.gotOutputLines, stderr_hook=self.gotErrorLines)
              FreeCADGui.doCommand("if proxy.running_from_macro:\n" +
-                                  "  mesh_process = CfdConsoleProcess.CfdConsoleProcess()\n" +
-                                  "  mesh_process.start(cmd, env_vars=env_vars)\n" +
-                                  "  mesh_process.waitForFinished()\n" +
+                                  "  proxy.check_mesh_process = CfdConsoleProcess()\n" +
+                                  "  proxy.check_mesh_process.start(cmd, env_vars=env_vars)\n" +
+                                  "  proxy.check_mesh_process.waitForFinished()\n" +
                                   "else:\n" +
++<<<<<<< HEAD
 +                                 "  proxy.mesh_process.start(cmd, env_vars=env_vars)")
 +            if self.mesh_obj.Proxy.mesh_process.waitForStarted():
 +                self.form.pb_check_mesh.setEnabled(False)   # Prevent user running a second instance
 +                self.form.pb_run_mesh.setEnabled(False)
 +                self.form.pb_write_mesh.setEnabled(False)
 +                #self.form.pb_write_remote_mesh.setEnabled(False)
 +                self.form.pb_stop_mesh.setEnabled(False)
 +                self.form.pb_paraview.setEnabled(False)
 +                self.form.pb_load_mesh.setEnabled(False)
++=======
+                                  "  proxy.check_mesh_process.start(cmd, env_vars=env_vars)")
+             if self.mesh_obj.Proxy.check_mesh_process.waitForStarted():
++>>>>>>> 2b67db0aab52d0005a8253944b109ddded72781b
                  self.consoleMessage("Mesh check started ...")
              else:
                  self.consoleMessage("Error starting mesh check process", 'Error')
@@@ -531,11 -310,8 +545,16 @@@
              FreeCADGui.doCommand("from CfdOF.Mesh import CfdMeshTools")
              FreeCADGui.doCommand("from CfdOF import CfdTools")
              FreeCADGui.doCommand("from CfdOF import CfdConsoleProcess")
++<<<<<<< HEAD
 +            FreeCADGui.doCommand("from FreeCAD import ParamGet")
 +            
 +            FreeCADGui.doCommand("cart_mesh = " +
 +                                 "CfdMeshTools.CfdMeshTools(FreeCAD.ActiveDocument." + self.mesh_obj.Name + ")")
 +            
++=======
+             FreeCADGui.doCommand("cart_mesh = "
+                                  "    CfdMeshTools.CfdMeshTools(FreeCAD.ActiveDocument." + self.mesh_obj.Name + ")")
++>>>>>>> 2b67db0aab52d0005a8253944b109ddded72781b
              FreeCADGui.doCommand("proxy = FreeCAD.ActiveDocument." + self.mesh_obj.Name + ".Proxy")
              FreeCADGui.doCommand("proxy.cart_mesh = cart_mesh")
              FreeCADGui.doCommand("cart_mesh.error = False")

I'll finish this tomorrow.

[linuxguy123]

I fixed the conflict and the push worked. My repo now has all my changes and Oliver's too. https://github.com/linuxguy123/CfdOF

I need to check that this was Oliver's intended change in TaskPanelCfdMesh.py:

FreeCADGui.doCommand("if proxy.running_from_macro:\n" +
                                 "  mesh_process = CfdConsoleProcess.CfdConsoleProcess()\n" +
                                 "  mesh_process.start(cmd, env_vars=env_vars)\n" +
                                 "  mesh_process.waitForFinished()\n" +
                                 "  proxy.check_mesh_process = CfdConsoleProcess()\n" +
                                 "  proxy.check_mesh_process.start(cmd, env_vars=env_vars)\n" +
                                 "  proxy.check_mesh_process.waitForFinished()\n" +
                                 "else:\n" +
                                 "  proxy.check_mesh_process.start(cmd, env_vars=env_vars)")

[linuxguy123]

Here is the file that I use for testing. I got it from a post in FreeCAD.org.

naca2412_BH.zip

[linuxguy123]

Right now the case files are moved over to the remote host and run in the remote output path. I have not implemented adding the filename to the output path and the results are not copied back to the workstation. I intend to change this in the near future.

I am contemplating giving the user the option to copy the results back and also the option of deleting the results on the server as shown in the GUI below.

If the results files are copied back to the workstation, the workstation will operate exactly as if the results were run locally. ie Checkmesh, Paraview, View mesh, etc. will work locally. Not everyone will want to do this. Some users will work with the results on the server.

Of those that do copy the results back, some will want to leave the results on the server for others to use in a shared work environment. Thus I've set up the Remote Output Path options the way I have.

Thoughts ?

This topic touches on another topic that will need addressing sooner or later. My strategy until now has been to touch as little of the existing code as possible and implement remote processing by adding the necessary code. To this end, I have not made any additions to the mesh object or the solver object. Everything needed to run a remote process has been passed into the routines that need it via global vars. Which is obviously messy and not good programming practice.

At some point I'll go back and add the variables that are needed for remote processing to the mesh and solver objects. It would make implementing things like copyBack and deleteOnServer a lot simpler. "At some point" = when I get some feedback on the addition I've done.

If you look at the TODOs in the code, there are lots of things that could be cleaned up. This isn't the only thing that could be addressed in the code base.

BTW, I'm doing this work because I need changes like this for a project I'm working on. It took a lot of effort to add remote processing to CfdOF. But that effort will be returned and a lot more going forward when I/we are using it.

[linuxguy123]

There is a thread on FreeCAD.org to discuss this release. https://forum.freecad.org/viewtopic.php?p=668412#p668412

[linuxguy123]

Just updated my repo with some code clean up and with copy_back working for meshing.

copy_back for meshing means we automatically have a local copy of the mesh case in which to ParaView, Check, Load Surface, etc.

I have not implemented add_filename_to_output yet.

I will now add copy_back to the solver.

The impetus for doing all this is that one can iterate a design with file1.FCstd, file2.FCStd, file3.FCstd and the output from each will go to /outputdir/file1, outputdir/file2, outputdir/file3 on the workstation automatically, without over writing the results of a previous design iteration. And if we open file1.FCstd in FreeCAD and click Paraview the results from file1 solve will open.

All this will happen automatically by copying the results back to the workstation and by adding the filename to the output dir. It should be a seamless experience whether things are processed on the workstation or on a remote host.

I'm doing this because I have several hundred CFDs to run, ASAP. I don't want to be wasting my time manually moving result files around and renaming directories. I want that to happen automatically so that I can focus on the design part of the work.

In a future update I'll set it up so that those results can be left on the server and opened there from the workstation. The impetus for this is that any user could access file1.FCstd located on a network server and have access to the mesh and solve results on a different server.

But baby steps first.

[linuxguy123]

I found a problem whereby process.finished is not firing even though the remote process is completing properly with zero errors. I'm looking into it.

Here is how a solve process ends on a remote host:

SIMPLE solution converged in 2107 iterations

18:34:05  forces ReportingFunction write:
    Sum of forces
        Total    : (0.82629446 29.246383 -3.2031941e-18)
        Pressure : (0.47268552 29.2442 -1.5489154e-18)
        Viscous  : (0.35360895 0.0021826911 -1.6542787e-18)
    Sum of moments
        Total    : (-2.924612 0.082628704 13.221892)
        Pressure : (-2.9243937 0.047268127 13.234571)
        Viscous  : (-0.00021826715 0.035360577 -0.012679146)
    writing force and moment files.

18:34:05  forceCoeffs ReportingFunction001 write:
    Coefficient Total   Pressure    Viscous Internal
    Cd: 0.01806582  0.010334635 0.0077311851    0
    Cd(f):  -0.036640459    -0.040502643    0.0038621839    0
    Cd(r):  0.054706279 0.050837278 0.0038690012    0
    Cl: 0.63943291  0.63938519  4.7721613e-05   0
    Cl(f):  0.1132315   0.11300963  0.00022186975   0
    Cl(r):  0.52620141  0.52637556  -0.00017414813  0
    CmPitch:    -0.20648495 -0.20668296 0.00019800894   0
    CmRoll: -0.045673369    -0.045669961    -3.408656e-06   0
    CmYaw:  0.0012904041    0.00073818156   0.00055222255   0
    Cs: 7.0033542e-20   3.3864958e-20   3.6168584e-20   0
    Cs(f):  0.0012904041    0.00073818156   0.00055222255   0
    Cs(r):  -0.0012904041   -0.00073818156  -0.00055222255  0

    writing force and moment coefficient files.

End

18:34:05  Finalising parallel run
18:34:05  

The finished code in CfdConsoleProcess.py is not even getting called:

def finished(self, exit_code):
        print("Process has finished")
        if self.finishedHook:
            print("finished hook was called with exit code:" + str(exit_code))
            self.finishedHook(exit_code)
        else:
            print("Error: finished wasn't hooked to a handler")