Need Simulation Output in Text Format for Postprocessing

[himakarganti]

I will be postprocessing simulation data and I need text format for it. Does SU2 have the option to report the simulation data in ASCII text format?

[clarkpede]

You've got two options:

1. Use the cfg option WRT_BINARY_RESTART= NO to dump the restart file(s) in human-readable ASCII.
2. Use the cfg option OUTPUT_FORMAT= TECPLOT or OUTPUT_FORMAT=PARAVIEW with SU2_SOL to export the solution to an ASCII-formatted tecplot or paraview file. If you're not familiar with SU2_SOL, then check out the documentation.

You can find those cfg options in config_template.cfg in the root SU2 source code directory.

In the end, using the tecplot or paraview option may be your best bet. Trying to manually parse the text files smells like an XY Problem. Tecplot and Paraview both have scripting capabilities that make post-processing very efficient.

Does that answer your question? Do you have any follow-up concerns?

[himakarganti]

Thank you for the help. I will be using the flow.dat file directly. Since it is in tecplot (ASCII) format - it appends the cell numbers at the end. I will be doing a parameter sweep and this adds additional steps when extracting the data in the format I need it for.

[clarkpede]

So does that answer your question? Do you have any follow-up concerns?

[himakarganti]

I am using the binary of SU2_CFD on Mac Os X for a single core run. There are no restart files for SU2_SOL to work with. IS there a way to simply export in CSV format? config_template.cfg is missing in the folders.

[clarkpede]

Just to verify, you're saying that SU2_CFD doesn't output a restart_flow.dat file?

[himakarganti]

It does output the file, Paraview & Tecplot formats (binary & ASCII) both work. However, I need the raw data as I will be subjecting it to a Machine Learning Algorithm in Python. Manually deleting the lines for cell numbers in Tecplot format is an additional time consuming step, that hinders full automation. Hence a simple file as flow.csv (CSV format) may help me run it more efficiently.

1. Ganti, Himakar & Khare, Prashant. (2018). Spatio-Temporal Prediction of Gaseous and Liquid Spray Fields using Machine Learning. 10.2514/6.2018-4760.

2. Ganti, Himakar & Kamin, Manu & Khare, Prashant. (2019). Design Space Exploration for Vaporizing Liquid Jet in Air Crossflow using Machine Learning. 10.2514/6.2019-2211.

[EduardoMolina]

Hi Himakar,

If you are using python, you don't need to manually delete the connectivity table from the ASCII Tecplot as you can use numpy genfromtxt.(https://docs.scipy.org/doc/numpy/reference/generated/numpy.genfromtxt.html).

Another option is to use the following restart_flow binary reader/writer that I wrote in Python.

----------

def read_restart_bin(filename):

# Not efficient way
infile = open(filename, 'rb')
data = infile.read()
infile.close()

print "Size of the file: ", len(data)

# The first is a magic number that we can use to check for binary files (it is the hex
# representation for "SU2").
if (struct.unpack('i',data[:4])[0] != 535532):
    print "Magic number 535532 not found in the solution file %s" %filename
    sys.exit()
    #try:
    #   data_file = read_restart_ascii(filename)
    #   return data_file
    #except:
        # Exit
    #   sys.exit()

# The second two values are number of variables and number of points (DoFs). 
nvar = struct.unpack('i',data[4:8])[0]
ndof = struct.unpack('i',data[8:12])[0]

# Read the variable names of the file. Note that we are adopting a
# fixed length of 33 for the string length to match with CGNS.
variables_names = []
for i in range(nvar):
    aux = (struct.unpack('33s',data[20+(i)*33:20+(i+1)*33])[0])
    for j in range(len(aux)):
        if aux[j] == "\x00":
            break
    variables_names.append(aux[:j])

# Read data in one shoot
start = 20 + nvar*33
end = start+nvar*ndof*8
array = np.asfarray(struct.unpack('%dd'%(nvar*ndof),data[start:end]))
array = array.reshape(ndof,nvar)

# The last two values are for metadata:  one int for ExtIter and 8 su2doubles.
# Metadata: 1 int for ExtIter and 8 doubles
#ncount = len(data) - nvar*ndof*8 - 4 - 64
ExtIter  = struct.unpack('i',data[end:end+4])[0]
metadata = struct.unpack('8d',data[end+4:end+4+8*8])

# Create dictionary
data_file = {'names':variables_names,'data':array, 'ExtIter':ExtIter, 'MetaData':metadata}
return data_file

def read_restart_ascii(filename): """

"""
infile = open(filename, 'r')
header = infile.readline()
variables_names = header.replace("\""," ").replace("\t", " ").strip().split()[1:]
infile.close()

array = np.loadtxt(filename, skiprows = 1)

array = array[:,1:]

# Create dictionary
data_file = {'names':variables_names,'data':array, 'ExtIter':0, 'MetaData':[]}  
return data_file

def write_restart_binary(data_file,filename="solution_flow_bin.dat"): """

"""
fout = open(filename,'wb')

# The first is a magic number that we can use to check for binary files (it is the hex
# representation for "SU2").
fout.write(struct.pack('i', 535532))

# The second two values are number of variables and number of points (DoFs).
nvar = data_file['data'].shape[1]
ndof = data_file['data'].shape[0]
fout.write(struct.pack('i', nvar))
fout.write(struct.pack('i', ndof))
fout.write(struct.pack('i', 1))
fout.write(struct.pack('i', 8))

# Write the variable names of the file. Note that we are adopting a
# fixed length of 33 for the string length to match with CGNS.
for variable_name in data_file['names']:
    fout.write(struct.pack('33s', variable_name))

# Write the entire data in one shoot
fout.write(struct.pack('%dd'%(nvar*ndof), *data_file['data'].flatten()))

# Write ExtIter and Metadata
fout.write(struct.pack('i',data_file['ExtIter']))
for i in range(8):
    fout.write(struct.pack('d', data_file['MetaData'][i]))
fout.close()

return None

----------

[clarkpede]

Run SU2_CFD with WRT_BINARY_RESTART= NO. That will give you an ASCII restart_flow.dat. It will look something like this:

"PointID"   "x" "y" "z" "Density"   "X-Momentum"    "Y-Momentum"    "Z-Momentum"    "Energy"    "Pressure"  "Temperature"   "C<sub>p</sub>" "Mach"
0   6.931794337794450e-01   1.199410270705132e+00   3.552713678800501e-15   9.247274759558524e-01   6.998395924205978e-01   6.744390123118781e-01   2.033333591408758e-01   2.825647424000119e+00   9.170095708259952e-01   9.784295110379145e-01   -1.930126447908861e-01  9.113348070779481e-01   
1   6.906841300000011e-01   1.196300000000000e+00   0.000000000000000e+00   1.118921803269151e+00   5.021166385546235e-01   7.896289876522433e-01   2.161057381347509e-01   3.352487832836354e+00   1.176133434271568e+00   1.037113520398247e+00   3.252350821002600e-01   7.075425651590469e-01   
2   6.924294614497271e-01   1.202091611452504e+00   1.468890958566504e-03   9.758692057411605e-01   8.423040306657703e-01   5.135454779704676e-01   2.297213358745695e-01   2.971366201327637e+00   9.782771280113181e-01   9.890989089014984e-01   -7.047753042024051e-02  8.761491078578573e-01   

This answers the original question, "How do I get SU2 to output raw ASCII data?"

You can also use the python scripting capabilities of Tecplot or Paraview (technically, VTK). You can find documentation for Tecplot's python API here and for VTK's python API here. That will allow you to use the Paraview or Tecplot files directly in python. You would use the paraview or tecplot libraries to load the data into python, then post-process it however you want. There's no need for "manually deleting the lines for cell numbers."

[himakarganti]

I am sorry, but neither of those 2 is helpful to me. I can obtain ASCII data with Tecplot format, and with the restart file. However, the end of the restart_flow file looks something like this, and it may change with conditions for simulation:

3748    2.000000000000002e-02   0.000000000000000e+00   1.161205517119654e+00   8.075360800285049e+02   -4.616675578067918e-14  5.307920350591170e+05   9.999999999999996e+04   2.999999999999997e+02   2.002826699082778e+00   -1.559133774467877e-16  
3749    0.000000000000000e+00   0.000000000000000e+00   1.161205517119653e+00   8.075360800285049e+02   -1.451560808848285e-13  5.307920350591170e+05   9.999999999999988e+04   2.999999999999997e+02   2.002826699082779e+00   -4.157690065247673e-16  
EXT_ITER= 195
AOA= 0.000000000000000e+00
SIDESLIP_ANGLE= 0.000000000000000e+00
INITIAL_BCTHRUST= 4.000000000000000e+03
DCD_DCL_VALUE= 0.000000000000000e+00
DCMX_DCL_VALUE= 0.000000000000000e+00
DCMY_DCL_VALUE= 0.000000000000000e+00
DCMZ_DCL_VALUE= 0.000000000000000e+00

It is the same case for the flow.dat file with Tecplot format. When reading this data file back with Python, the last lines will cause errors (with skip_footer option in numpy.genfromtxt) if they are not consistent in number for the parameter sweep. I only need the array w/o the tail data and wanted to avoid additional scripting through Tecplot or Paraview for a faster turnaround for speedup. Hence I was requesting if we can have a CSV format, with just the raw grid data and headings.

[economon]

@himakarganti : the short answer is that we can not dump straight to CSV at the moment.

However, @talbring and I were just discussing potentially switching our ASCII restart to a proper CSV format with the feature_input_output PR in #724 (and moving the metadata to a separate file). So, this may by coming soon, but not yet.

[himakarganti]

Thanks for the clarification @economon - appreciate the help. I had another request - if there can be an option to dump straight to text (*.dat) w/o the cell numbers (shown below) (like csv format but tab separated) that would be helpful too in certain cases.

2.000000e-02    0.000000e+00    1.161206e+00    8.075361e+02    -4.616676e-14   5.307920e+05    1.000000e+05    3.000000e+02    2.002827e+00    -1.559134e-16   
0.000000e+00    0.000000e+00    1.161206e+00    8.075361e+02    -1.451561e-13   5.307920e+05    1.000000e+05    3.000000e+02    2.002827e+00    -4.157690e-16   
1   2   77  76
2   3   78  77
3   4   79  78
4   5   80  79
5   6   81  80
6   7   82  81
7   8   83  82
8   9   84  83
9   10  85  84
10  11  86  85
11  12  87  86

[clarkpede]

If you must use restart_flow.dat, then look at np.genfromtxt and the invalid_raise option. Here's a simple example:

import numpy as np
import matplotlib.pyplot as plt

data = np.genfromtxt("solution_adj_combo.dat", names=True, invalid_raise=False)

plt.tricontourf(data["x"], data["y"], data["Adjoint_Density"])
plt.show()

[monika1387]

@clarkpede Can you please help how I can dump tecplot files? I am getting this error when it is trying to dump tecplot - YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Bus error (signal 7). Earlier I was using PARAVIEW and never see any issues in dumping them (both with FV and FEM solvers). But now files are so huge or some issues with Paraview to open dumped files from FEM solver for LES (DG scheme). So trying to switch back to Tecplot. But Can you suggest if I am missing something here in FEM solvers for dumping these files? Paraview should work or we have issues with FEM solvers with Paraview pr tecplot dumping files ?

[clarkpede]

@monika1387 Can you describe what you mean by "dumping" the files? Are you talking about exporting the solution as a tecplot file? Or are you talking about post-processing the tecplot files themselves?

When does the error occur? Can you create a minimal working example?

[monika1387]

@clarkepede yes I mean exporting the solution as tecplot because paraview is not showing the result in visualisation when trying to open. Yes post processing but before post processing it needs to export solution in Tecplot format which is not happening

Monika Chauhan Sent from my iPhone

On 16-Mar-2020, at 4:02 PM, Clark Pederson notifications@github.com wrote:

@monika1387 Can you describe what you mean by "dumping" the files? Are you talking about exporting the solution as a tecplot file? Or are you talking about post-processing the tecplot files themselves? When does the error occur? Can you create a minimal working example?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[clarkpede]

@monika1387 I'm sorry, I still don't understand. Do you have a minimal working example, where you can reproduce the problem you're having?

Where does the error occur? In SU2_CFD? In SU2_SOL? In Paraview? In Tecplot? What's the specific error message? What's the context?

[monika1387]

@clarkepede Sure, I will try to make an example to show . So my problem is CD nozzle and I am running FEM-LES solver and exported data in .csv and PARAVIEW type. Everything seems good its just I was unable to visualize exported .vtk files (open but does not show me the final result- stuck in just opening it - I am not sure if its paraview issue or its 4 GB file which causing this error). So I tried to export my file using Tecplot type option but unable to do so and getting this Bus error which I never encountered earlier in SU2 ever (what is that error - YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Bus error (signal 7)). So trying to dump my files in anyway to visualize them, since its FEM-LES solver with DG scheme so files are pretty huge. Still let me know if I need to send you an example. Attached .cfg file for reference.

On Mon, Mar 16, 2020 at 6:13 PM Clark Pederson notifications@github.com wrote:

@monika1387 https://github.com/monika1387 I'm sorry, I still don't understand. Do you have a minimal working example, where you can reproduce the problem you're having?

Where does the error occur? In SU2_CFD? In SU2_SOL? In Paraview? In Tecplot? What's the specific error message? What's the context?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/su2code/SU2/issues/787#issuecomment-599781328, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALJ3OXFU367WBC2BSOGLCILRH2P7VANCNFSM4IT7MM2A .

-- Thank you,

Monika Chauhan

Graduate Research Assistant, Doctoral Program

Aerospace and Ocean Engineering Dept, Virginia Tech,Blacksburg, VA 24061 Cell# 540-998-5012

[monika1387]

Also since this issue is getting resolved, is there anyway or option I can reduced number of variables in exported results? I tried using VOLUME_OUTPUT= (COORDINATES, PRIMITIVE) deleted SOLUTIONS so that I can able to reduce the size of file from redundant parameters which i don't need as of now. But seems like that does not work either and exported .vtk with same size. It supposed to work like that if I understand correctly.

On Mon, Mar 16, 2020 at 6:53 PM Monika Chauhan monika1387@vt.edu wrote:

@clarkepede Sure, I will try to make an example to show . So my problem is CD nozzle and I am running FEM-LES solver and exported data in .csv and PARAVIEW type. Everything seems good its just I was unable to visualize exported .vtk files (open but does not show me the final result- stuck in just opening it - I am not sure if its paraview issue or its 4 GB file which causing this error). So I tried to export my file using Tecplot type option but unable to do so and getting this Bus error which I never encountered earlier in SU2 ever (what is that error - YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Bus error (signal 7)). So trying to dump my files in anyway to visualize them, since its FEM-LES solver with DG scheme so files are pretty huge. Still let me know if I need to send you an example. Attached .cfg file for reference.

On Mon, Mar 16, 2020 at 6:13 PM Clark Pederson notifications@github.com wrote:

@monika1387 https://github.com/monika1387 I'm sorry, I still don't understand. Do you have a minimal working example, where you can reproduce the problem you're having?

Where does the error occur? In SU2_CFD? In SU2_SOL? In Paraview? In Tecplot? What's the specific error message? What's the context?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/su2code/SU2/issues/787#issuecomment-599781328, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALJ3OXFU367WBC2BSOGLCILRH2P7VANCNFSM4IT7MM2A .

-- Thank you,

Monika Chauhan

Graduate Research Assistant, Doctoral Program

Aerospace and Ocean Engineering Dept, Virginia Tech,Blacksburg, VA 24061 Cell# 540-998-5012

-- Thank you,

Monika Chauhan

Graduate Research Assistant, Doctoral Program

Aerospace and Ocean Engineering Dept, Virginia Tech,Blacksburg, VA 24061 Cell# 540-998-5012

[GomerOfDoom]

I can also confirm that there is an issue with the binary Paraview output for large files. The code appears to write out the .vtu file just fine, but Paraview throws errors when attempting to open it. This does not occur for ASCII format Paraview files or smaller binary Paraview files. The case I am working on consists of a 180 million cell mesh.

Also, @monika1387 , please note that the DG-FEM LES capability is not currently fully validated, and may be unstable in certain circumstances. Use this at your own risk. We are working on fixes for the DG solver, but they are still a ways away from being finished.

[monika1387]

@ GomerOfDoom https://github.com/GomerOfDoom

Then it makes sense why I am having this trouble. Thank you for letting me know. This helps a lot. I will try to figure this one out atleast for my case, what I can do. Appreciate your response!

On Mon, Mar 16, 2020 at 7:36 PM Paul notifications@github.com wrote:

I can also confirm that there is an issue with the binary Paraview output for large files. The code appears to write out the .vtu file just fine, but Paraview throws errors when attempting to open it. This does not occur for ASCII format Paraview files or smaller binary Paraview files. The case I am working on consists of a 180 million cell mesh.

Also, @monika1387 https://github.com/monika1387 , please note that the DG-FEM LES capability is not currently fully validated, and may be unstable in certain circumstances. Use this at your own risk. We are working on fixes for the DG solver, but they are still a ways away from being finished.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/su2code/SU2/issues/787#issuecomment-599805153, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALJ3OXCKD44SRSZPDWKSDM3RH2ZZFANCNFSM4IT7MM2A .

-- Thank you,

Monika Chauhan

Graduate Research Assistant, Doctoral Program

Aerospace and Ocean Engineering Dept, Virginia Tech,Blacksburg, VA 24061 Cell# 540-998-5012

[talbring]

@monika1387 Can you please open a new issue with information on the platform that you are using and whether you compiled from scratch or use the precompiled binaries? Thanks!

[monika1387]

Yes I will open the new issue. I am doing from scratch as well as from restart solution(ASCII) but nothing is working to visualise this DG FEM LES solver exported vtk file. I will keep working to see what is the issue in the code.

On 17-Mar-2020, at 7:21 AM, Tim Albring notifications@github.com wrote:

 @monika1387 Can you please open a new issue with information on the platform that you are using and whether you compiled from scratch or use the precompiled binaries? Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[Karthik-ksd]

You've got two options:

1. Use the cfg option WRT_BINARY_RESTART= NO to dump the restart file(s) in human-readable ASCII.
2. Use the cfg option OUTPUT_FORMAT= TECPLOT or OUTPUT_FORMAT=PARAVIEW with SU2_SOL to export the solution to an ASCII-formatted tecplot or paraview file. If you're not familiar with SU2_SOL, then check out the documentation.

You can find those cfg options in config_template.cfg in the root SU2 source code directory.

In the end, using the tecplot or paraview option may be your best bet. Trying to manually parse the text files smells like an XY Problem. Tecplot and Paraview both have scripting capabilities that make post-processing very efficient.

Does that answer your question? Do you have any follow-up concerns?

Hello @clarkpede can you please tell me how to convert the output flile format from vtu to vtk I am facing a lots of trouble to open the file in Paraview but the output format is coming vtu everytime Can you please explain how to solve this problem And whenever i tries to change the output format by editting in notepad it tells an error that check option in SU2 config_template.cfg Please Help

[monika1387]

I found the solution. It's just a VTU file that was way too huge to open in the system and crashes every time. Thank you for the help!

On Mon, Jun 29, 2020 at 1:17 PM Karthik-ksd notifications@github.com wrote:

You've got two options:

1. Use the cfg option WRT_BINARY_RESTART= NO to dump the restart file(s) in human-readable ASCII.
2. Use the cfg option OUTPUT_FORMAT= TECPLOT or OUTPUT_FORMAT=PARAVIEW with SU2_SOL to export the solution to an ASCII-formatted tecplot or paraview file. If you're not familiar with SU2_SOL, then check out the documentation https://su2code.github.io/docs/Post-processing/.

You can find those cfg options in config_template.cfg in the root SU2 source code directory.

In the end, using the tecplot or paraview option may be your best bet. Trying to manually parse the text files smells like an XY Problem https://en.wikipedia.org/wiki/XY_problem. Tecplot and Paraview both have scripting capabilities that make post-processing very efficient.

Does that answer your question? Do you have any follow-up concerns?

Hello @clarkpede https://github.com/clarkpede can you please tell me how to convert the output flile format from vtu to vtk I am facing a lots of trouble to open the file in Paraview but the output format is coming vtu everytime Can you please explain how to solve this problem And whenever i tries to change the output format by editting in notepad it tells an error that check option in SU2 config_template.cfg Please Help

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/su2code/SU2/issues/787#issuecomment-651252372, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALJ3OXHBNWF5UFMV7GZ4WDTRZDEC3ANCNFSM4IT7MM2A .

-- Thank you,

Monika Chauhan

Graduate Research Assistant, Doctoral Program

Aerospace and Ocean Engineering Dept, Virginia Tech,Blacksburg, VA 24061 Cell# 540-998-5012

[clarkpede]

@Karthik-ksd You said:

And whenever i tries to change the output format by editting in notepad it tells an error that check option in SU2 config_template.cfg

What cfg setting are you changing? What are you changing it to? Can you supply the (non-working) cfg file? What is the error you're seeing?

[Karthik-ksd]

@Karthik-ksd You said:

And whenever i tries to change the output format by editting in notepad it tells an error that check option in SU2 config_template.cfg

What cfg setting are you changing? What are you changing it to? Can you supply the (non-working) cfg file? What is the error you're seeing? Now I am able to open it in paraview after writing one extra code in cfg file in notepad after TABULAR_FORMAT = CSV
OUTPUT_FILES = PARAVIEW_ASCII due to which running the code in command prompt I am able to get vtk format instead of vtu format. Thank you for responding . If this is not the solution please respond. As in my case i tried every examples in tutorial everything is working fine.