Dataloader for Tau simulation files

[AndreWeiner]

• dataloader to load simulation results produced by the Tau code; each snapshot has to be saved in a single file

[AndreWeiner]

Hi @MehdiMak96,

TAU writes netCDF4 files. You can install the netCDF4 library with pip:

pip3 install netCDF4

To create a Dataloader, the following information must be extracted from the test files you will receive:

• [x] find all relevant simulation files (solution and mesh) in a location provided by the user (tip: use glob)
• [x] extract the time value from each file name; at the end, there should be a list with the time values as strings, e.g., ["1.0e-3", "2.0e-3", ...]
• [x] the available fields as a list of strings, e.g., ["x_velocity", "density", "pressure", ...]; this list should not contain the vertices of cell volumes
• [x] the vertices as a PyTorch tensor of shape n_points x 3, where each column holds the x/y/z values; you can convert the arrays via NumPy; netCDF4 can convert to NumPy and PyTorch can create tensors from NumPy arrays
• [x] the cell volumes as PyTorch tensor
• [x] the values of any field as PyTorch tensor

Here is some starter code to explore a TAU simulation file:

from netCDF4 import Dataset

file_path = "/path/to/tau_backward_facing_step/"
grid_path = file_path + "PW_DES-HybQuadTRex-v2_yp-50_s1.15_ny67.grd"
data_path = file_path + "sol.pval.unsteady_i=14790_t=2.9580000000e-02"

grid = Dataset(grid_path)
data = Dataset(data_path)
print("Variables in grid file:")
for key in grid.variables.keys():
    print(key, grid.variables[key].shape, grid.variables[key].dtype)
print("-"*80)

print("Variables in data file:")
for key in data.variables.keys():
    print(key, data[key].shape)
print("-"*80)

You can solve the task outside of the flowTorch repository and post your solution here.

Best, Andre

[MehdiMak96]

Hallo Andre,

I'm at the point now where I have to create an array via Numpy but I'm a bit confused cause all the values are the same (1119348) ,am I doing something wrong? Or should I continue like that?

Best, Mehdi

[AndreWeiner]

The value in parenthesis indicates the size of the dataset. So, for a given mesh, it makes sense that all fields have the same size. To access the field values, you do something like

field = dataset["field_name"][:]

Best, Andre

[MehdiMak96]

Hello Andre,

Here is my final solution :

import glob
import os
import torch as pt
from netCDF4 import Dataset

Unwnted_list = ['x', 'y', 'z', 'volume', 'global_id']
coords = ["x", "y", "z"]

print('Named explicitly:')
for name in glob.glob('/home/mak/Hiwi/Dataloader/test_data_tau/*'):
    print(name)

filepath = glob.glob('/home/mak/Hiwi/Dataloader/test_data_tau/*sol*')

for x in filepath:
    print((os.path.basename(x)).split('t=')[-1])

file_path = "/home/mak/Hiwi/Dataloader/test_data_tau/"
grid_path = file_path + "PW_DES-HybQuadTRex-v2_yp-50_s1.15_ny67.grd"
data_path = file_path + "sol.pval.unsteady_i=14790_t=2.9580000000e-02"

grid = Dataset(grid_path)
data = Dataset(data_path)
dict_keys = list(data.variables.keys())
print("Variables in grid file:")
for key in grid.variables.keys():
    print(key, grid.variables[key].shape, grid.variables[key].dtype)
print("-"*80)
print(data.variables.keys())
print("Variables in data file:")
for key in data.variables.keys():
    print(key, data[key].shape)

print("-"*80)

for s in Unwnted_list:
    if any(xs in s for xs in coords):
        vertices = pt.cat([pt.from_numpy(data.variables[s][:]).view(
            1119348, 1) for s in coords], dim=-1)
    else:
        print(s, "=", (pt.from_numpy(data.variables[s][:])))
print(vertices, vertices.shape)

print("-"*80)

new_list = []
for x in dict_keys:
    if x not in Unwnted_list:
        new_list.append(x)
print(new_list)
for x in new_list:
    print(x, '=', (pt.from_numpy(data.variables[x][:])))

print("-"*80)

I hope you don't find any mistakes in it :)

[AndreWeiner]

Hi Mehdi, thanks for your efforts! Best, Andre