Data storage and handling slice files

[chraibi]

In GitLab by @vanweyenberge1 on Dec 30, 2017, 16:00

We noticed that handling .csv files is very time consuming and might not be the best way for doing data handling in this case. I have working with large data slice files before and what we used was an array in a .mat or .npy (or npz) file. This file would hold multiple dimensions:

• Dim1: type (Soot extinction, CO-concentration)
• Dim2: Timestep
• Dim3: X-dim
• Dim4: Y-dim
• Dim5: Z-dim

What do you think?

[chraibi]

In GitLab by @chraibi on Dec 30, 2017, 16:05

Great idea Bart!

How can we read these files with jpscore (C++)?

[chraibi]

In GitLab by @vanweyenberge1 on Dec 30, 2017, 16:19

Several methods exist to read .npy files or this or this.

for a .mat file

[chraibi]

In GitLab by @chraibi on Dec 30, 2017, 16:28

I would not use .mat file, since we use Python already.

The first link is interesting. I found this little library interesting too.

[chraibi]

In GitLab by @chraibi on Jan 10, 2018, 08:39

We use npz files. See !3

[chraibi]

In GitLab by @chraibi on Jan 10, 2018, 08:39

closed