Add Lower-precision, compressed volumetric output option

[rjokonski]

Problem/Opportunity

At Thunderhead, we currently use volumetric data from FDS (e.g. s3d, 3d slices, Plot3d) in a few different ways:

• 3D smoke/fire rendering in our Results application.
• Volumetric data rendering in our Results application.
• FED calculations in Pathfinder when using FDS integration.

There are some downsides to the to the current volumetric output options for our uses, including the following:

• s3d files:
  • Low precision (8-bit, normalized integer)
  • Geared toward smoke/fire visualization, as each value is adjusted based on delta X before being normalized.
  • Normalization range is found in different files (I believe HRRPUV in the smv file for fire and EXTINCTION COEFFICTION for smoke).
• Plot3D files:
  • Limit of 5 quantities
  • Always writes 5 quantities (I think)
  • One file per time step, which can result in lots of files if you want high time-resolution
  • Always output for every mesh
  • More precision than is often needed (32-bit floating point)
  • No data compression
• 3D slices:
  • More precision than is often needed (32-bit floating point)
  • No data compression

The biggest downsides to us are either the lack of precision in s3d files or very large output files for Plot3D and 3D slices.

Request

We would like to request an alternate volumetric output that can provide the following:

• A single file per quantity per mesh, similar to 3D slices, possibly with control over the mesh regions that are output.
• Lower precision data, such as FP16 or a normalized 16-bit integer. If using the latter, the normalization range would ideally be readily available in the file per frame. Icing on the cake would be multiple, user-configurable precision options (i.e. 32-bit floating-point, 32-bit normalized integer, 16-bit floating-point, 16-bit normalized integer, or 8-bit normalized integer. But a single 16-bit format would be a great first step.
• Data compression per frame. Some options are:
  • RLE like the s3d file already uses
  • LZ4 compression. This might give better compression for the higher-precision formats and is extremely fast. It's also available as open source.

Possible Solutions

I think there might be a few avenues to accomplish this request:

• Modify the s3d file format. The SM3D record would be used to define the desired quantities.
  • Include a 16-bit precision option. The precision being used would be written as a flag in the file header.
  • If using a normalized integer format, write the normalization range for the quantity in each frame header.
  • Write a flag indicating whether the values are adjusted by delta X. I believe this adjustment would only be needed if the s3d file is being used for fire/smoke visualization.
  • The precision and and delta X adjustment inputs would be specified in the SM3D record.
• Modify the 3d slice file format.
  • Include a 16-bit precision option. The precision being used would be written as a flag in the file header.
  • If using a normalized integer format, write the normalization range for the quantity in each frame header.
  • Add an option for whether to use LZ4 compression (or whatever compression is chosen). This is indicated by a flag in the file header.
  • The precision and compression inputs would be specified in the SLCF record.
• Create a new output option that writes using HDF5 and uses one of the compression options (LZ4 is available). I'm not as familiar with this file format, so it might need more investigation.

[drjfloyd]

I had developed at one point a two byte real output.

bit 1: flag for negative or positive bits 2-11: 0.000 to 1.000 bits 11-16: E-16 to E15

As an example:

-1,567
-0.157E4

bit 1: 1 (-) bits 2-10: 0010011101 (157) bits 11-16: 10100 (20 - 16 = 4) output: 10010011 10110100

This has an advantage over FP16 in that it gives a much larger real range (FP16 is 2^-14 to 2^15). E-16 should cover anything small that we care about and E+15 should cover anything large that we care about. For the purpose of doing FED calcs or coloring contours, three digits of precision should be good enough.

[drjfloyd]

Be a function like this for the conversion from real to two bytes

INTGER(2), FUNCTION TWO_BYTE_REAL(REAL_IN)
REAL(EB), INTENT(IN) :: REAL_IN
INTEGER:: EXP, TEMP

EXP = MAX(-16_2,MIN(15_2,INT(FLOOR(LOG10(ABS(REAL_IN))),2)))
TEMP = MIN(1000,INT(ABS(REAL_IN*10._EB**(-REAL(EXP+1,EB)+4)),2))
TWO_BYTE_REAL = TEMP*2**5+ EXP + 16
IF (REAL_IN <0._EB) TWO_BTYE_REAL = IBSET(TWO_BYTE_REAL,15)

END FUNCTION TWO_BYTE_REAL

[rjokonski]

I had forgotten about that fairly big range limitation with FP16. I'm certainly open to other formats, such as Jason's two-byte format. I think the normalized integer format could also be a viable option, especially if the data is normalized over the range of the data set for a single frame.

[drjfloyd]

two-byte integer normalized over the range would be pretty good resolution for most quantities. With temperature that would mean 0.02 K with a peak 1200 C. For velocity 4 mm/s for -100 to 100 m/s. CO would be 2 ppm with 2 step-chemistry where the peak could be near 10 %.

[mcgratta]

The smoke3d output is still very specific to visualizing smoke and fire. If you look in smvv.f90, around line 924, you'll see three formats for GAS, FIRE and TEMPERATURE. GAS is aimed at obscuration. For a GAS, the entry in the .smv file is

SMOKF3D     2   8700.000
 Case_old_2_1.s3d
 SOOT DENSITY
 rho_C0.9H0.1
 kg/m3

You see there the 8700 m2/kg with which one could back out the original SOOT DENSITY in kg/m3. That is, the 1 byte integers are 254(1-exp(-8700dx*rho_soot)). TEMPERATURE and FIRE are handled differently.

[gforney]

If anyone wants to play around with something, I've added output for 3d smoke quantities higher in resolution than what we've output earlier but lower than full fledged floating point. fds (in nightly bundles) now outputs .v3d files of 16 bit integers for various 3d smoke values if you put VOL3D=.TRUE. on the &MISC line. Each 16 bit integer VAL16 is computed using VAL16 = (2*16-1)(VAL-VALMIN)/(VALMAX-VALMIN) where VALMIN and VALMAX are the min nad max values for the time frame being output.

For now these files are uncompressed. The header is one record containing INTEGER_ONE,VOL3D_VERSION,VOL3D_COMPRESS,IBP1,JBP1,KBP1 where INTEGER_ONE is the integer 1, VOL3D_VERSION is the file format version, VOL3D_COMPRESS is a flag indicating whether file is compressed , IBP1, JPB1, KPB1 are the node dimensions of the mesh where the data is taken from

Each time frame contains 3 records TIME NVALS,NVALS16_OUT,VOL3D_VALMIN,VOL3D_VALMAX (BUFFER(I),I=1,NVALS)

NVALS is number of 16 bit integers before compression, just IBP1JBP1JBP1, NVALS16_OUT is the number of data values output currently the same as NVALS since there is no compression, VOL3D_VALMIN and VOL3D_VALMAX are the smallest and largest data values in the mesh for that time frame.

I'm looking at compressing these values

[mcgratta]

Is this in the FDS User's Guide?

[gforney]

not yet. give me a few days. think I can get compression figured out in that time.

On Tue, Dec 13, 2022 at 2:32 PM Kevin McGrattan @.***> wrote:

Is this in the FDS User's Guide?

— Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/11212#issuecomment-1349586370, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC6UCRSKRMXQFLKKMBN4HSDWNDFN3ANCNFSM6AAAAAASZUQL2Q . You are receiving this because you were assigned.Message ID: @.***>

-- Glenn Forney

[mcgratta]

Wouldn't compression work the same way as for 8 bit?

[drjfloyd]

RLE won't be nearly as effiicent with 16 bit. Were you thinking of another method?

[gforney]

was going to call a comporession library much the same way that we call routines in the MKL library now using fortran bindings. RLE won't work well for 16 bit data because it is unlikely that two adjacent 16 values will be the same. I will do this in my own branch. then you all call can decide whether you want me to merge my changes.

On Tue, Dec 13, 2022 at 2:42 PM Jason Floyd @.***> wrote:

RLE won't be nearly as effiicent with 16 bit. Were you thinking of another method?

— Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/11212#issuecomment-1349595198, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC6UCRSGMPCMYLUW6E4IBYLWNDGRHANCNFSM6AAAAAASZUQL2Q . You are receiving this because you were assigned.Message ID: @.***>

-- Glenn Forney

[mcgratta]

If that's the case, wouldn't any other compression algorithm be somewhat lossy, and do we want that?

[gforney]

I was going to use the same compression library used by smokeview, zlib. It is lossless. zlib is also used to compress images in png files. There are fortran bindings for it.

On Tue, Dec 13, 2022 at 2:51 PM Kevin McGrattan @.***> wrote:

If that's the case, wouldn't any other compression algorithm be somewhat lossy, and do we want that?

— Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/11212#issuecomment-1349604113, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC6UCRTYWLDPGBQGNULD36DWNDHVPANCNFSM6AAAAAASZUQL2Q . You are receiving this because you were assigned.Message ID: @.***>

-- Glenn Forney

[mcgratta]

Do we have the source code for it, or do we have to link to binaries. I'm not a fan of the latter. Slippery slop to a point where we cannot compile the code anymore.

[gforney]

yes we have the source code. zlib is one of the libraries I build when building smokeview.

On Tue, Dec 13, 2022 at 3:20 PM Kevin McGrattan @.***> wrote:

Do we have the source code for it, or do we have to link to binaries. I'm not a fan of the latter. Slippery slop to a point where we cannot compile the code anymore.

— Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/11212#issuecomment-1349635257, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC6UCRRCDTGSE2LXEEK4DUTWNDLCBANCNFSM6AAAAAASZUQL2Q . You are receiving this because you were assigned.Message ID: @.***>

-- Glenn Forney

[mcgratta]

Mixed C and Fortran? If we can write a simple Fortran routine to do what we need, that would be better. No end of headaches when we start mixing languages.

[gforney]

if there is any interest you can reopen the issue

[rjokonski]

I finally got around to testing this feature. I'm sorry it took so long. I really appreciate how responsive you were when I first opened the case - it actually took me by surprise, but I was bogged down with other projects at the time.

I would like to request a few additions/changes to the feature as it went out in 6.8.0. Should I post them here or in a new issue? If the latter, would it be better to lump them into one issue or each separate?

[gforney]

You can put them here

On Wed, May 3, 2023, 6:03 PM rjokonski @.***> wrote:

I finally got around to testing this feature. I'm sorry it took so long. I really appreciate how responsive you were when I first opened the case - it actually took me by surprise, but I was bogged down with other projects at the time.

I would like to request a few additions/changes to the feature as it went out in 6.8.0. Should I post them here or in a new issue? If the latter, would it be better to lump them into one issue or each separate?

— Reply to this email directly, view it on GitHub https://github.com/firemodels/fds/issues/11212#issuecomment-1533810426, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC6UCRVDHWDJSLLRXHDBOVDXELI2FANCNFSM6AAAAAASZUQL2Q . You are receiving this because you modified the open/close state.Message ID: @.***>

[rjokonski]

Thanks. Here are my requests:

• Move the SMOKE3D_16 flag from MISC to the SM3D record. This would allow us to only write 16-bit files for the quantities we really want to be higher resolution. We don't necessarily need it for HRRPUV and soot density for smoke/fire visualization.
• If SMOKE3D_16 is true for an SM3D record, don't write the 8-bit version. This will help save some disk space and duplicate data.
• On the DUMP record, add a separate DT parameter for 16-bit SM3D records (e.g. DT_SMOKE3D_16 ). This way, we can have 8-bit SM3D quantities (e.g. hrrpuv/soot density) write at a high frequency for visualization and 16-bit quantities write at a lower frequency to save disk space.
• Add compression. I understand there are some concerns about using C bindings in FORTRAN, but I think it's still pretty important, as even these 16-bit data files could get pretty large uncompressed. In addition, there are cases where we need several of these files. For example, in Pathfinder, we need 4 quantities to compute FED, which increases the cost even more. We could potentially compress these files ourselves with a post-processing tool, but if it ends up being trivial to do from within FDS, it sure would be nice 😄

Thanks again for the rapid responses!

[rjokonski]

Hello,

@gforney I was wondering if you ever had a chance to revisit this or think about how the above could be accomplished. Even if we could get the top 3 bullets done, I think that would be a big win for us.

Thanks!

[gforney]

This fell through the cracks so to speak. Ill look at this again