Error with gadget2 file - Githubissues

JuliaAstroSim / AstroIO.jl

I/O interface for astrophysical simulation codes

GNU General Public License v3.0

5 stars 1 forks source link

Error with gadget2 file #1

Closed elehcim closed 2 years ago

elehcim commented 3 years ago

Hi @islent, many thanks for this package and the whole JuliaAstroSim ecosystem. I'd like to try out these packages, and I'm interested in using Julia with astrophysical simulations. I'm a contributor to pynbody. I was testing your IO method on test files used by pynbody.

If I use the file gadget1.snap, it works, whereas the file test_g2_snap.0 (which is respecting Gadget2 labeled blocks format) throws an error:

ERROR: LoadError: Wrong location symbol while reading Header!

Here the corresponding logic used in pynbody. https://github.com/pynbody/pynbody/blob/73a9e3626efd96d2b8aba02202e6ab4861c76773/pynbody/snapshot/gadget.py#L262

islent commented 3 years ago

Hello elehcim:

Pleasure to help 😄. However it seems that gadget1.snap and test_g2_snap.0 are directed to the same link: https://github.com/pynbody/testdata/blob/master/gadget1.snap

elehcim commented 3 years ago

Hello elehcim:

Pleasure to help . However it seems that gadget1.snap and test_g2_snap.0 are directed to the same link: https://github.com/pynbody/testdata/blob/master/gadget1.snap

Hi, I fixed the link.

islent commented 3 years ago

Here is the header data of test_g2_snap.0 (debug mode of Debugger.jl):

1|debug> w add header
1] temp1: 8
2] temp2: 877658038
3] header: Gadget2 Header:
  Particle Info :
    |    Type   |  Amount, Mass
    |-----------|-------------------------
    |    Gas    | 1145128264, 0.0 M⊙
    |    Halo   | 264, 1.92685601878e-312 M⊙
    |    Disk   | 8, 0.0 M⊙
    |    Bulge  | 256, 4.0616077817493397e8 M⊙
    |    Star   | 1994, 0.0 M⊙
    | BlackHole | 2050, 0.0 M⊙
    (If zero, mass would be read from file)  Start time: 0.0
  Redshift: 0.0

The data seems to be misplaced so that there could be some impossible numbers. Then I tried to read the first few bytes:

julia> f = open("temp\\test_g2_snap.0")
IOStream(<file temp\test_g2_snap.0>)

julia> tempf = read(f, 8)
8-element Array{UInt8,1}:
 0x08
 0x00
 0x00
 0x00
 0x48
 0x45
 0x41
 0x44

julia> g = open("temp\\gadget1.snap")
IOStream(<file temp\gadget1.snap>)

julia> temp2 = read(g, 8)
8-element Array{UInt8,1}:
 0x00
 0x01
 0x00
 0x00
 0xed
 0xdc
 0x00
 0x00

So the header of test_g2_snap.0 might be broken.

islent commented 3 years ago

Or maybe not, if 1.92685601878e-312 comes from numeric errors. However temp1 has to equal to temp2, and implicitly they both equal to 256.

elehcim commented 3 years ago

The test_g2_snap.0 file follows the labeled block convention (enabled with SnapFormat=2 in a gadget param file). This is a bit more flexible w.r.t. SnapFormat=1 (btw, my simulation snapshots are stored in format 2 :). It is briefly described at page 34 of the Gadget2 User Guide. For reference, I report it here. Each block has a label and is structured like 8<NAME><SIZE+8>8<SIZE><DATA><SIZE>. In fact, in the above, if you try:

julia> String(tempf)
"\b\0\0\0HEAD"

For the case of test_g2_snap.0 it is: 8<HEAD><264>8<256><...256 bytes of header...><256> 8<POS >8<49004>8<48996><...4083*3*4=48996 bytes of 4083 Float32 particle positions (3 dimensions)...><48996> 8<VEL >8<49004>8<48996><...4083*3*4=48996 bytes of 4083 Float32 particle velocities (3 dimensions)...><48996> etc... Other blocks are 'MASS', 'U ', 'RHO ', 'NHP ', 'NHEP', 'NHEQ', 'NH ', 'NHE ', 'HSML', 'SFR '.

islent commented 3 years ago

Thanks for the info. AstroIO.jl has not implemented SnapFormat=2 I/O scheme yet.

I would try it in a few days (less than two weeks), if you're not urgent about this.

btw, my simulation snapshots are stored in format 2

elehcim commented 3 years ago

Thank you, it's not urgent at all. I perhaps can test when you think you have something.

islent commented 3 years ago

The latest commit https://github.com/JuliaAstroSim/AstroIO.jl/commit/5911dd8b23ca3a1d20e1c531d1c9f44fc168d05a has supported format2 (input only), have a try!

POS, VEL, ID, MASS, RHO, HSML blocks are supported. It's quite easy to add other blocks in the future. format2 will be the default format of AstroIO.jl.

I would implement output interfaces later.

elehcim commented 3 years ago

I had a try, it works, thanks!