Problem with R200m unit

[SuhoRyu]

I'm following the tutorial and trying to run a commend " echo "137477798 66 33.03249 182.29579 167.79785 2.245273" | shellfish shell "

I'm 100% sure all the numbers and units are correct (id, snap index, x, y, z [cMpc/h], R200m [cMpc/h]); however, the code returns an error:

" panic: No input r and phi seuqences to NewKDETree. This can sometimes happen if incorrect halo positions are given (e.g. using a halo catalog from the wrong simulation suite, configuring your halo finder incorrectly, mistyping a coordinate, etc). If you are sure that your input locations correspond to actual halo centers, this might also be an internal Shellfish error and you should submit a bug report.

goroutine 1 [running]: github.com/phil-mansfield/shellfish/los/analyze.NewKDETree(0xc00019d508, 0x0, 0x0, 0xc00019d508, 0x0, 0x0, 0x3, 0x3fe58df9bdc69f8d, 0x10000c00019d570, 0x41a32ba) /{GOPATH}/go/pkg/mod/github.com/phil-mansfield/shellfish@v1.0.2/los/analyze/kde.go:73 +0x3e9 github.com/phil-mansfield/shellfish/los/analyze.FilterPoints(0xc000248000, 0x64, 0x64, 0x3, 0x3fe58df9bdc69f8d, 0xba3acb0, 0x1f04061417, 0xba3ac58, 0xc00019d6d8, 0x401f845, ...) /{GOPATH}/go/pkg/mod/github.com/phil-mansfield/shellfish@v1.0.2/los/analyze/penna.go:147 +0x655 github.com/phil-mansfield/shellfish/cmd.calcCoeffs(0xc0043cc000, 0xc000248000, 0x64, 0x64, 0xc0000b0080, 0xc000014540, 0x40408428a1dfb939, 0x4066c9771c970f7c, 0x4064f987fcb923a3) /{GOPATH}/go/pkg/mod/github.com/phil-mansfield/shellfish@v1.0.2/cmd/shell.go:618 +0xf1 github.com/phil-mansfield/shellfish/cmd.haloAnalysis(0xc000010008, 0x1, 0x1, 0xc0001faf00, 0x1, 0x1, 0xc0000b0080, 0xc000248000, 0x64, 0x64, ...) /{GOPATH}/go/pkg/mod/github.com/phil-mansfield/shellfish@v1.0.2/cmd/shell.go:517 +0x145 github.com/phil-mansfield/shellfish/cmd.loop(0xc0001fae38, 0x1, 0x1, 0xc0001fae40, 0x1, 0x1, 0xc0000e2420, 0x4, 0x4, 0xc0000b0080, ...) /{GOPATH}/go/pkg/mod/github.com/phil-mansfield/shellfish@v1.0.2/cmd/shell.go:360 +0xfeb github.com/phil-mansfield/shellfish/cmd.(*ShellConfig).Run(0xc0000b0080, 0xc000014540, 0xc0001d2150, 0xc000012060, 0x1, 0x2, 0x0, 0x0, 0x1363ef9c1acdc4de, 0xc0001de000, ...) /{GOPATH}/go/pkg/mod/github.com/phil-mansfield/shellfish@v1.0.2/cmd/shell.go:242 +0x4c5 main.main() /{GOPATH}/go/pkg/mod/github.com/phil-mansfield/shellfish@v1.0.2/shellfish.go:407 +0x4b4 "

Error occurs no matter how I set "HaloRadiusUnits = " option in the config file. My halo catalog (which is probably not read in this step since I haven't put any halo cat path in the config) has radius unit of ckpc/h, and the number in the catalog(for R200m) is "2245.273000". I wrote 2.245273 in the commend input because the tutorial said to write in in cMpc/h.

When I put 2245.273 for R200m instead, the code finished without an error message and returned,

" Column contents: ID(0) Snapshot(1) X cMpc/h Y cMpc/h Z cMpc/h R200m cMpc/h P_ijk(6-23) 137477798 66 33.0325 182.296 167.798 2245.27 846.305 32.8656 203.307 192.045 -654.128 100.714 62.0411 -63.9591 -818.228 -17.2061 -371.407 -245.007 590.596 -1327.03 -33.4791 468.477 1211.72 -3370.05 " (when "HaloRadiusUnits" is set to ckpc/h in the config)

" Column contents: ID(0) Snapshot(1) X cMpc/h Y cMpc/h Z cMpc/h R200m cMpc/h P_ijk(6-23) 137477798 66 33.0325 182.296 167.798 2245.27 871.953 56.7776 170.773 74.5379 -670.96 307.785 -106.902 -119.194 -606.173 -19.3962 -395.785 -299.032 460.241 -1608.09 -164.442 290.083 621.525 -3361.45 " (when "HaloRadiusUnits" is set to cMpc/h in the config)

Though, I don't think the output has much reliability since the output seems to change every time I run the code with same numbers and config file. I do see the tutorial tells me to remove all the config settings starting with "Halo" and "Tree", but since the code returned " 2021/08/03 20:40:15 Error running mode shell: The 'HaloRadiusUnits' variable isn't set. " I had to put those options: " HaloPositionUnits = cMpc/h HaloRadiusUnits = cMpc/h HaloMassUnits = Msun/h "

[SuhoRyu]

It's just a guess without evidence but, I think maybe it's due on the units of the snapshot file.

I'm using lgadget2 snapshots. But, unlike normal gadget files I see mine does not have unit infos in the header, and all the lengths including position, box size are in kpc/h. If shellfish is thinking all the snapshot data are in Mpc/h, it will definitely cause a problem.

Is there a way I could test this hypothesis or fix the problem?

[phil-mansfield]

Hi, just wanted to send you a quick note that I've seen this and will take a careful look at it after my morning meetings finish up. It sounds reasonable that there might be an issue with units: if that turns out to be the case, I'll add some configuration variables that lets you tell Shellfish what units your files use.

[SuhoRyu]

Thank you! :-)

[phil-mansfield]

Okay, sorry that it took me a bit to get back to you.

The error message you're getting usually happens when there are no/very few particles inside a region that Shellfish is trying to analyze. Most of the time when this happens, it's because of tying to find haloes that are outside the simulation box or using different units for the particles and haloes.

The variance between different analysis runs isn't neccessarily a problem, although by eye it seems a bit larger than I expect. By default, Shellfish uses a random number generator to distribute unit vectors over a sphere relatively evenly (you can change this, but it's usually not worth it). Appendix C.2 and Figure 16 in the code paper estimate that on average, you'll have ~1%-sized effects in the most important quantities. Additionally, there are some modest degeneracies between different fit parameters, so sometimes the difference between two shell shapes is smaller than you would think just by eye-balling the fit parameters.

That said, I don't think that fit is centered on a halo, since it looks like the positions are still in Mpc/h.

In terms of a fix, there's a set of config variables you can use to automatically convert your Gadget units to Shellfish's code units (I forgot they existed, or else I would have pointed you to them earlier. Sorry!).

# GadgetPositionUnits indicates how positions are stored within your Gadget
# snapshot. Set this variable so the following equation is true:
# (1 Mpc/h) * GadgetPositionUnits = (Your position units).
# (i.e. if your position units are smaller than 1 Mpc/h, this variable should
# be less than one). This variable only needs to be set if
# SnapshotType = Gadget-2 and your units are not 1 Mpc/h.
# GadgetPositionUnits = 1.0

# GadgetMassUnits indicates how positions are stored within your Gadget
# snapshot. Set this variable so the following equation is true:
# (1 Msun/h) * GadgetMassUnits = (Your mass units).
# (i.e. if your mass units are smaller than 1 Msun/h, this variable should be
# less than one). This variable only needs to be set if SnapshotType = Gadget-2
# and your units are not 1 Msun/h.
# GadgetMassUnits = 1.0

In principle, your fix (converting all the distance units to kpc/h instead of Mpc/h) would also work, but you'd have to be careful about making sure that all the input and config variables are handled consistently and it would make the comments on some of the output incorrect. Also, if I'm forgetting some place where the code needs to know the actual value of a density or something, you could run into a problem. Probably simpler to use GadgetPositionUnits.

[SuhoRyu]

Thank you.

But unfortunately, neither 'Gadget-2' type on SnapshotType variable nor 'Gadget{Mass/Position}Units' variable is recognized but the code.

" 2021/08/04 15:13:44 Error running mode shell: The 'SnapshotType' variable is set to 'Gadget-2', which I don't recognize. Shellfish terminating. "

" 2021/08/04 15:12:56 Error running mode shell: Line 4 of the config file /Users/ryusuho/shellfish/my.config assigns a value to the variable 'GadgetPositionUnits', but config files of type config don't have that variable. Shellfish terminating. "

[phil-mansfield]

That's odd. Could you send me the following:

1. The commands you ran when you got this error.
2. The contents of any config files you exported or passed to shellfish while running this command.
3. The same two things for the original command you ran which crashed with the much longer error message about phi and r.

As an initial check, I looked at the config setup, and GadgetPositionUnits should be included in the main [config] file but not in the [shell] one (code is here). So double check that this is being passed to the main config file and not the shell file. I'm a bit confused by the "Gadget-2" issue, since I have old config files that used this setting and since the code looks like it should work (the conditional is here). I was wondering if maybe the dash in your version of "Gadget-2" was a special unicode character or something, but I checked and it isn't. Of course, it's possible that there's some bug in the code that isn't obvious to me at first look.

If I can't figure it out by looking at the commands and config files, I'll probably ask you to add some print statements into the source code to see if I can debug it that way.

[phil-mansfield]

As another check, I pulled out the code that checks the snapshot type and ran it with the string that gets reported in the error message. This runs as expected. https://play.golang.org/p/RNADg7vmt2w So, it seems like something subtle is happening here.

[SuhoRyu]

1. The commands you ran when you got this error. -> I'm still with " echo "137477798 66 33.03249 182.29579 167.79785 2.245273" | shellfish shell "

2. The contents of any config files you exported or passed to shellfish while running this command. -> Here is the full content of the config file. " [config] Version = 1.0.0 SnapshotType = Gadget-2 GadgetPositionUnits = 0.001 GadgetMassUnits = 1e10 HaloType = nil TreeType = nil HaloPositionUnits = cMpc/h HaloRadiusUnits = ckpc/h HaloMassUnits = Msun/h SnapshotFormat = /Users/ryusuho/CosmologyRawData/MassiveNuS/snapshot/mnv0.0/snapshot%03d.%d SnapshotFormatMeanings = Snapshot, Block BlockMins = 0 BlockMaxes = 27 SnapMin = 12 SnapMax = 66 # ScaleFactorFile = path/to/file.txt MemoDir = /Users/ryusuho/Cosmology_RawData/MassiveNuS/shellfish/mnv0.0/memo Endianness = SystemOrder ValidateFormats = false Threads = -1 Logging = nil # GadgetNpartNum = 2 "

and if this helps, this is what is printed if I type "shellfish help config" " (base) ryusuho@ryusuho-MacPro ~ % shellfish help config [config] # Target version of shellfish. This option merely allows Shellfish to notice # when its source and configuration files are not from the same version. It will # not allow previous versions to be run from earlier versions. Version = 1.0.0

# These variables describe the formats used by the files which Shellfish reads. # If your simulation output uses a format not included here, you can submit a # request for support on https://github.com/phil-mansfield/shellfish/issues. # # The "nil" flag tells shellfish that you won't try to use any of the tools # related to this type of tile. If you set HaloType to nil, you can't use # id, coord, or tree, and if you set TreeType to nil, you can't use tree. Do # this if you're planning to read your own halo catlogs/merger trees and hand # input directly to shellfish's shell mode. If HaloType is nil, you don't need # to fill out any of the Halo variables in this config file and if TreeType # is nil, you don't need to fill out any of the Tree variables. # # Supported SnapshotTypes: LGadget-2, ARTIO (experimental), gotetra # Supported HaloTypes: Text, nil # Supported TreeTypes: consistent-trees, nil SnapshotType = LGadget-2 HaloType = Text TreeType = consistent-trees

# HaloValueNames and HaloValueColumns tell Shellfish about the structure of your # halo catalogs. It needs to be able to read three pieces of information: the # halos IDs, the positions and the masses. HaloValueNames should be an ordered # list of columns (the set below is the minimum required) and the 0-indexed # columns of your halo catalog that they appear in. HaloValueNames = ID, X, Y, Z, M200m HaloValueColumns = 0, 2, 3, 4, 20 # HaloValueComments can be used to include notes about, e.g. units in output # catalogs. These are not analyzed by Shellfish in any way, but will be # propagated to output catalogs when relevant. HaloValueComments = "", "cMpc/h", "cMpc/h", "cMpc/h", "Msun/h"

# HaloPositionUnits are the units which your halo catalog reports positions in. # Currently supported values are "cMpc/h" and "ckpc/h" (the "c" stands for # "comoving") and "pMpc/h" and "pkpc/h" (the "p" stands for "physical"). The # same units without the "/h" are also supported. HaloPositionUnits = cMpc/h # HaloRadiusUnits are the units which your halo catalog reports radii in. # Currently supported values are "cMpc/h" and "ckpc/h" (the "c" stands for # "comoving"). HaloRadiusUnits = ckpc/h # HaloMassUnits are the units which your halo catalog reports masses in. # Currently only "Msun/h" is supported. HaloMassUnits = Msun/h

# These next couple of variables are neccessary evils due to the fact that there # are a wide range of directory structures used in different simulations. They # will be sufficient to specify the location of snapshots in the vast majority # of cases. I give an in-depth description of how to use them in the file # doc/directoryconfig.md. SnapshotFormat = path/to/snapshots/snapdir%03d/snapshot_%03d.%d # Valid values are "Snapshot", "ScaleFactor", "Block", "Block0", "Block1", etc. # BlockN will reference the Nth element of the BlockMins and Block Maxes # variables. SnapshotFormatMeanings = Snapshot, Snapshot, Block BlockMins = 0 BlockMaxes = 511 SnapMin = 0 SnapMax = 100

# ScaleFactorFile should only be set if one of the elements of # SnapshotFormatMeanings is 'ScaleFactor'. This should point to a file which # contains the scale factors of your files. A file like this can usually be # generated in a few lines of Python: look in doc/scale_factor_ex.py # for an example. # ScaleFactorFile = path/to/file.txt

# Directory containing halo catalogs. It is assumed that when the catalog # catalog files in this directory are sorted in alphabetical order (really: in # lexicographical order), they will also be sorted temporally. It's also assumed # that there are no missing snapshots in the middle of your simulation (e.g. # if snapshot #83 and #85 exist, but snapshot #84 got corrupted and was deleted, # that would be bad, but if you don't have catalogs for the firt ten snapshots # your simulation, that would be fine.) # # If either of these isn't true, you'll still be able to use the shell finding # and analyzing parts of Shellfish, but won't be able to use its catalog reading # tools. HaloDir = path/to/halos/dir/

# Directory containing merger tree. TreeDir = path/to/merger/tree/dir/

# A directory you create the first time you run Shellfish for a particular # simulation. Shellfish will cache certain partial results in this directoy. # Every time a value is changed in this file, you must change the location of # this directory. # ("memo" is a reference to the term "memoization," which is just a fancy # word for caching.) MemoDir = path/to/memo/dir/

# Endianness of any external binary data files read by Shellfish. It should be # set to either SystemOrder, LittleEndian, BigEndian. This variable defaults to # SystemOrder. # # (Any binaries data files written by Shellfish will ignore this variable and # will be written in little endian order. This and a few other details allows # binary files written by Shellfish on one machine to be read by it on any other # machine.) Endianness = SystemOrder

# ValidateFormats checks the the specified halo files and snapshot catalogs all # exist at startup before running any other code. Otherwise, these will be # checked only immediately before a particular file is opened. In general, # it's best to set this to false for short jobs because checking every file # is a lot of system calls and can take minutes. That said, it's generally a # good idea to check at least once after making the config file that you aren't # accidentally specifying nonexistent files. ValidateFormats = false

# Threads is the number of threads that should be run simultaneously. If Threads # is set to a non-positive value (as it is by default), it will automatically # be set equal to the number of available cores on the current node. All threads # will be balanced across available cores. Setting this to a value larger than # the number of cores on the node might result in slightly suboptimal # performance. Threads = -1

# The logging mode to be used. There are three different logging modes: # nil - no logging is performed. # performance - runtime and memory consumption logging are written to stderr. # debugging - debugging information is written to stderr Logging = nil

# GadgetNpartNum is an optional variable which should only be set when using # Gadget files. If your Gadget files use two elements of Npart and Nall to # represent the number of dark matter particles in your simulation, set this # variable to 2. If every element corresponds to a different particle species, # set this variable to 1. GadgetNpartNum defaults to the most common value, 2. # # If you don't know what your Gadget file does, leave this alone: Shellfish will # fail and tell you to change this variable. # GadgetNpartNum = 2 "

3. The same two things for the original command you ran which crashed with the much longer error message about phi and r. -> Everything is identical but without "Gadget{Mass/Position}Units" variable and with "SnapshotType = LGadget-2"

Let me check the thing that you've mentioned and get back to you.

+) my cmd.go file also has all the config options (Gadget{Mass/Position}Units for config, "Gadget-2 for SnapshotType) but for some reason it is returning the error.

[SuhoRyu]

If this helps,

This is what was printed when I tried just "go install github.com/phil-mansfield/shellfish" when installing shellfish. " go install: version is required when current directory is not in a module Try 'go install github.com/phil-mansfield/shellfish@latest' to install the latest version "

so when I ran 'go install github.com/phil-mansfield/shellfish@latest' instead, and shellfish was installed, and this is what was output in the console " go: finding module for package github.com/phil-mansfield/consistent_trees go: finding module for package github.com/phil-mansfield/go-artio go: finding module for package github.com/phil-mansfield/pyplot go: finding module for package github.com/gonum/matrix/mat64 go: found github.com/phil-mansfield/go-artio in github.com/phil-mansfield/go-artio v0.0.0-20160705224018-09340eaddafe go: found github.com/gonum/matrix/mat64 in github.com/gonum/matrix v0.0.0-20181209220409-c518dec07be9 go: found github.com/phil-mansfield/pyplot in github.com/phil-mansfield/pyplot v0.0.0-20150905195044-eee789bae701 go: found github.com/phil-mansfield/consistent_trees in github.com/phil-mansfield/consistent_trees v0.0.0-20150908160215-a8699173df13 go: finding module for package github.com/gonum/lapack/lapack64 go: finding module for package github.com/gonum/floats go: finding module for package github.com/gonum/lapack go: finding module for package github.com/gonum/blas go: finding module for package github.com/gonum/blas/blas64 go: finding module for package github.com/gonum/internal/asm/f64 go: found github.com/gonum/blas in github.com/gonum/blas v0.0.0-20181208220705-f22b278b28ac go: found github.com/gonum/blas/blas64 in github.com/gonum/blas v0.0.0-20181208220705-f22b278b28ac go: found github.com/gonum/floats in github.com/gonum/floats v0.0.0-20181209220543-c233463c7e82 go: found github.com/gonum/internal/asm/f64 in github.com/gonum/internal v0.0.0-20181124074243-f884aa714029 go: found github.com/gonum/lapack in github.com/gonum/lapack v0.0.0-20181123203213-e4cdc5a0bff9 go: found github.com/gonum/lapack/lapack64 in github.com/gonum/lapack v0.0.0-20181123203213-e4cdc5a0bff9 # github.com/phil-mansfield/consistent_trees read_tree.c:88:28: warning: using floating point absolute value function 'fabs' when argument is of integer type [-Wabsolute-value] read_tree.c:88:28: note: use function 'llabs' instead read_tree.c:88:38: warning: using floating point absolute value function 'fabs' when argument is of integer type [-Wabsolute-value] read_tree.c:88:38: note: use function 'llabs' instead "

[phil-mansfield]

Okay, this is interesting! Looks like this is an old version of shellfish. I guess the @latest thing is going back to an earlier commit or something... Thanks for printing out the config file help output: would've taken me a while to work it out without that :)

I'll figure out what the new module system is doing and fix things accordingly.

[SuhoRyu]

Maybe I'll try something like ~shellfish@1.0.0 or something first. Thanks!

[SuhoRyu]

So,

I tried "go install github.com/phil-mansfield/shellfish@v1.0.0" at /usr/local/go which returned " go: downloading github.com/phil-mansfield/shellfish v1.0.0 go: finding module for package github.com/phil-mansfield/consistent_trees go: finding module for package github.com/phil-mansfield/go-artio go: finding module for package github.com/phil-mansfield/pyplot go: finding module for package github.com/gonum/matrix/mat64 go: found github.com/phil-mansfield/go-artio in github.com/phil-mansfield/go-artio v0.0.0-20160705224018-09340eaddafe go: found github.com/gonum/matrix/mat64 in github.com/gonum/matrix v0.0.0-20181209220409-c518dec07be9 go: found github.com/phil-mansfield/pyplot in github.com/phil-mansfield/pyplot v0.0.0-20150905195044-eee789bae701 go: found github.com/phil-mansfield/consistent_trees in github.com/phil-mansfield/consistent_trees v0.0.0-20150908160215-a8699173df13 go: finding module for package github.com/gonum/lapack/lapack64 go: finding module for package github.com/gonum/blas/blas64 go: finding module for package github.com/gonum/internal/asm/f64 go: finding module for package github.com/gonum/blas go: finding module for package github.com/gonum/floats go: finding module for package github.com/gonum/lapack go: found github.com/gonum/blas in github.com/gonum/blas v0.0.0-20181208220705-f22b278b28ac go: found github.com/gonum/blas/blas64 in github.com/gonum/blas v0.0.0-20181208220705-f22b278b28ac go: found github.com/gonum/floats in github.com/gonum/floats v0.0.0-20181209220543-c233463c7e82 go: found github.com/gonum/internal/asm/f64 in github.com/gonum/internal v0.0.0-20181124074243-f884aa714029 go: found github.com/gonum/lapack in github.com/gonum/lapack v0.0.0-20181123203213-e4cdc5a0bff9 go: found github.com/gonum/lapack/lapack64 in github.com/gonum/lapack v0.0.0-20181123203213-e4cdc5a0bff9 # github.com/phil-mansfield/consistent_trees read_tree.c:88:28: warning: using floating point absolute value function 'fabs' when argument is of integer type [-Wabsolute-value] read_tree.c:88:28: note: use function 'llabs' instead read_tree.c:88:38: warning: using floating point absolute value function 'fabs' when argument is of integer type [-Wabsolute-value] read_tree.c:88:38: note: use function 'llabs' instead # github.com/phil-mansfield/shellfish/los/analyze /Users/ryusuho/go/pkg/mod/github.com/phil-mansfield/shellfish@v1.0.0/los/analyze/shell.go:152:23: not enough arguments in call to eigen.Factorize have (*mat64.Dense, bool) want (mat64.Matrix, bool, bool) "

Then I tried "go install github.com/phil-mansfield/shellfish" at /usr/local/go/src, /usr/local/go/src/github.com, /usr/local/go/src/github.com/phil-mansfield and /usr/local/go/src/github.com/phil-mansfield/shellfish which all returned " go: inconsistent vendoring in /usr/local/go/src: github.com/gonum/blas@v0.0.0-20181208220705-f22b278b28ac: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt github.com/gonum/floats@v0.0.0-20181209220543-c233463c7e82: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt github.com/gonum/internal@v0.0.0-20181124074243-f884aa714029: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt github.com/gonum/lapack@v0.0.0-20181123203213-e4cdc5a0bff9: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt github.com/gonum/matrix@v0.0.0-20181209220409-c518dec07be9: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt github.com/phil-mansfield/consistent_trees@v0.0.0-20150908160215-a8699173df13: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt github.com/phil-mansfield/go-artio@v0.0.0-20160705224018-09340eaddafe: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt

To ignore the vendor directory, use -mod=readonly or -mod=mod. To sync the vendor directory, run: go mod vendor ".

[phil-mansfield]

Okay, I've looked into this: this is actually a several-hour fix (don't worry about that: I'd need to do this anyway). The most recent compiler version made some changes that are pretty work-intensive if you set things up that way I did a few years ago.

I should be about to get to this and then finish it by tomorrow afternoon/evening.

[phil-mansfield]

Can you try pulling the newest version and then running "go install" from the command line inside the shellfish directory? If you get any error message, let me know what they are.

[SuhoRyu]

Good news is that unlike previous time, it installed without an error or warning. Plus, before, path for shellfish wasn't created automatically so I had to set an alias to use the command "shellfish", but now it's fine. I see the version I got installed before was 1.0.2 and the config told me to use 1.0.0, but now it is both 1.0.3. I also now see extra config settings for each snapshot types in the "shellfish help config" output.

The bad news is that " echo "137477798 66 33.03249 182.29579 167.79785 2.245273" | shellfish shell " still gives error. " (base) ryusuho@ryusuho-MacPro ~ % echo "137477798 66 33.03249 182.29579 167.79785 2.245273" | shellfish shell panic: EOF

goroutine 1 [running]: github.com/phil-mansfield/shellfish/io.readInt32(0x426af88, 0xc00019a098, 0x426cf98, 0x4389478, 0xc01f96d130) /usr/local/go/src/github.com/phil-mansfield/shellfish/io/lgadget2.go:65 +0xb7 github.com/phil-mansfield/shellfish/io.(Gadget2Buffer).readGadget2Particles(0xc0001fab40, 0xc000283c70, 0x4a, 0x426cf98, 0x4389478, 0x0, 0x0, 0x3f2da26, 0x0, 0x0, ...) /usr/local/go/src/github.com/phil-mansfield/shellfish/io/gadget2.go:123 +0x6ca github.com/phil-mansfield/shellfish/io.(Gadget2Buffer).Read(0xc0001fab40, 0xc000283c70, 0x4a, 0x4389478, 0xc0001fab58, 0x0, 0x0, 0x0, 0x7e0, 0x4450a68, ...) /usr/local/go/src/github.com/phil-mansfield/shellfish/io/gadget2.go:368 +0x17e github.com/phil-mansfield/shellfish/io.(Gadget2Buffer).ReadHeader(0xc0001fab40, 0xc000283c70, 0x4a, 0xc0004bc000, 0x0, 0x0) /usr/local/go/src/github.com/phil-mansfield/shellfish/io/gadget2.go:392 +0xf6 github.com/phil-mansfield/shellfish/cmd/memo.readUnmemoizedHeaders(0x42, 0x426cb40, 0xc0001fab40, 0xc0001f6150, 0x426af68, 0xc0001828a0, 0xd, 0x40114bb, 0xba7ba78, 0x0, ...) /usr/local/go/src/github.com/phil-mansfield/shellfish/cmd/memo/memoize.go:250 +0x1ac github.com/phil-mansfield/shellfish/cmd/memo.ReadHeaders(0x42, 0x426cb40, 0xc0001fab40, 0xc0001f6150, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /usr/local/go/src/github.com/phil-mansfield/shellfish/cmd/memo/memoize.go:270 +0x1fe github.com/phil-mansfield/shellfish/cmd.loop(0xc00023a700, 0x1, 0x1, 0xc00023a708, 0x1, 0x1, 0xc00019c540, 0x4, 0x4, 0xc0001f8000, ...) /usr/local/go/src/github.com/phil-mansfield/shellfish/cmd/shell.go:305 +0x8f1 github.com/phil-mansfield/shellfish/cmd.(ShellConfig).Run(0xc0001f8000, 0xc0001fa480, 0xc0001f6150, 0xc000212000, 0x33, 0x200, 0x0, 0x0, 0x4016e9b, 0xc0001bfdc8, ...) /usr/local/go/src/github.com/phil-mansfield/shellfish/cmd/shell.go:241 +0x491 main.main() /usr/local/go/src/github.com/phil-mansfield/shellfish/shellfish.go:440 +0x568 (base) ryusuho@ryusuho-MacPro ~ % "

I tried with several different halos but none of them seems to be giving me results.

[phil-mansfield]

This seems like a problem with the Gadget2 reader. It's trying to read more data than is actually in the file. That code is pretty well tested, but Gadget files are a more complicated format than most people think, so there could be something off.

Can you tell me a bit about the contents of your Gadget files?

• How big are the IDs?
• What sort of particles are in it, and what order are they in?
• Are all these particles the same mass, or is the simulation multi-reosolution?
• Are particle masses in their own MASS block, or are the masses just stored in the header?
• What blocks are in the file, and what order are they in?

(If you're just running standard Gadget-2 or standard LGadget-2, you could also just send me the config file and compile options.)

It's odd that you were able to read it initially, but not in the new version. Makes me wonder if there's a new config variable that's acting up (i.e. one of the variables that got introduced to deal with how complicated Gadget files can get)... After we resolve this, I'll do another pass on the tutorial: it seems like I might not have updated it recently enough.

[phil-mansfield]

Also, can you clarify whether the files were generated by Gadget-2 or by LGadget-2?

[SuhoRyu]

I'm using the mnv0.0 model of MassiveNuS simulation/ Columbia Lensig (http://columbialensing.org). I wanted to sent you a block of snapshot file but it's kind of too big. It's in Gadget-2 binary(format2 according to the papar) and only has DM particles(index 1) with identical mass stored both in header and their mass block. their are snapshot 0 to 66, block 0 to 27. Though I'm only using snapshots >= 12 since there aren't any halos in snapshot < 12.

I'm using PyGadgetReader(https://github.com/jveitchmichaelis/pygadgetreader) to read the snapshots so, I'm not sure about the order but their are pos, vel, pid and mass blocks but none of u, rho ,ne, nh, hsml, sfr, age, z, pot, delaytime, fH2, Sigma, tmax, nspawn and zarray in the snapshot.

Here is the header file for snap 66(z=0). " {'npartThisFile': array([ 0, 66247206, 0, 0, 0, 0], dtype=uint32), 'npartTotal': array([ 0, 1073741824, 0, 0, 0, 0], dtype=uint32), 'npartTotalHW': array([0, 0, 0, 0, 0, 0], dtype=uint32), 'ngas': 0, 'ndm': 1073741824, 'ndisk': 0, 'nbulge': 0, 'nstar': 0, 'nbndry': 0, 'massTable': array([0. , 1.04036745, 0. , 0. , 0. ,

0. ]), 'time': 0.9999999999999996, 'nfiles': 28, 'redshift': 4.440892098500626e-16, 'boxsize': 512000.0, 'O0': 0.3, 'Ol': 0.7, 'h': 0.7, 'flag_cooling': 0, 'flag_sfr': 0, 'flag_fb': 0, 'flag_fh2': 1065353216, 'flag_age': 0, 'flag_metals': 0, 'flag_potential': 1, 'flag_delaytime': 1147463884, 'flag_tmax': 1309414073, 'rhocrit': 9.204287651105119e-30} "

and here is the config setting. I changed the "GadgetSingleMassIndices" variable from "GadgetSingleMassIndices = 0, 1, 2, 3, 4, 5" to "GadgetSingleMassIndices =" since I've noticed there was mass blocks. And now the error message has changed to " (base) ryusuho@ryusuho-MacPro ~ % echo "137425942 66 100.24786 207.19266 20.49122 1.845204" | shellfish shell panic: runtime error: index out of range [25] with length 19

goroutine 1 [running]: github.com/phil-mansfield/shellfish/parse.ReadConfig(0xc000016098, 0x22, 0xc00019dc38, 0x11, 0xbff0000000000000) /usr/local/go/src/github.com/phil-mansfield/shellfish/parse/config.go:354 +0xdd5 github.com/phil-mansfield/shellfish/cmd.(*GlobalConfig).ReadConfig(0xc0001d2480, 0xc000016098, 0x22, 0xc00019dce0, 0x0, 0x0, 0x33, 0x200) /usr/local/go/src/github.com/phil-mansfield/shellfish/cmd/cmd.go:137 +0x9fd main.getGlobalConfig(0xc000012040, 0x2, 0x2, 0xa, 0x32, 0x0, 0x0, 0x4016782) /usr/local/go/src/github.com/phil-mansfield/shellfish/shellfish.go:470 +0xa5 main.main() /usr/local/go/src/github.com/phil-mansfield/shellfish/shellfish.go:377 +0x2bd (base) ryusuho@ryusuho-MacPro ~ % "

[phil-mansfield]

Intersting. Could you send me the first kB of the file? You could get this by running head -c 1024 input_file > output_file. I'll check what Shellfish would do given that header: hopefully I'll be able to work it out from there. Could you also let me know how large the full file is?