dnarayanan / powderday

powderday dust radiative transfer
BSD 3-Clause "New" or "Revised" License
22 stars 17 forks source link

ValueError #135

Closed ACDylan closed 3 years ago

ACDylan commented 3 years ago

Hi, I want to test Powderday with the example snapshot provided by the website (gizmo cosmological zoom simulation of a Milky Way mass galaxy) before applying on my own Gadget simulation. However, reaching this step with yt (below), I am facing a ValueError issue.

yt : [INFO ] 2021-07-27 01:59:39,347 Octree bound 155088345 particles /home/chosson/POWDERDAY/yt/yt/fields/field_info_container.py:380: VisibleDeprecationWarning: 'particle_type' keyword argument is deprecated in favour of the positional argument 'sampling_type'. Deprecated since v4.0.0. This feature will be removed in v4.1.0 sampling_type = self._sanitize_sampling_type( Traceback (most recent call last): File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/front_ends/gadget2pd.py", line 322, in gadget_field_add File "/home/chosson/POWDERDAY/yt/yt/data_objects/static_output.py", line 1537, in add_field self.field_info.add_field(name, function, sampling_type, kwargs) File "/home/chosson/POWDERDAY/yt/yt/fields/field_info_container.py", line 394, in add_field self.alias(name, tuple_name) File "/home/chosson/POWDERDAY/yt/yt/fields/field_info_container.py", line 484, in alias self.add_field( File "/home/chosson/POWDERDAY/yt/yt/fields/field_info_container.py", line 396, in add_field self[name] = DerivedField(name, sampling_type, function, kwargs) File "/home/chosson/POWDERDAY/yt/yt/fields/field_info_container.py", line 428, in setitem ftype, fname = key ValueError: too many values to unpack (expected 2)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/chosson/local/Python-3.9.4/bin/pd_front_end.py", line 4, in import('pkg_resources').run_script('powderday==0.1.0', 'pd_front_end.py') File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/pkg_resources/init.py", line 665, in run_script self.require(requires)[0].run_script(script_name, ns) File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/pkg_resources/init.py", line 1470, in run_script exec(script_code, namespace, namespace) File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/EGG-INFO/scripts/pd_front_end.py", line 74, in File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/sph_tributary.py", line 35, in sph_m_gen File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/grid_construction.py", line 49, in yt_octree_generate File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/front_ends/gadget2pd.py", line 334, in gadget_field_add File "/home/chosson/POWDERDAY/yt/yt/data_objects/static_output.py", line 1537, in add_field self.field_info.add_field(name, function, sampling_type, kwargs) File "/home/chosson/POWDERDAY/yt/yt/fields/field_info_container.py", line 396, in add_field self[name] = DerivedField(name, sampling_type, function, kwargs) File "/home/chosson/POWDERDAY/yt/yt/fields/field_info_container.py", line 428, in setitem ftype, fname = key ValueError: too many values to unpack (expected 2)

I don't know if it is related to my own 'parameters_master_401 & parameters_model_401', or if I have missed something during installation, or an issue with the code itself. I am open to any advice and I can provide parameters if needed!

ACDylan commented 3 years ago

Edit : https://github.com/dnarayanan/powderday/issues/127 Just check closed issue and it seems to have a solution.

dnarayanan commented 3 years ago

Hi - thanks for reporting this! Please do let me know if the latest commit doesn't work for you.

ACDylan commented 3 years ago

I have (git) pulled both YT and Powderday, and now I have: yt : [INFO ] 2021-07-27 03:47:24,721 Octree bound 155088345 particles /home/chosson/POWDERDAY/yt/yt/fields/field_info_container.py:380: VisibleDeprecationWarning: 'particle_type' keyword argument is deprecated in favour of the positional argument 'sampling_type'. Deprecated since v4.0.0. This feature will be removed in v4.1.0 sampling_type = self._sanitize_sampling_type( /home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/front_ends/gadget2pd.py:369: VisibleDeprecationWarning: This function is deprecated. Since yt-4.0, it's no longer necessary to add a field specifically for smoothing, because the global octree is removed. The old behavior of interpolating onto a grid structure can be recovered through data objects like ds.arbitrary_grid, ds.covering_grid, and most closely ds.octree. The visualization machinery now treats SPH fields properly by smoothing onto pixel locations. See this page to learn more: https://yt-project.org/doc/yt4differences.html Deprecated since v4.0.0. This feature will be removed in v4.1.0 Traceback (most recent call last): File "/home/chosson/local/Python-3.9.4/bin/pd_front_end.py", line 4, in import('pkg_resources').run_script('powderday==0.1.0', 'pd_front_end.py') File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/pkg_resources/init.py", line 665, in run_script self.require(requires)[0].run_script(script_name, ns) File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/pkg_resources/init.py", line 1470, in run_script exec(script_code, namespace, namespace) File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/EGG-INFO/scripts/pd_front_end.py", line 74, in File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/sph_tributary.py", line 35, in sph_m_gen File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/grid_construction.py", line 49, in yt_octree_generate File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/front_ends/gadget2pd.py", line 388, in gadget_field_add File "/home/chosson/POWDERDAY/yt/yt/data_objects/static_output.py", line 1547, in addfield deps, = self.field_info.check_derived_fields([name]) File "/home/chosson/POWDERDAY/yt/yt/fields/field_info_container.py", line 613, in check_derived_fields fd = fi.get_dependencies(ds=self.ds) File "/home/chosson/POWDERDAY/yt/yt/fields/derived_field.py", line 253, in get_dependencies e[self.name] File "/home/chosson/POWDERDAY/yt/yt/fields/field_detector.py", line 125, in missing vv = finfo(self) File "/home/chosson/POWDERDAY/yt/yt/fields/derived_field.py", line 294, in call dd = self._function(self, data) File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/front_ends/gadget2pd.py", line 141, in _dustmass_manual File "/home/chosson/POWDERDAY/yt/yt/fields/field_detector.py", line 108, in missing finfo = self.ds._get_field_info(*field) File "/home/chosson/POWDERDAY/yt/yt/data_objects/static_output.py", line 819, in _get_field_info field_info, is_ambiguous = self._get_field_info_helper(ftype, fname) File "/home/chosson/POWDERDAY/yt/yt/data_objects/static_output.py", line 897, in _get_field_info_helper raise YTFieldNotFound(field=INPUT, ds=self) yt.utilities.exceptions.YTFieldNotFound: Could not find field ('PartType0', 'Dust_Masses') in snapshot_134.

dnarayanan commented 3 years ago

do you mind please pasting your parameters_master file here? I mostly wonder if the issue is in your choice of:

dust_grid_type = 'dtm'

if this is set to manual (which is what I worry is happening here) then it will look for dust masses that are tied to parttype0. but the example snapshot that ships with powderday doesn't have that kind of field in the PartType0 (i.e., it didn't have an on-the-fly dust model in the original hydro run) so it can't find the required field.

either way, if this is the issue, then this warrants me putting in a more clear error message

ACDylan commented 3 years ago

Indeed, I had 'manual' option, so I have modified my parameters_master. Here it is: `snapshot_num = 134 snapnum_str = '{:03d}'.format(snapshot_num)

snapshotname = 'snapshot'+snapnum_str+'.hdf5'

Resolution Keywords

oref = 0 n_ref = 64 zoom_box_len = 2000 bbox_lim = 100

Parallelization

n_processes = 10 n_MPI_processes = 10

RT Information

n_photons_initial = 1e7 n_photons_imaging = 1e5

n_photons_raytracing_sources =

n_photons_raytracing_dust =

FORCE_RANDOM_SEED = False

seed = -1 # if True, must be a negative integer

Dust Information

dustdir = '/home/chosson/POWDERDAY/hyperion-dust-0.1.0/dust_files/' dustfile = 'd03_3.1_6.0_A.hdf5' PAH = True dust_grid_type = 'dtm' dusttometals_ratio = 1.2 # to set if dust_grid_type = 'dtm' enforce_energy_range = False SUBLIMATION = False

SUBLIMATION_TEMPERATURE = # Meaningless if False beyond

Hydro Code Units (to use if not contained in the HDF5)

unit_mass = 1.989E43 unit_length = 3.085678E21

unit_age = 0.2

unit_velocity = 100000.0

Stellar SEDs Info

FORCE_BINNED = True

max_age_direct = # if Force_BINNED is set to False

imf_type = 1 # 0. Salpeter 1. Chabrier 2. Kroupa 3. Van Dokkum 4. Dave imf_1 = 1.3 imf_2 = 2.3 imf_3 = 2.3 pagb = 1

add_agb_dust_model

Nebular Emission Info

add_neb_emission = True use_cloudy_tables = True use_cmdf = True cmdf_min_mass = 3.5 cmdf_max_mass = 5.0 cmdf_bins = 6.0 cmdf_beta = 2

COMMON PARAMETERS

FORCE_gas_logu = [False,False,False] gas_logu = [-2.0,-2.0,-2.0] # relevant only if 'add_neb_emission' is True and FORCE_gas_logu True also gas_logu_init = [0.0,0.0,0.0] FORCE_gas_logz = [False,False,False] gas_logz = [0.0,0.0,0.0] # relevant only if 'add_neb_emission' is True and FORCE_gas_logz True also Force_gas_logq = [False,False,False] source_logq = [1.0e47,1.0e47,1.0e47] # relevant only if 'add_neb_emission' is True, use_cloudy_tables True and FORCE_gas_logq True also FORCE_inner_radius = [False,False,True] inner_radius = [1.0e19,1.0e19,2.777e20] # relevant only if 'add_neb_emission' is True, use_cloudy_tables True and FORCE_inner_radius True also FORCE_N_O_ratio = False N_O_ratio = -0.85 # relevant only if 'add_neb_emission' is True, use_cloudy_tables False and neb_abund = "direct" neb_abund = ["dopita","dopita","dopita"] # if 'direct', FORCE_BINNED needs to be False

Young Stars

add_young_stars = True HII_Rinner_per_RS = 0.01 HII_nh = 1.0e2 HII_max_age = 1.0e-2 HII_escape_fraction = 0.0

Post-AGB stars

add_pagb_stars = False PAGB_N_enhancement = 0.4 PAGB_C_enhancement = 0.4 PAGB_Rinner_per_Rs = 0.01 PAGB_nh = 1.0e2 PAGB_min_age = 0.1 PAGB_max_age = 10 PAGB_escape_fraction = 0.0

AGN

add_AGN_neb = False AGN_nh = 1.0e3 AGN_num_gas = 32

DIG

add_DIG_neb = False DIG_nh = 10 DIG_min_factor = 1

DEBUGGING AND CLEAN UP

dump_emlines = True # Only True if add_neb_emission = True cloudy_cleanup = True

Birth Cloud Information

CF_on = False birth_cloud_clearing_age = 1 # Meaningless if CF_on = False

Idealized Galaxy SED Parameters

Z_init = 0.02 # 0 = use of Cosmological simulations disk_stars_age = 4 # meaningless if use of cosmo sim bulge_stars_age = 8 # meaningless if use of cosmo sim disk_stars_metals = 20 # meaningless if use of cosmo sim bulge_stars_metals = 20 # meaningless if use of cosmo sim

Stellar Ages and Metallicities

N_STELLAR_AGE_BINS = 3

Black Holes

BH_SED = False

BH_eta = 0.1

BH_model =

BH_modelfile =

nenkova_params =

Images and SED Parameters

NTHETA = 2 IMAGING = True filterdir = '/home/chosson/POWDERDAY/powderday/filters/' filterfiles = 'galex2500.filter' # to join! in filterdir: 'shopt -s globstar; printf "# '%s'\n" *.filter' to print all of them IMAGING_TRANSMISSION_FILTER = False `

The issue is now: RuntimeError: yt attempted to read outside the boundaries of a non-periodic domain along dimension 0. Region left edge = -1393.999944284708 code_length, Region right edge = 1393.999944284708 code_length Dataset left edge = 0.0 code_length, Dataset right edge = 50000.0 code_length

This commonly happens when trying to compute ghost cells up to the domain boundary. Two possible solutions are to select a smaller region that does not border domain edge (see https://yt-project.org/docs/analyzing/objects.html?highlight=region) or override the periodicity with ds.force_periodicity()

dnarayanan commented 3 years ago

great! let me know if it runs please. one thing I noticed is your dtm ratio. the relevant line for a gizmo dataset is around 34 in dust_grid_gen:

dust_smoothed[wFalse] = metallicity_smoothed * density_smoothed * cfg.par.dusttometals_ratio

so what this will do is create the dust density (octree) grid as the metal density * the dust to metal mass ratio . based on this definition, a dust to metals ratio of <=1 would be physical I think.

ACDylan commented 3 years ago

Corrected!

I think this will be the last one: The issue is now: RuntimeError: yt attempted to read outside the boundaries of a non-periodic domain along dimension 0. Region left edge = -1393.999944284708 code_length, Region right edge = 1393.999944284708 code_length Dataset left edge = 0.0 code_length, Dataset right edge = 50000.0 code_length

This commonly happens when trying to compute ghost cells up to the domain boundary. Two possible solutions are to select a smaller region that does not border domain edge (see https://yt-project.org/doc/analyzing/objects.html?highlight=region) or override the periodicity with ds.force_periodicity()

dnarayanan commented 3 years ago

very happy to help!

so two possible issues may be at play.

  1. the octree is constructed within yt which means that it originally needs to start with a box size that encapsulates all particles. this is the reasoning behind two different parameters - bbox_lim (the limit that should be outside of all of your particles) and zoom_box_len (which should be smaller than bbox_lim, and the region you actually want to encompass for your radiative transfer calculations).

https://powderday.readthedocs.io/en/latest/parameters_description.html#resolution-keywords

  1. one other possibility is that the galaxy is not centered correctly in parameters_model

but my gut is that its the first one.

ACDylan commented 3 years ago

"very happy to help!" Thanks a lot, your help is really appreciate!

Is there a solution to know which values should we put for each of these parameters? Because (for extremely large values): zoom_box_len = 50000 bbox_lim = 100000

it gives: yt : [INFO ] 2021-07-27 22:54:40,533 Allocating octree with spatial range [-3.4850e+04, -3.4850e+04, -3.4850e+04] code_length to [3.4850e+04, 3.4850e+04, 3.4850e+04] code_length yt : [INFO ] 2021-07-27 22:54:57,618 Allocating Octree for 155088345 particles yt : [INFO ] 2021-07-27 22:55:06,300 Allocated 4688425 nodes in octree yt : [INFO ] 2021-07-27 22:55:06,300 Octree bound 66332598 particles Interpolating (scatter) SPH field PartType0: 0it [00:00, ?it/s]

And I think, this is too heavy to compute (stuck a 0it for 10+ minutes)

And if I put back reasonable values like: zoom_box_len = 25000 bbox_lim = 50000 I have the same issue again: RuntimeError: yt attempted to read outside the boundaries of a non-periodic domain along dimension 0. Region left edge = -17424.99930355885 code_length, Region right edge = 17424.99930355885 code_length Dataset left edge = 0.0 code_length, Dataset right edge = 50000.0 code_length

dnarayanan commented 3 years ago

are you running the gizmo_mw_zoom example or a different one? does this work?

zoom_box_len = 100 # kpc; so the box will be +/- zoom_box_len from the center
bbox_lim = 1.e5 # kpc - this is the initial bounding box of the grid (+/- bbox_lim)
                # This *must* encompass all of the particles in the
                # simulation.

alternatively if you're using your own simulation, I wonder if there's a way you can make it available so that I can help debug.

ACDylan commented 3 years ago

At first, I am using "gizmo_mw_zoom example" to check that Powderday works well. Then I will use my own 'isolated galaxy's simulation' yes.

With your parameters, I have now:

[octree zoom_bbox_filter:] Calculating Center of Mass [octree zoom_bbox_filter:] Center of Mass is at coordinates (kpc): [unyt_quantity(1.24139586e+26, 'cm'), unyt_quantity(9.81877541e+25, 'cm'), unyt_quantity(1.26603195e+26, 'cm')] [octree zoom_bbox_filter:] using center: [0.0, 0.0, 0.0] [octree zoom] new zoomed bbox (comoving/h) in code units= [[-69.6999972142354, 69.6999972142354], [-69.6999972142354, 69.6999972142354], [-69.6999972142354, 69.6999972142354]] yt : [INFO ] 2021-07-27 23:55:26,089 Calculating time from 9.756e-01 to be 4.263e+17 seconds yt : [INFO ] 2021-07-27 23:55:26,090 Assuming length units are in kpc/h (comoving) yt : [INFO ] 2021-07-27 23:55:26,189 Parameters: current_time = 4.263438511011126e+17 s yt : [INFO ] 2021-07-27 23:55:26,190 Parameters: domain_dimensions = [1 1 1] yt : [INFO ] 2021-07-27 23:55:26,190 Parameters: domain_left_edge = [0. 0. 0.] yt : [INFO ] 2021-07-27 23:55:26,190 Parameters: domain_right_edge = [50000. 50000. 50000.] yt : [INFO ] 2021-07-27 23:55:26,190 Parameters: cosmological_simulation = 1 yt : [INFO ] 2021-07-27 23:55:26,191 Parameters: current_redshift = 0.02499995903287333 yt : [INFO ] 2021-07-27 23:55:26,191 Parameters: omega_lambda = 0.7 yt : [INFO ] 2021-07-27 23:55:26,191 Parameters: omega_matter = 0.3 yt : [INFO ] 2021-07-27 23:55:26,191 Parameters: omega_radiation = 0.0 yt : [INFO ] 2021-07-27 23:55:26,191 Parameters: hubble_constant = 0.68 yt : [INFO ] 2021-07-27 23:55:26,550 Allocating for 1.551e+08 particles Loading particle index: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 514/514 [00:00<00:00, 6721.18it/s] yt : [INFO ] 2021-07-27 23:55:30,535 Allocating octree with spatial range [-6.9700e+01, -6.9700e+01, -6.9700e+01] code_length to [6.9700e+01, 6.9700e+01, 6.9700e+01] code_length yt : [INFO ] 2021-07-27 23:55:30,630 No particles found! Traceback (most recent call last): File "/home/chosson/local/Python-3.9.4/bin/pd_front_end.py", line 4, in import('pkg_resources').run_script('powderday==0.1.0', 'pd_front_end.py') File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/pkg_resources/init.py", line 665, in run_script self.require(requires)[0].run_script(script_name, ns) File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/pkg_resources/init.py", line 1470, in run_script exec(script_code, namespace, namespace) File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/EGG-INFO/scripts/pd_front_end.py", line 74, in File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/sph_tributary.py", line 35, in sph_m_gen File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/grid_construction.py", line 54, in yt_octree_generate File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/zoom.py", line 84, in octree_zoom_bbox_filter File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/front_ends/gadget2pd.py", line 373, in gadget_field_add File "/home/chosson/POWDERDAY/yt/yt/data_objects/static_output.py", line 1547, in addfield deps, = self.field_info.check_derived_fields([name]) File "/home/chosson/POWDERDAY/yt/yt/fields/field_info_container.py", line 613, in check_derived_fields fd = fi.get_dependencies(ds=self.ds) File "/home/chosson/POWDERDAY/yt/yt/fields/derived_field.py", line 253, in get_dependencies e[self.name] File "/home/chosson/POWDERDAY/yt/yt/fields/field_detector.py", line 125, in missing vv = finfo(self) File "/home/chosson/POWDERDAY/yt/yt/fields/derived_field.py", line 294, in call dd = self._function(self, data) File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/front_ends/gadget2pd.py", line 132, in _metalsmoothedmasses File "/home/chosson/POWDERDAY/yt/yt/data_objects/data_containers.py", line 258, in getitem self.get_data(f) File "/home/chosson/POWDERDAY/yt/yt/data_objects/construction_data_containers.py", line 2831, in get_data self._scatter_smooth(fields, units, normalize) File "/home/chosson/POWDERDAY/yt/yt/data_objects/construction_data_containers.py", line 2873, in _scatter_smooth buff = np.zeros(self.tree.num_nodes, dtype="float64") AttributeError: 'NoneType' object has no attribute 'num_nodes'

dnarayanan commented 3 years ago

Hi,

Just to be sure can you please post the hash of powderday (assuming you're on the 'master' branch?) and the yt hash you're on?

On Tue, Jul 27, 2021 at 11:01 AM ACDylan @.***> wrote:

At first, I am using "gizmo_mw_zoom example" to check that Powderday works well. Then I will use my own 'isolated galaxy's simulation' yes. With your parameters, I have now:

[octree zoom_bbox_filter:] Calculating Center of Mass [octree zoom_bbox_filter:] Center of Mass is at coordinates (kpc): [unyt_quantity(1.24139586e+26, 'cm'), unyt_quantity(9.81877541e+25, 'cm'), unyt_quantity(1.26603195e+26, 'cm')] [octree zoom_bbox_filter:] using center: [0.0, 0.0, 0.0] [octree zoom] new zoomed bbox (comoving/h) in code units= [[-69.6999972142354, 69.6999972142354], [-69.6999972142354, 69.6999972142354], [-69.6999972142354, 69.6999972142354]] yt : [INFO ] 2021-07-27 23:55:26,089 Calculating time from 9.756e-01 to be 4.263e+17 seconds yt : [INFO ] 2021-07-27 23:55:26,090 Assuming length units are in kpc/h (comoving) yt : [INFO ] 2021-07-27 23:55:26,189 Parameters: current_time = 4.263438511011126e+17 s yt : [INFO ] 2021-07-27 23:55:26,190 Parameters: domain_dimensions = [1 1 1] yt : [INFO ] 2021-07-27 23:55:26,190 Parameters: domain_left_edge = [0. 0. 0.] yt : [INFO ] 2021-07-27 23:55:26,190 Parameters: domain_right_edge = [50000. 50000. 50000.] yt : [INFO ] 2021-07-27 23:55:26,190 Parameters: cosmological_simulation = 1 yt : [INFO ] 2021-07-27 23:55:26,191 Parameters: current_redshift = 0.02499995903287333 yt : [INFO ] 2021-07-27 23:55:26,191 Parameters: omega_lambda = 0.7 yt : [INFO ] 2021-07-27 23:55:26,191 Parameters: omega_matter = 0.3 yt : [INFO ] 2021-07-27 23:55:26,191 Parameters: omega_radiation = 0.0 yt : [INFO ] 2021-07-27 23:55:26,191 Parameters: hubble_constant = 0.68 yt : [INFO ] 2021-07-27 23:55:26,550 Allocating for 1.551e+08 particles Loading particle index: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 514/514 [00:00<00:00, 6721.18it/s] yt : [INFO ] 2021-07-27 23:55:30,535 Allocating octree with spatial range [-6.9700e+01, -6.9700e+01, -6.9700e+01] code_length to [6.9700e+01, 6.9700e+01, 6.9700e+01] code_length yt : [INFO ] 2021-07-27 23:55:30,630 No particles found! Traceback (most recent call last): File "/home/chosson/local/Python-3.9.4/bin/pd_front_end.py", line 4, in import('pkg_resources').run_script('powderday==0.1.0', 'pd_front_end.py') File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/pkg_resources/ init.py", line 665, in run_script self.require(requires)[0].run_script(script_name, ns) File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/pkg_resources/ init.py", line 1470, in run_script exec(script_code, namespace, namespace) File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/EGG-INFO/scripts/pd_front_end.py", line 74, in File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/sph_tributary.py", line 35, in sph_m_gen File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/grid_construction.py", line 54, in yt_octree_generate File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/zoom.py", line 84, in octree_zoom_bbox_filter File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/front_ends/gadget2pd.py", line 373, in gadget_field_add File "/home/chosson/POWDERDAY/yt/yt/data_objects/static_output.py", line 1547, in addfield deps, = self.field_info.check_derived_fields([name]) File "/home/chosson/POWDERDAY/yt/yt/fields/field_info_container.py", line 613, in check_derived_fields fd = fi.get_dependencies(ds=self.ds) File "/home/chosson/POWDERDAY/yt/yt/fields/derived_field.py", line 253, in get_dependencies e[self.name] File "/home/chosson/POWDERDAY/yt/yt/fields/field_detector.py", line 125, in missing vv = finfo(self) File "/home/chosson/POWDERDAY/yt/yt/fields/derived_field.py", line 294, in call dd = self._function(self, data) File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/front_ends/gadget2pd.py", line 132, in _metalsmoothedmasses File "/home/chosson/POWDERDAY/yt/yt/data_objects/data_containers.py", line 258, in getitem self.get_data(f) File "/home/chosson/POWDERDAY/yt/yt/data_objects/construction_data_containers.py", line 2831, in get_data self._scatter_smooth(fields, units, normalize) File "/home/chosson/POWDERDAY/yt/yt/data_objects/construction_data_containers.py", line 2873, in _scatter_smooth buff = np.zeros(self.tree.num_nodes, dtype="float64") AttributeError: 'NoneType' object has no attribute 'num_nodes'

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dnarayanan/powderday/issues/135#issuecomment-887587907, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCBCW7UCM7HXTSU7HTNTZ3TZ3C6FANCNFSM5BAQPN5Q .

ACDylan commented 3 years ago

Yes, I am on the 'master' branch for Powderday:

commit c30f1a998a427f8d46165a6fc7d0a6236d5160c5 Author: dnarayanan desika.narayanan@gmail.com Date: Wed Jul 21 13:40:00 2021 -0400 fixing a bug introduced in commit 224e78a72d75f597f32b52dae00481bd583b0ffc

for yt: commit ba44836cfb6171e843458a1f66b54000875222d3 Merge: dd3f795 e2fc73b Author: Clément Robert cr52@protonmail.com Date: Wed Jul 21 10:17:43 2021 +0200

Merge pull request #3445 from yt-project/amrex_dim_fix Fix for 2D AMReX datasets in which 3 values are in the amr.n_cell entry in the job_info file.

dnarayanan commented 3 years ago

hmm I was able to just successfully run the gizmo_mw_zoom run.

yt hash: f580385c595a941f07754de5103905c2149dacb2 [this is the latest pull as of an hour ago, on the 'main' branch] pd hash: c30f1a998a427f8d46165a6fc7d0a6236d5160c5

my parameters_master file: http://paste.yt-project.org/show/340/ my parameters_model file: http://paste.yt-project.org/show/341/

can you try with those hashes, those parameters files (aside from your local paths) just to see ?

ACDylan commented 3 years ago

It is working better, I have edited both parameters files and get the last version of yt, but now it is saying

Setting Octree Grid with Parameters: INFO: Checking consistency of refined array [hyperion.grid.octree_grid] INFO: Setting refined with maximum depth of 14 levels [hyperion.grid.octree_grid] WARNING: AstropyDeprecationWarning: BlackBody provides the same capabilities [powderday.helpers] /home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/astropy-4.2.1-py3.9-linux-x86_64.egg/astropy/units/quantity.py:479: RuntimeWarning: overflow encountered in expm1 result = super().array_ufunc__(function, method, *arrays, **kwargs) /home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/astropy-4.2.1-py3.9-linux-x86_64.egg/astropy/units/quantity.py:479: RuntimeWarning: overflow encountered in multiply result = super().array_ufunc__(function, method, *arrays, **kwargs) SPS_SETUP ERROR: error opening dust emission file

which is surprising because I have downloaded 'hyperion-dust-0.1.0': dustdir = '/home/chosson/POWDERDAY/hyperion-dust-0.1.0/dust_files/' dustfile = 'd03_3.1_6.0_A.hdf5'

I have tried to download it again or change location but always this issue. SPS is related with fsps, and the command 'sp = fsps.StellarPopulation' seems to not work.

dnarayanan commented 3 years ago

Hi - I'm wondering if this issue is with fsps as you suggest. Can you please try the following:

within python, what does this return:

import fsps
sp = fsps.StellarPopulation

also, within a linux terminal shell, what does this return:

echo $SPS_HOME

?

ACDylan commented 3 years ago

I was already working on it and it works now!

The issue now is: assigning stars to SED bins Running SPS for Binned SEDs calculating the SEDs for 845 bins /home/chosson/local/Python-3.9.4/bin/hyperion:69: DeprecationWarning: The SafeConfigParser class has been renamed to ConfigParser in Python 3.2. This alias will be removed in future versions. Use ConfigParser directly instead. config = SafeConfigParser()

mpirun was unable to find the specified executable file, and therefore did not launch the job. This error was first reported for process rank 0; it may have occurred for other processes as well.

NOTE: A common cause for this error is misspelling a mpirun command line parameter option (remember that mpirun interprets the first unrecognized command line token as the executable).

Node: virgo01 Executable: hyperion_oct_mpi

Run did not complete successfully: output file appears to be corrupt delta_chunk_indices = 845 Entering Pool.map multiprocessing for Stellar SED generation Execution time for SED generation in Pool.map multiprocessing = 0:01:23.727736 [SED_gen: ] total_lum_in_sed_gen = 98878.02028815863 adding point source collections [source_creation/add_binned_seds:] Execution time for point source collection adding = 0:00:20.035827 [source_creation/add_binned_seds:] Total Luminosity of point source collection is: 3.8752917290525584e+44 Done adding Sources Setting up Model [pd_front_end]: Beginning RT Stage: Calculating SED using a binned spectrum An error occurred, and the run did not complete

dnarayanan commented 3 years ago

hmmm - yeah it can be a bit tricky to get the MPI set up to work with the wide ranging options out there.

my first step to suggest would be to ensure that it works in command line without mpi -- i.e. a simple:

pd_front_end.py directory parameters_master parameters_model

if this works, maybe then we can explore the mpi capabilities? (I'll note that without the need for very high resolution imaging, most of the time the MPI is a bit overkill since it only impacts the radiative transfer itself, which doesn't take that long in itself for simple SED calculations)

dnarayanan commented 3 years ago
I have reinstalled hyperion because the function 'hyperion_sph' was not working.
But now, I have

Traceback (most recent call last):
File "/home/chosson/local/Python-3.9.4/bin/pd_front_end.py", line 4, in 
import('pkg_resources').run_script('powderday==0.1.0', 'pd_front_end.py')
File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/pkg_resources/init.py", line 665, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/pkg_resources/init.py", line 1470, in run_script
exec(script_code, namespace, namespace)
File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/EGG-INFO/scripts/pd_front_end.py", line 8, in 
File "", line 259, in load_module
File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/init.py", line 25, in 
File "", line 259, in load_module
File "/home/chosson/local/Python-3.9.4/lib/python3.9/site-packages/powderday-0.1.0-py3.9.egg/powderday/source_creation.py", line 13, in 
ModuleNotFoundError: No module named 'hyperion.grid.yt3_wrappers'

sorry about the delay - I'm on a trip for a few days so may be somewhat delayed In responding.

are you still having trouble with Hyperion looking for the yt3 wrpapers/? you might need to re-install yt since when you re-installed Hyperion, this might have clobbered your yt4.x installation ( if you type yt.version_version() it will reveal what yt version you're running)

ACDylan commented 3 years ago

No problem! Yes, I have reinstalled everything and I am currently running a simulation. I will tell you if everything goes well. I have somewhat found other issues like the new one I have opened here: https://github.com/dnarayanan/powderday/issues/136

I have also used this close issue : https://github.com/dnarayanan/powderday/issues/34 Because I also met the astropy.extern.six issue.

dnarayanan commented 3 years ago

okay good! sorry for your troubles so far but glad it's working. I'll try to see if I can reproduce #136 though ..currently I'm unable to reproduce it.

ACDylan commented 3 years ago

[main] exiting raytracing iteration [image_write] writing out SEDs [image_write] done

Total CPU time elapsed: 631.28 Ended on 31 July 2021 at 02:51:17

It is working now! Thank you. This issue can be closed. For the archive, if someone met the same issues as me, here are some advices: