search

aabdurrouf / piXedfit

piXedfit is a Python package designed for analyzing spatially resolved SEDs of galaxies
https://pixedfit.readthedocs.io/en/latest/index.html
MIT License
40 stars 7 forks source link

About SED fitting (based on bin map and pixel-by-pixel) #9

Closed markliuchina closed 1 year ago

markliuchina commented 1 year ago

Hi, Abdurro'uf! First of all, thank you for making these wonderful code available! We are now working on performing pixel-by-pixel spatially resolved analysis on JWST data. Fortunately, we find that your have already applied spatially resolved analysis on JWST observation and the result is interesting and inspiring! After reading most of your recent works which involve spatially resolved SED fitting, I have been trying to use your code and find the 'image_process' and 'pixel_binning' modules are quite satisfying and convenient! I follow the instructions of the example tutorial of fitting NGC-5194 carefully and I am confident that the SFH model and flux_map are generated properly. But the SED fitting have run for 26 hours till now and it does not seem to come to an end. It is just repeating the same information in the terminal: image It seems like the programme cannot find a hdf5 file in the '/data/temp/' directory. (But, with this error message, the code can still be running) Could you please tell me how to fix this or I just need to ignore it and let the machine running? By the way, I was wondering whether it is caused by my machine. (I am using a DELL workstation with a intel i5-12500 CPU and 32G RAM) In the same time, I was trying to do the SED fitting with Bagpipes but the SPS method is different from your code and my laptop run slowly and still cannot give me a result. Thanks in advance! Best.

Mingfeng Liu Nanjing Normal University

markliuchina commented 1 year ago

The image cannot show well, I copy the information of the terminal here:

Number of models for initial fitting: 100000
rank 5 --> progress: 334 of 10000 (3%)[I 19:32:12.616 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 5 --> progress: 1092 of 10000 (10%)[I 19:34:12.903 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 6 --> progress: 1928 of 10000 (19%)[I 19:36:13.632 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 9 --> progress: 2728 of 10000 (27%)[I 19:38:13.618 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 4 --> progress: 3254 of 10000 (32%)[I 19:40:13.632 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 0 --> progress: 4812 of 10000 (48%)[I 19:42:13.625 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 7 --> progress: 5077 of 10000 (50%)[I 19:44:13.627 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 4 --> progress: 6073 of 10000 (60%)[I 19:46:13.619 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 1 --> progress: 6784 of 10000 (67%)[I 19:48:14.203 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 8 --> progress: 8115 of 10000 (81%)[I 19:50:14.765 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 3 --> progress: 8505 of 10000 (85%)[I 19:52:15.611 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 6 --> progress: 9590 of 10000 (95%)[I 19:54:16.390 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 9 --> progress: 9994 of 10000 (99%)[I 19:56:16.741 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 9 --> progress: 10000 of 10000 (100%)
Traceback (most recent call last):
  File "/home/ubuntu/lib/piXedfit/piXedfit/piXedfit_fitting/./mc_p1.py", line 629, in <module>
    sampler.run_mcmc(pos, nsteps, progress=True)
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/emcee/sampler.py", line 171, in run_mcmc
    for results in self.sample(pos0, lnprob0, rstate0, iterations=N,
TypeError: sample() got an unexpected keyword argument 'progress'
mv: cannot stat 'samplers_43375.hdf5': No such file or directory
Traceback (most recent call last):
  File "/home/ubuntu/lib/piXedfit/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 533, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 226, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/ubuntu/lib/piXedfit/data/temp/samplers_43375.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/ubuntu/lib/piXedfit/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 533, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 226, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/ubuntu/lib/piXedfit/data/temp/samplers_43375.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/ubuntu/lib/piXedfit/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 533, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 226, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/ubuntu/lib/piXedfit/data/temp/samplers_43375.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/ubuntu/lib/piXedfit/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 533, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 226, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/ubuntu/lib/piXedfit/data/temp/samplers_43375.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/ubuntu/lib/piXedfit/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 533, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 226, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/ubuntu/lib/piXedfit/data/temp/samplers_43375.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/ubuntu/lib/piXedfit/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 533, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 226, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/ubuntu/lib/piXedfit/data/temp/samplers_43375.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
Traceback (most recent call last):
  File "/home/ubuntu/lib/piXedfit/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
Traceback (most recent call last):
  File "/home/ubuntu/lib/piXedfit/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
Traceback (most recent call last):
  File "/home/ubuntu/lib/piXedfit/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
  File "/home/ubuntu/lib/piXedfit/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 533, in __init__
    f = h5py.File(name_file, 'r')
    f = h5py.File(name_file, 'r')
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 533, in __init__
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 533, in __init__
    f = h5py.File(name_file, 'r')
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 533, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 226, in make_fid
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 226, in make_fid
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 226, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 226, in make_fid
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/ubuntu/lib/piXedfit/data/temp/samplers_43375.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/ubuntu/lib/piXedfit/data/temp/samplers_43375.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/ubuntu/lib/piXedfit/data/temp/samplers_43375.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/ubuntu/lib/piXedfit/data/temp/samplers_43375.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
rm: cannot remove '/home/ubuntu/lib/piXedfit/data/temp/samplers_43375.hdf5': No such file or directory
rank 8 --> progress: 1 of 10000 (0%)Number of free parameters: 14
Free parameters: 
['logzsol', 'log_tau', 'log_alpha', 'log_beta', 'log_age', 'dust_index', 'dust1', 'dust2', 'log_gamma', 'log_umin', 'log_qpah', 'log_fagn', 'log_tauagn', 'log_mass']
Number of fix parameters: 0
Number of parent models in models_spec: 100000
Number of models for initial fitting: 100000
rank 9 --> progress: 553 of 10000 (5%)[I 19:58:17.584 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 8 --> progress: 1219 of 10000 (12%)[I 20:00:17.875 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 6 --> progress: 1928 of 10000 (19%)[I 20:02:18.497 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 4 --> progress: 2841 of 10000 (28%)[I 20:04:18.700 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 9 --> progress: 3568 of 10000 (35%)[I 20:06:18.925 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 6 --> progress: 4235 of 10000 (42%)[I 20:08:18.986 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 0 --> progress: 5497 of 10000 (54%)[I 20:10:19.878 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 2 --> progress: 6442 of 10000 (64%)[I 20:12:20.679 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 5 --> progress: 7012 of 10000 (70%)[I 20:14:21.634 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 1 --> progress: 8406 of 10000 (84%)[I 20:16:21.703 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 0 --> progress: 8867 of 10000 (88%)[I 20:18:22.302 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 7 --> progress: 9342 of 10000 (93%)[I 20:20:23.255 NotebookApp] Saving file at /Python Codes/pixel_fitting/3_SED.ipynb
rank 6 --> progress: 10000 of 10000 (100%)
markliuchina commented 1 year ago

Sorry！ I now find the problem. Cause I use Lenstools this day, and this package require a emcee version < 3. The older version of this package seems do not support mpi running for sampling stage and many functions have different augments and keywords compared with the new version. After upgrading the emcee, the fitting can run suceessivefuly! (the results are quite satisfying!) Thanks you again for providing this code. : ) Now I will close this issue.

aabdurrouf commented 1 year ago

Thanks for the information and feedback! I am sorry that I completely missed reading your message earlier.