Closed Christophe-pere closed 1 year ago
Christophe-pere
commented
3 years ago After investigation, I found that the problem isn't located at the place described below. But, in the _run_full_retrieval function in the file retriever.py which call the function save_results function located in the output_handler.py file. I obtained this error:
Exception raised whilst saving results:
index 1 is out of bounds for axis 0 with size 1
Unable to save results
joshjchayes
commented
3 years ago Hi Cristophe
Thanks for highlighting this issue.
Can you let me know what version of transitfit you are using? Run transitfit.__version__ to see. Have you installed this from PyPi or direct from the GitHub?
Christophe-pere
commented
3 years ago Hi,
I'm currently using version 2.2.4 downloaded and installed from PyPI via pip install.
joshjchayes
commented
3 years ago Hi Cristophe,
Thanks, I'm on it.
While I'm tracing this back, can I suggest that you try including fitting_mode='batched' into your run_retrieval call? This should get around the issue before I get a fix out.
Christophe-pere
commented
3 years ago Hi, I tried your solution but in fact I obtain this error:
Total 12 fitting parameters
10357it [03:07, 55.35it/s, +300 | bound: 111 | nc: 1 | ncall: 564377 | eff(%): 1.888 | loglstar: -inf < -66.228 < inf | logz: -99.528 +/- 0.463 | dlogz: 0.001 > 0.309]
Extracting results dict
Calculating best values for this run
Combining results dicts
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-14-e0bb310848d9> in <module>
----> 1 results = run_retrieval('input_data.csv', 'priors.csv', 'filter_profiles.csv', plot=True, fitting_mode='batched')
~/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/_pipeline.py in run_retrieval(data_files, priors, filter_info, detrending_list, limb_darkening_model, ld_fit_method, fitting_mode, max_batch_parameters, batch_overlap, host_T, host_logg, host_z, host_r, nlive, dlogz, maxiter, maxcall, dynesty_sample, dynesty_bounding, normalise, detrend, results_output_folder, final_lightcurve_folder, summary_file, full_output_file, plot_folder, plot, marker_color, line_color, ldtk_cache, n_ld_samples, do_ld_mc, data_skiprows, allow_ttv, filter_delimiter, detrending_limits, bin_data, cadence, binned_color, walks, slices)
318
319 # Run the retrieval!
--> 320 results = retriever.run_retrieval(ld_fit_method, fitting_mode,
321 max_batch_parameters, maxiter, maxcall,
322 dynesty_sample, nlive, dlogz, plot,
~/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/retriever.py in run_retrieval(self, ld_fit_method, fitting_mode, max_parameters, maxiter, maxcall, sample, nlive, dlogz, plot, output_folder, lightcurve_folder, summary_file, full_output_file, plot_folder, marker_color, line_color, bound, normalise, detrend, overlap, bin_data, cadence, binned_color, walks, slices)
630 batches = self._get_non_folding_batches(self.all_lightcurves, max_parameters, detrend, normalise, overlap)
631
--> 632 results = self._run_batched_retrieval(self.all_lightcurves, batches, ld_fit_method,detrend,
633 normalise, maxiter, maxcall, sample, nlive, dlogz, False,
634 False, None, None, output_folder, summary_file,
~/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/retriever.py in _run_batched_retrieval(self, lightcurves, batches, ld_fit_method, detrend, normalise, maxiter, maxcall, sample, nlive, dlogz, full_return, folded, folded_P, folded_t0, output_folder, summary_file, full_output_file, lightcurve_folder, plot, plot_folder, marker_color, line_color, bound, filter_idx, walks, slices)
407 all_lightcurves.append(batch_lightcurves)
408
--> 409 output_handler._quicksave_result(results, batch_prior, batch_lightcurves, output_folder, filter_idx, bi)
410
411 # Make outputs etc
~/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/output_handler.py in _quicksave_result(self, results, priors, lightcurves, base_output_path, filter, batch)
676 '''
677 result_dict = self.get_results_dict(results, priors, lightcurves)
--> 678 result_dict, _ = self.get_best_vals([result_dict], priors.fit_ld)
679 base_fname = ''
680 if filter is not None:
~/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/output_handler.py in get_best_vals(self, results_dicts, priors, fit_ld, return_combined)
439 # Limb darkening bits
440 if fit_ld:
--> 441 best_vals, combined_dict = self.add_best_u(best_vals, combined_dict)
442 if return_combined:
443 return best_vals, combined_dict
~/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/output_handler.py in add_best_u(self, best_dict, combined_dict)
528 for b in range(n_batches):
529 for qi, q in enumerate(self.ld_coeffs):
--> 530 filter_q[b,0,qi] = combined_dict[q][i][b][0]
531 filter_q[b,1,qi] = combined_dict[q][i][b][-1]
532
IndexError: index 1 is out of bounds for axis 0 with size 1Hi Christophe,
I've identified the bug and have a fix which I should be releasing with a code update either later today or tomorrow.
Christophe-pere
commented
3 years ago Hi, @joshjchayes do you have released the code?
joshjchayes
commented
3 years ago Hi @Christophe-pere, this issue should be fixed as of version 2.2.5, which is now available on PyPi.
Thanks!
Christophe-pere
commented
3 years ago Hi @joshjchayes
Thanks for the new version but... I still have the error in normal mode and with fitting_mode='batched' when using the run_retrieval function.
import transitfit
transitfit.__version__
'2.2.5'After that:
from transitfit import run_retrieval
results = run_retrieval('input_data.csv', 'priors.csv', 'filter_profiles.csv', plot=True, fitting_mode='batched')The result:
0it [00:00, ?it/s]
Batch 1 of 1
Priors:
Limb darkening model: quadratic
n telescopes: 1
n filters: 1
n epochs: 1
P: t - f - e - Gaussian - mean: 0.7 - stdev: 0.001
t0: t - f - e - Gaussian - mean: 0.0 - stdev: 0.007
a: t - f - e - Gaussian - mean: 1.1 - stdev: 0.5
rp: t - f 0 e - Uniform - min: 0.023 - max: 0.03
inc: t - f - e - Gaussian - mean: 90.0 - stdev: 1.2
ecc: t - f - e - Fixed - value: 0.0
w: t - f - e - Gaussian - mean: 90.0 - stdev: 1.2
q0: t - f 0 e - Uniform - min: 0 - max: 1
q1: t - f 0 e - Uniform - min: 0 - max: 1
norm: t 0 f 0 e 0 Uniform - min: 0.9995734727623046 - max: 1.0006452864728834
d0_0: t 0 f 0 e 0 Uniform - min: -1000 - max: 1000
Total 12 fitting parameters
10084it [03:23, 49.43it/s, +300 | bound: 106 | nc: 1 | ncall: 567381 | eff(%): 1.830 | loglstar: -inf < -58.499 < inf | logz: -90.932 +/- 0.452 | dlogz: 0.001 > 0.309]
Extracting results dict
Calculating best values for this run
Combining results dicts
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-4-e0bb310848d9> in <module>
----> 1 results = run_retrieval('input_data.csv', 'priors.csv', 'filter_profiles.csv', plot=True, fitting_mode='batched')
~/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/_pipeline.py in run_retrieval(data_files, priors, filter_info, detrending_list, limb_darkening_model, ld_fit_method, fitting_mode, max_batch_parameters, batch_overlap, host_T, host_logg, host_z, host_r, nlive, dlogz, maxiter, maxcall, dynesty_sample, dynesty_bounding, normalise, detrend, results_output_folder, final_lightcurve_folder, summary_file, full_output_file, plot_folder, plot, marker_color, line_color, ldtk_cache, n_ld_samples, do_ld_mc, data_skiprows, allow_ttv, filter_delimiter, detrending_limits, bin_data, cadence, binned_color, walks, slices)
318
319 # Run the retrieval!
--> 320 results = retriever.run_retrieval(ld_fit_method, fitting_mode,
321 max_batch_parameters, maxiter, maxcall,
322 dynesty_sample, nlive, dlogz, plot,
~/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/retriever.py in run_retrieval(self, ld_fit_method, fitting_mode, max_parameters, maxiter, maxcall, sample, nlive, dlogz, plot, output_folder, lightcurve_folder, summary_file, full_output_file, plot_folder, marker_color, line_color, bound, normalise, detrend, overlap, bin_data, cadence, binned_color, walks, slices)
628 batches = self._get_non_folding_batches(self.all_lightcurves, max_parameters, detrend, normalise, overlap)
629
--> 630 results = self._run_batched_retrieval(self.all_lightcurves, batches, ld_fit_method,detrend,
631 normalise, maxiter, maxcall, sample, nlive, dlogz, False,
632 False, None, None, output_folder, summary_file,
~/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/retriever.py in _run_batched_retrieval(self, lightcurves, batches, ld_fit_method, detrend, normalise, maxiter, maxcall, sample, nlive, dlogz, full_return, folded, folded_P, folded_t0, output_folder, summary_file, full_output_file, lightcurve_folder, plot, plot_folder, marker_color, line_color, bound, filter_idx, walks, slices)
405 all_lightcurves.append(batch_lightcurves)
406
--> 407 output_handler._quicksave_result(results, batch_prior, batch_lightcurves, output_folder, filter_idx, bi)
408
409 # Make outputs etc
~/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/output_handler.py in _quicksave_result(self, results, priors, lightcurves, base_output_path, filter, batch)
675 '''
676 result_dict = self.get_results_dict(results, priors, lightcurves)
--> 677 result_dict, _ = self.get_best_vals([result_dict], priors.fit_ld)
678 base_fname = ''
679 if filter is not None:
~/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/output_handler.py in get_best_vals(self, results_dicts, priors, fit_ld, return_combined)
438 # Limb darkening bits
439 if fit_ld:
--> 440 best_vals, combined_dict = self.add_best_u(best_vals, combined_dict)
441 if return_combined:
442 return best_vals, combined_dict
~/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/output_handler.py in add_best_u(self, best_dict, combined_dict)
527 for b in range(n_batches):
528 for qi, q in enumerate(self.ld_coeffs):
--> 529 filter_q[b,0,qi] = combined_dict[q][i][b][0]
530 filter_q[b,1,qi] = combined_dict[q][i][b][-1]
531
IndexError: index 1 is out of bounds for axis 0 with size 1Without the fitting_mode:
results = run_retrieval('input_data.csv', 'priors.csv', 'filter_profiles.csv', plot=True)The result:
133it [00:00, 1325.90it/s, bound: 0 | nc: 1 | ncall: 489 | eff(%): 27.198 | loglstar: -inf < -inf < inf | logz: -inf +/- nan | dlogz: inf > 0.309]
Auto mode detect has set 'all' mode
Priors:
Limb darkening model: quadratic
n telescopes: 1
n filters: 1
n epochs: 1
P: t - f - e - Gaussian - mean: 0.7 - stdev: 0.001
t0: t - f - e - Gaussian - mean: 0.0 - stdev: 0.007
a: t - f - e - Gaussian - mean: 1.1 - stdev: 0.5
rp: t - f 0 e - Uniform - min: 0.023 - max: 0.03
inc: t - f - e - Gaussian - mean: 90.0 - stdev: 1.2
ecc: t - f - e - Fixed - value: 0.0
w: t - f - e - Gaussian - mean: 90.0 - stdev: 1.2
q0: t - f 0 e - Uniform - min: 0 - max: 1
q1: t - f 0 e - Uniform - min: 0 - max: 1
norm: t 0 f 0 e 0 Uniform - min: 0.9995734727623046 - max: 1.0006452864728834
d0_0: t 0 f 0 e 0 Uniform - min: -1000 - max: 1000
Total 12 fitting parameters
9184it [02:49, 54.12it/s, +300 | bound: 96 | nc: 1 | ncall: 489329 | eff(%): 1.938 | loglstar: -inf < -64.242 < inf | logz: -93.582 +/- 0.436 | dlogz: 0.001 > 0.309]
Best fit results:
P: 0.699295 ± 0.000955
t0: 0.003133 ± 0.002131
a: 1.040423 ± 0.02331
inc: 91.43354 ± 1.187391
w: 89.945509 ± 0.424187
rp_0: 0.024026 ± 0.000172
q0_0: 0.6428194541139004 ± 0.07900176321237204
q0_0: 0.04003098812817056 ± 0.1744323109135007
q0_0: 0.0616044607761707 ± 0.009944088876804502
q1_0: 0.050051337278037886 ± 0.02233275967089502
d0_0: 1.1e-05 ± 4.5e-05
norm_t0_f0_e0: 0.999959 ± 2.5e-05
chi2: 64.24202
red chi2: 1.14718
Saving full results...
Extracting results dict
Plotting batch 0 samples to ./plots/unfolded/batch_0_samples.png
WARNING:root:Too few points to create valid contours
WARNING:root:Too few points to create valid contours
WARNING:root:Too few points to create valid contours
WARNING:root:Too few points to create valid contours
WARNING:root:Too few points to create valid contours
Calculating best values for this run
Combining results dicts
Exception raised whilst saving results:
index 1 is out of bounds for axis 0 with size 1
Unable to save results
Initialising best fit model
File "/Users/gen06846/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/retriever.py", line 349, in _run_full_retrieval
output_handler.save_results([results], [priors], [lightcurves],
File "/Users/gen06846/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/output_handler.py", line 326, in save_results
best_vals, combined_results = self.get_best_vals(results_dicts, fit_ld)
File "/Users/gen06846/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/output_handler.py", line 440, in get_best_vals
best_vals, combined_dict = self.add_best_u(best_vals, combined_dict)
File "/Users/gen06846/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/output_handler.py", line 529, in add_best_u
filter_q[b,0,qi] = combined_dict[q][i][b][0]
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
<ipython-input-5-7a751e249084> in <module>
----> 1 results = run_retrieval('input_data.csv', 'priors.csv', 'filter_profiles.csv', plot=True)#, fitting_mode='batched')
~/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/_pipeline.py in run_retrieval(data_files, priors, filter_info, detrending_list, limb_darkening_model, ld_fit_method, fitting_mode, max_batch_parameters, batch_overlap, host_T, host_logg, host_z, host_r, nlive, dlogz, maxiter, maxcall, dynesty_sample, dynesty_bounding, normalise, detrend, results_output_folder, final_lightcurve_folder, summary_file, full_output_file, plot_folder, plot, marker_color, line_color, ldtk_cache, n_ld_samples, do_ld_mc, data_skiprows, allow_ttv, filter_delimiter, detrending_limits, bin_data, cadence, binned_color, walks, slices)
318
319 # Run the retrieval!
--> 320 results = retriever.run_retrieval(ld_fit_method, fitting_mode,
321 max_batch_parameters, maxiter, maxcall,
322 dynesty_sample, nlive, dlogz, plot,
~/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/retriever.py in run_retrieval(self, ld_fit_method, fitting_mode, max_parameters, maxiter, maxcall, sample, nlive, dlogz, plot, output_folder, lightcurve_folder, summary_file, full_output_file, plot_folder, marker_color, line_color, bound, normalise, detrend, overlap, bin_data, cadence, binned_color, walks, slices)
646 output_handler = OutputHandler(self.all_lightcurves, self._full_prior, self.host_r)
647
--> 648 output_handler.save_complete_results(fitting_mode, self._full_prior, output_folder, summary_file)
649
650 output_handler.save_final_light_curves(self.all_lightcurves, self._full_prior, lightcurve_folder)
~/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/output_handler.py in save_complete_results(self, mode, global_prior, output_folder, summary_file)
261 Once all batches etc are run, collates all results and saves to csv
262 '''
--> 263 _ = self._initialise_best_model(mode, global_prior, output_folder, summary_file)
264
265 print('Saving final results')
~/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/output_handler.py in _initialise_best_model(self, mode, global_prior, output_folder, summary_file)
592
593 # First use the top-level output
--> 594 top_output = pd.read_csv(os.path.join(output_folder, summary_file))
595
596 for i, row in top_output.iterrows():
~/anaconda3/envs/QML/lib/python3.9/site-packages/pandas/io/parsers.py in read_csv(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, dialect, error_bad_lines, warn_bad_lines, delim_whitespace, low_memory, memory_map, float_precision, storage_options)
608 kwds.update(kwds_defaults)
609
--> 610 return _read(filepath_or_buffer, kwds)
611
612
~/anaconda3/envs/QML/lib/python3.9/site-packages/pandas/io/parsers.py in _read(filepath_or_buffer, kwds)
460
461 # Create the parser.
--> 462 parser = TextFileReader(filepath_or_buffer, **kwds)
463
464 if chunksize or iterator:
~/anaconda3/envs/QML/lib/python3.9/site-packages/pandas/io/parsers.py in __init__(self, f, engine, **kwds)
817 self.options["has_index_names"] = kwds["has_index_names"]
818
--> 819 self._engine = self._make_engine(self.engine)
820
821 def close(self):
~/anaconda3/envs/QML/lib/python3.9/site-packages/pandas/io/parsers.py in _make_engine(self, engine)
1048 )
1049 # error: Too many arguments for "ParserBase"
-> 1050 return mapping[engine](self.f, **self.options) # type: ignore[call-arg]
1051
1052 def _failover_to_python(self):
~/anaconda3/envs/QML/lib/python3.9/site-packages/pandas/io/parsers.py in __init__(self, src, **kwds)
1865
1866 # open handles
-> 1867 self._open_handles(src, kwds)
1868 assert self.handles is not None
1869 for key in ("storage_options", "encoding", "memory_map", "compression"):
~/anaconda3/envs/QML/lib/python3.9/site-packages/pandas/io/parsers.py in _open_handles(self, src, kwds)
1360 Let the readers open IOHanldes after they are done with their potential raises.
1361 """
-> 1362 self.handles = get_handle(
1363 src,
1364 "r",
~/anaconda3/envs/QML/lib/python3.9/site-packages/pandas/io/common.py in get_handle(path_or_buf, mode, encoding, compression, memory_map, is_text, errors, storage_options)
640 errors = "replace"
641 # Encoding
--> 642 handle = open(
643 handle,
644 ioargs.mode,
FileNotFoundError: [Errno 2] No such file or directory: './output_parameters/summary_output.csv'As previously describe I have the same errors:
Exception raised whilst saving results:
index 1 is out of bounds for axis 0 with size 1
Unable to save resultsCould you reopen the issue, please?
Regards,
Christophe
joshjchayes
commented
3 years ago Hi @Christophe-pere
That's very odd, because I can't seem to replicate the issue. Can you provide a copy of your input files? I'm wondering if that's the issue.
I will carry on having a look at this, but I'm working on a fairly major update which should fix any issues in the places you're seeing them. That version should be coming to pip soon, but in the meantime, if you want to try the version on the multiprocessing_dev branch (v2.3.0dev) then that might help
Josh
joshjchayes
commented
3 years ago Hi @Christophe-pere
Version 2.3.0 is now on PyPi and should have fixed your issues. Can you try it and let me know please?
Josh
Christophe-pere
commented
3 years ago Hi @joshjchayes,
Sorry for the delay. Try the same with the 2.3 version and still got the error.
You can find the files and the notebook here: transitfit
Chris
joshjchayes
commented
3 years ago Hi @Christophe-pere
Thanks for sharing the files, I've managed to replicate the issue and have spotted the cause now. I have released version 2.3.1 with fixes.
As an aside, you should remove the q0 values from your prior files. You're setting the prior for the same filter twice. Also, in general, you don't need to set the LDC priors in the priors file, as TransitFit will do that for you (using the Kipping parameterisation). I did think that setting them explicitly in the priors file would override that, but something is stopping that happening. I will investigate that but, for now, I would suggest removing all references to q0, q1 etc from your priors file and just let TransitFit do its thing with them.
Thanks for your time in helping fix this!
Josh
Christophe-pere
commented
3 years ago Hi Josh,
Thanks for your reply. Sorry for my mistakes.
I tried without q0 parameter inside the error disappeared but, I see what seems to be a little bug?
Exception raised whilst saving results:
local variable 'a_AU' referenced before assignment
Unable to save results
Initialising best fit modelI also have this warning at the end:
Plotting final curves
File "/Users/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/retriever.py", line 360, in _run_full_retrieval
output_handler.save_results([results], [priors], [lightcurves],
File "/Users/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/output_handler.py", line 337, in save_results
self._save_results_dict(combined_results, os.path.join(output_folder, full_output_file), True)
File "/Users/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/output_handler.py", line 756, in _save_results_dict
df = self._results_dict_to_dataframe(results_dict, batched)
File "/Users/anaconda3/envs/QML/lib/python3.9/site-packages/transitfit/output_handler.py", line 954, in _results_dict_to_dataframe
vals_arr = np.append(vals_arr, np.array([['a/AU', tidx, fidx, eidx, a_AU, a_AU_err]]), axis=0)Thanks for your time,
Chris
Hi there,
I try to fit a lightcurve with Transitfit. For the moment I just want to test the library in basic mode.
I have the different files needed for this. When I run:
results = run_retrieval('input_data.csv', 'priors.csv', 'filter_profiles.csv')The model fit the different parameters but, when it's time to generate the output I obtain this:
I just use the parameters by default so... How can I fix this?
Best,
C.