mboquien / cigale

10 stars 0 forks source link

Issue with chunksize or ncores? #68

Open brentmsmith opened 9 months ago

brentmsmith commented 9 months ago

I'm not sure why, but I keep getting this error: IndexError: index -501 is out of bounds for axis 0 with size 500

I'm using 128 cores, 128 chunks.

Here's the full traceback:

`╭───────────────────── Traceback (most recent call last) ─────────────────────╮

│ /home/user/.local/bin/pcigale:11 in │ │ │ │ 8 if name == 'main': │ │ 9 │ sys.argv[0] = re.sub(r'(-script.pyw?|.exe)?$', '', sys.argv[0]) │ │ 10 │ sys.exit( │ │ ❱ 11 │ │ load_entry_point('pcigale==2022.1', 'console_scripts', 'pcigale │ │ 12 │ ) │ │ 13 │ │ │ │ /home/user/.local/lib/python3.6/site-packages/pcigale/init.py:139 in │ │ main │ │ │ │ 136 │ │ elif args.parser == "check": │ │ 137 │ │ │ check(config) │ │ 138 │ │ elif args.parser == "run": │ │ ❱ 139 │ │ │ run(config) │ │ 140 │ │ │ │ /home/user/.local/lib/python3.6/site-packages/pcigale/init.py:71 in │ │ run │ │ │ │ 68 │ │ console.print(f"{INFO} Start: {start.isoformat('/', 'seconds') │ │ 69 │ │ start = time.monotonic() # Simpler time for run duration │ │ 70 │ │ │ │ ❱ 71 │ │ analysis_module.process(configuration) │ │ 72 │ │ │ │ 73 │ │ end = dt.datetime.now() │ │ 74 │ │ console.print(f"{INFO} End: {end.isoformat('/', 'seconds')}") │ │ │ │ /home/user/.local/lib/python3.6/site-packages/pcigale/analysis_modules/p │ │ df_analysis/init.py:235 in process │ │ │ │ 232 │ │ obs = ObservationsManager(conf, params) │ │ 233 │ │ obs.save("observations") │ │ 234 │ │ │ │ ❱ 235 │ │ results = self._compute(conf, obs, params) │ │ 236 │ │ console.print(f"{INFO} Sanity check of the analysis results.") │ │ 237 │ │ results.best.analyse_chi2() │ │ 238 │ │ │ │ /home/user/.local/lib/python3.6/site-packages/pcigale/analysis_modules/p │ │ df_analysis/init.py:182 in _compute │ │ │ │ 179 │ │ │ # avoid recomputing the models when we do a mock analysis │ │ 180 │ │ │ if not hasattr(self, "_models"): │ │ 181 │ │ │ │ console.print(f"{INFO} Computing models.") │ │ ❱ 182 │ │ │ │ models = self._compute_models(conf, obs, params, ibloc │ │ 183 │ │ │ │ if nblocks == 1: │ │ 184 │ │ │ │ │ self._models = models │ │ 185 │ │ │ else: │ │ │ │ /home/user/.local/lib/python3.6/site-packages/pcigale/analysis_modules/p │ │ df_analysis/init.py:113 in _compute_models │ │ │ │ 110 │ │ │ params.blocks[iblock], │ │ 111 │ │ │ initargs, │ │ 112 │ │ │ init_worker_sed, │ │ ❱ 113 │ │ │ conf["cores"] │ │ 114 │ │ ) │ │ 115 │ │ │ │ 116 │ │ # Print the final value as it may not otherwise be printed │ │ │ │ /home/user/.local/lib/python3.6/site-packages/pcigale/analysis_modules/p │ │ df_analysis/init.py:168 in _parallel_job │ │ │ │ 165 │ │ │ with mp.Pool( │ │ 166 │ │ │ │ processes=ncores, initializer=initializer, initargs=in │ │ 167 │ │ │ ) as pool: │ │ ❱ 168 │ │ │ │ pool.starmap(worker, enumerate(items), chunksize) │ │ 169 │ │ │ │ │ 170 │ │ │ # After the parallel processes have exited, it can be rest │ │ 171 │ │ │ counter.progress = progress │ │ │ │ /usr/lib64/python3.6/multiprocessing/pool.py:274 in starmap │ │ │ │ 271 │ │ be iterables as well and will be unpacked as arguments. Hence │ │ 272 │ │ func and (a, b) becomes func(a, b). │ │ 273 │ │ ''' │ │ ❱ 274 │ │ return self._map_async(func, iterable, starmapstar, chunksize) │ │ 275 │ │ │ 276 │ def starmap_async(self, func, iterable, chunksize=None, callback=N │ │ 277 │ │ │ error_callback=None): │ │ │ │ /usr/lib64/python3.6/multiprocessing/pool.py:644 in get │ │ │ │ 641 │ │ if self._success: │ │ 642 │ │ │ return self._value │ │ 643 │ │ else: │ │ ❱ 644 │ │ │ raise self._value │ │ 645 │ │ │ 646 │ def _set(self, i, obj): │ │ 647 │ │ self._success, self._value = obj │ ╰──────────────────────────────────────────────────────────────────────────────╯ IndexError: index -501 is out of bounds for axis 0 with size 500`

mboquien commented 9 months ago

This is curious. Unclear what the problem might be. However, I see your python version is not supported anymore. Can you check if the error still occurs with a more recent python?