search

NNPDF / nnpdf

An open-source machine learning framework for global analyses of parton distributions.
https://docs.nnpdf.science/
GNU General Public License v3.0
29 stars 6 forks source link

Problem when running fits with MHOU #1769

Closed enocera closed 1 year ago

enocera commented 1 year ago

Dear @andreab1997 (cc @scarlehoff), I'm trying to perform fits with the theory cov matrix. When I run n3fit I get the following error (for each replica; this is replica 10, for illustration purposes):

[INFO]: Creating replica output folder in /data/theorie/enocera/nnpdfgit/fits/230704-ern-009a/nnfit/replica_10
[WARNING]: Using q2min from runcard
[WARNING]: Using w2min from runcard
[WARNING]: Using maxTau from runcard
[INFO]: All requirements processed and checked successfully. Executing actions.
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: NonUniqueGrids', src/fk_table.rs:47:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/bin/n3fit", line 33, in <module>
    sys.exit(load_entry_point('n3fit', 'console_scripts', 'n3fit')())
  File "/data/theorie/enocera/nnpdfgit/nnpdf/n3fit/src/n3fit/scripts/n3fit_exec.py", line 298, in main
    a.main()
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/app.py", line 395, in main
    self.run()
  File "/data/theorie/enocera/nnpdfgit/nnpdf/n3fit/src/n3fit/scripts/n3fit_exec.py", line 286, in run
    super().run()
  File "/data/theorie/enocera/nnpdfgit/nnpdf/validphys2/src/validphys/app.py", line 152, in run
    super().run()
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/app.py", line 380, in run
    rb.execute_sequential()
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 166, in execute_sequential
    result = self.get_result(callspec.function,
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 175, in get_result
    fres =  function(**kwdict)
  File "/data/theorie/enocera/nnpdfgit/nnpdf/validphys2/src/validphys/covmats.py", line 252, in dataset_t0_predictions
    return central_predictions(dataset, t0set).to_numpy().reshape(-1)
  File "/data/theorie/enocera/nnpdfgit/nnpdf/validphys2/src/validphys/convolution.py", line 190, in central_predictions
    return _predictions(dataset, pdf, central_fk_predictions)
  File "/data/theorie/enocera/nnpdfgit/nnpdf/validphys2/src/validphys/convolution.py", line 122, in _predictions
    fk_w_cuts = load_fktable(fk).with_cuts(cuts)
  File "/data/theorie/enocera/nnpdfgit/nnpdf/validphys2/src/validphys/fkparser.py", line 61, in load_fktable
    tabledata = pineappl_reader(spec)
  File "/data/theorie/enocera/nnpdfgit/nnpdf/validphys2/src/validphys/pineparser.py", line 242, in pineappl_reader
    pines = [FkTable.read(i) for i in fkspec.fkpath]
  File "/data/theorie/enocera/nnpdfgit/nnpdf/validphys2/src/validphys/pineparser.py", line 242, in <listcomp>
    pines = [FkTable.read(i) for i in fkspec.fkpath]
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/pineappl/fk_table.py", line 35, in read
    return cls(PyFkTable.read(path))
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: NonUniqueGrids

[CRITICAL]: A critical error ocurred. This is likely due to one of the following reasons:

 - A bug in validphys.
 - Corruption of the provided resources (e.g. incorrect plotting files).
 - Cosmic rays hitting your CPU and altering the registers.

The traceback above should help determine the cause of the problem. If you
believe this is a bug in validphys (please discard the cosmic rays first),
please open an issue on GitHub<https://github.com/NNPDF/nnpdf/issues>,
including the contents of the following file:

/pbs/jobdir/14654373.burrell.nikhef.nl/n3fit-crash-mq3pqefy
andreab1997 commented 1 year ago

Mmm this seems a pineappl error. Can you check the pineappl version you are using?

enocera commented 1 year ago

Ah, OK.

enocera commented 1 year ago

I also have another problem:

(base) [enocera@fiora nnfit-230704-ern-008b]$ more nnfit-230704-ern-008b-1.q.e14654244 
[INFO]: Creating replica output folder in /data/theorie/enocera/nnpdfgit/fits/230704-ern-008b/nnfit/replica_1
[WARNING]: Using q2min from runcard
[WARNING]: Using w2min from runcard
[WARNING]: Using maxTau from runcard
[CRITICAL]: Bug in n3fit ocurred. Please report it.
Traceback (most recent call last):
  File "/data/theorie/enocera/nnpdfgit/nnpdf/n3fit/src/n3fit/scripts/n3fit_exec.py", line 286, in run
    super().run()
  File "/data/theorie/enocera/nnpdfgit/nnpdf/validphys2/src/validphys/app.py", line 152, in run
    super().run()
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/app.py", line 358, in run
    rb.resolve_fuzzytargets()
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 370, in resolve_fuzzytargets
    self.resolve_fuzzytarget(target)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 379, in resolve_fuzzytarget
    self.process_targetspec(fuzzytarget.name, spec, fuzzytarget.extraargs)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 388, in process_targetspec
    gen.send(None)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 450, in _process_requirement
    yield from self._make_node(name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 466, in _make_node
    yield from self._make_callspec(f, name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 499, in _make_callspec
    index, _ = gen.send(None)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 413, in _process_requirement
    yield from self._make_node(name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 464, in _make_node
    yield from self._make_collect(f, name, nsspec, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 593, in _make_collect
    _, newcs = gen.send(None)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 450, in _process_requirement
    yield from self._make_node(name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 466, in _make_node
    yield from self._make_callspec(f, name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 499, in _make_callspec
    index, _ = gen.send(None)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 413, in _process_requirement
    yield from self._make_node(name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 464, in _make_node
    yield from self._make_collect(f, name, nsspec, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 593, in _make_collect
    _, newcs = gen.send(None)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 450, in _process_requirement
    yield from self._make_node(name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 466, in _make_node
    yield from self._make_callspec(f, name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 499, in _make_callspec
    index, _ = gen.send(None)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 450, in _process_requirement
    yield from self._make_node(name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 466, in _make_node
    yield from self._make_callspec(f, name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 499, in _make_callspec
    index, _ = gen.send(None)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 432, in _process_requirement
    yield from self._make_node((name, val.value), nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 466, in _make_node
    yield from self._make_callspec(f, name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 499, in _make_callspec
    index, _ = gen.send(None)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 417, in _process_requirement
    put_index, val = self.input_parser.resolve_key(name, ns, parents=parents, currspec=nsspec)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/configparser.py", line 429, in resolve_key
    return self._resolve_key(key=key, ns=ns, input_params=input_params,
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/configparser.py", line 491, in _resolve_key
    val = produce_func(**kwargs)
  File "/data/theorie/enocera/nnpdfgit/nnpdf/validphys2/src/validphys/config.py", line 762, in produce_loaded_theory_covmat
    theory_covmat = pd.read_csv(
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/pandas/util/_decorators.py", line 211, in wrapper
    return func(*args, **kwargs)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/pandas/util/_decorators.py", line 331, in wrapper
    return func(*args, **kwargs)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 950, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 605, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1442, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1735, in _make_engine
    self.handles = get_handle(
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/pandas/io/common.py", line 856, in get_handle
    handle = open(
FileNotFoundError: [Errno 2] No such file or directory: '/data/theorie/enocera/nnpdfgit/fits/230704-ern-008b/tables/datacuts_theory_theorycovmatconfig_theory_covmat_custom.csv'

Is this because I have to run vp-setupfit first, and vp-setupfit generates the file, or someone should provide that file?

andreab1997 commented 1 year ago

I also have another problem:

(base) [enocera@fiora nnfit-230704-ern-008b]$ more nnfit-230704-ern-008b-1.q.e14654244 
[INFO]: Creating replica output folder in /data/theorie/enocera/nnpdfgit/fits/230704-ern-008b/nnfit/replica_1
[WARNING]: Using q2min from runcard
[WARNING]: Using w2min from runcard
[WARNING]: Using maxTau from runcard
[CRITICAL]: Bug in n3fit ocurred. Please report it.
Traceback (most recent call last):
  File "/data/theorie/enocera/nnpdfgit/nnpdf/n3fit/src/n3fit/scripts/n3fit_exec.py", line 286, in run
    super().run()
  File "/data/theorie/enocera/nnpdfgit/nnpdf/validphys2/src/validphys/app.py", line 152, in run
    super().run()
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/app.py", line 358, in run
    rb.resolve_fuzzytargets()
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 370, in resolve_fuzzytargets
    self.resolve_fuzzytarget(target)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 379, in resolve_fuzzytarget
    self.process_targetspec(fuzzytarget.name, spec, fuzzytarget.extraargs)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 388, in process_targetspec
    gen.send(None)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 450, in _process_requirement
    yield from self._make_node(name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 466, in _make_node
    yield from self._make_callspec(f, name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 499, in _make_callspec
    index, _ = gen.send(None)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 413, in _process_requirement
    yield from self._make_node(name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 464, in _make_node
    yield from self._make_collect(f, name, nsspec, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 593, in _make_collect
    _, newcs = gen.send(None)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 450, in _process_requirement
    yield from self._make_node(name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 466, in _make_node
    yield from self._make_callspec(f, name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 499, in _make_callspec
    index, _ = gen.send(None)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 413, in _process_requirement
    yield from self._make_node(name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 464, in _make_node
    yield from self._make_collect(f, name, nsspec, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 593, in _make_collect
    _, newcs = gen.send(None)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 450, in _process_requirement
    yield from self._make_node(name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 466, in _make_node
    yield from self._make_callspec(f, name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 499, in _make_callspec
    index, _ = gen.send(None)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 450, in _process_requirement
    yield from self._make_node(name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 466, in _make_node
    yield from self._make_callspec(f, name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 499, in _make_callspec
    index, _ = gen.send(None)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 432, in _process_requirement
    yield from self._make_node((name, val.value), nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 466, in _make_node
    yield from self._make_callspec(f, name, nsspec, extraargs, parents)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 499, in _make_callspec
    index, _ = gen.send(None)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 417, in _process_requirement
    put_index, val = self.input_parser.resolve_key(name, ns, parents=parents, currspec=nsspec)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/configparser.py", line 429, in resolve_key
    return self._resolve_key(key=key, ns=ns, input_params=input_params,
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/reportengine/configparser.py", line 491, in _resolve_key
    val = produce_func(**kwargs)
  File "/data/theorie/enocera/nnpdfgit/nnpdf/validphys2/src/validphys/config.py", line 762, in produce_loaded_theory_covmat
    theory_covmat = pd.read_csv(
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/pandas/util/_decorators.py", line 211, in wrapper
    return func(*args, **kwargs)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/pandas/util/_decorators.py", line 331, in wrapper
    return func(*args, **kwargs)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 950, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 605, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1442, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1735, in _make_engine
    self.handles = get_handle(
  File "/data/theorie/enocera/miniconda3/envs/nnpdf-new-new/lib/python3.9/site-packages/pandas/io/common.py", line 856, in get_handle
    handle = open(
FileNotFoundError: [Errno 2] No such file or directory: '/data/theorie/enocera/nnpdfgit/fits/230704-ern-008b/tables/datacuts_theory_theorycovmatconfig_theory_covmat_custom.csv'

Is this because I have to run vp-setupfit first, and vp-setupfit generates the file, or someone should provide that file?

Indeed, you need to first run vp-setupfit to get the thcovmat file and then run n3fit

andreab1997 commented 1 year ago

So has updating the pineappl version solved the first problem?

enocera commented 1 year ago

Indeed, you need to first run vp-setupfit to get the thcovmat file and then run n3fit

Then this should be specified in the docs (if it isn't yet).

andreab1997 commented 1 year ago

Yes you are right, for some reason I was sure it was written somewhere in the docs (but it is not)

RoyStegeman commented 1 year ago

In principle the first step for "how to run a fit" in the docs is exactly to rujn vp-setupfit: https://docs.nnpdf.science/tutorials/run-fit.html#running-the-fitting-code

However, it also mentions that for n3fit this is not necessary so perhaps that statement should be clarified a bit.

enocera commented 1 year ago

However, it also mentions that for n3fit this is not necessary so perhaps that statement should be clarified a bit.

That was my point. I'd just remove that sentence.

enocera commented 1 year ago

So has updating the pineappl version solved the first problem?

Yes. I see that the module fiatlux is also needed, otherwise there's another error message.