Code is not working out of the box

[yradwan147]

I am using Python3.8.17, I created an environment and followed the instructions but out of the box, when using the examples given in the documentation, they do not work.

Running this example from the docs:

import esr.generation.duplicate_checker

runname = 'core_maths'
comp = 5
esr.generation.duplicate_checker.main(runname, comp)

I get this error: File d:\stuff\ESR\esr\generation\duplicate_checker.py:268, in main(runname, compl, track_memory, search_tmax, expand_tmax, seed) 266 print('\nChecking Results', flush=True) 267 if compl > 2: --> 268 simplifier.check_results(dirname, compl) 270 sys.stdout.flush() 271 comm.Barrier()

File d:\stuff\ESR\esr\generation\simplifier.py:1283, in check_results(dirname, compl, tmax) 1279 nfun = comm.bcast(nfun, root=0) 1281 print(nfun, rank, size) -> 1283 imin, imax = utils.split_idx(nfun, rank, size) 1284 imax += 1 1285 imin = comm.gather(imin, root=0)

ValueError: not enough values to unpack (expected 2, got 0)

Am I using the wrong python version? What's the problem here?

[DeaglanBartlett]

Hi! I'm sorry to here this step didn't work. I just tried running this code with python3.8.17 and did not receive that error which is confusing. I used the attached environment, both using one or several cores. I'm running on MacOS with openmpi. What architecture are you using? Can you see what happens with this environment?

name: esr_env
channels:
  - defaults
dependencies:
  - ca-certificates=2023.08.22=hecd8cb5_0
  - libcxx=14.0.6=h9765a3e_0
  - libffi=3.4.4=hecd8cb5_0
  - libgfortran=3.0.1=h93005f0_2
  - mpi=1.0=mpich
  - mpi4py=3.1.4=py38hce9ae6d_0
  - mpich=3.3.2=hc856adb_0
  - ncurses=6.4=hcec6c5f_0
  - openssl=3.0.11=hca72f7f_2
  - pip=23.2.1=py38hecd8cb5_0
  - python=3.8.17=h5ee71fb_0
  - readline=8.2=hca72f7f_0
  - setuptools=68.0.0=py38hecd8cb5_0
  - sqlite=3.41.2=h6c40b1e_0
  - tk=8.6.12=h5d9f67b_0
  - wheel=0.41.2=py38hecd8cb5_0
  - xz=5.4.2=h6c40b1e_0
  - zlib=1.2.13=h4dc903c_0
  - pip:
      - alabaster==0.7.13
      - astropy==5.2.2
      - babel==2.13.0
      - certifi==2023.7.22
      - charset-normalizer==3.3.0
      - contourpy==1.1.1
      - cycler==0.12.1
      - docutils==0.20.1
      - fonttools==4.43.1
      - idna==3.4
      - imagesize==1.4.1
      - importlib-metadata==6.8.0
      - importlib-resources==6.1.0
      - jinja2==3.1.2
      - kiwisolver==1.4.5
      - markdown-it-py==3.0.0
      - markupsafe==2.1.3
      - matplotlib==3.7.3
      - mdit-py-plugins==0.4.0
      - mdurl==0.1.2
      - mpmath==1.3.0
      - myst-parser==2.0.0
      - networkx==3.1
      - numdifftools==0.9.41
      - numpy==1.24.4
      - packaging==23.2
      - pandas==2.0.3
      - pillow==10.0.1
      - prettytable==3.9.0
      - psutil==5.9.5
      - pyerfa==2.0.0.3
      - pygments==2.16.1
      - pympler==1.0.1
      - pyparsing==3.1.1
      - python-dateutil==2.8.2
      - pytz==2023.3.post1
      - pyyaml==6.0.1
      - requests==2.31.0
      - scipy==1.10.1
      - six==1.16.0
      - snowballstemmer==2.2.0
      - sphinx==7.1.2
      - sphinxcontrib-applehelp==1.0.4
      - sphinxcontrib-devhelp==1.0.2
      - sphinxcontrib-htmlhelp==2.0.1
      - sphinxcontrib-jsmath==1.0.1
      - sphinxcontrib-qthelp==1.0.3
      - sphinxcontrib-serializinghtml==1.1.5
      - sympy==1.12
      - treelib==1.7.0
      - tzdata==2023.3
      - urllib3==2.0.6
      - wcwidth==0.2.8
      - zipp==3.17.0

[yradwan147]

I am on Windows 11

When I try to create the environment with your yml configuration, I get this,

Collecting package metadata (repodata.json): done Solving environment: failed

ResolvePackageNotFound:

• setuptools==68.0.0=py38hecd8cb5_0
• sqlite==3.41.2=h6c40b1e_0
• openssl==3.0.11=hca72f7f_2
• xz==5.4.2=h6c40b1e_0
• ca-certificates==2023.08.22=hecd8cb5_0
• wheel==0.41.2=py38hecd8cb5_0
• ncurses==6.4=hcec6c5f_0
• pip==23.2.1=py38hecd8cb5_0
• libcxx==14.0.6=h9765a3e_0
• python==3.8.17=h5ee71fb_0
• mpi==1.0=mpich
• readline==8.2=hca72f7f_0
• libffi==3.4.4=hecd8cb5_0
• mpi4py==3.1.4=py38hce9ae6d_0
• tk==8.6.12=h5d9f67b_0
• zlib==1.2.13=h4dc903c_0
• libgfortran==3.0.1=h93005f0_2
• mpich==3.3.2=hc856adb_0

[yradwan147]

I tried using the same pip dependencies, the same error in the initial issue pops up. There seems to be a problem with this snippet of the code:

if rank == 0:
        print('\nChecking Results', flush=True)
    if compl > 2:
        simplifier.check_results(dirname, compl)

When complexity is higher than 2, it breaks because in the split_idx function

def split_idx(Ntotal, r, indices_or_sections):
    """ Returns the rth set indices for numpy.array_split(a,indices_or_sections)
    where len(a) = Ntotal

    Args:
        :Ntotal (int): length of array to split
        :r (int): rank whose indices are required
        :indices_or_sections (int): how many parts to split array into

    Returns:
        :i (list): [min, max] index used by rank
    """
    try:
        # handle array case.
        Nsections = len(indices_or_sections) + 1
        div_points = [0] + list(indices_or_sections) + [Ntotal]
    except TypeError:
        # indices_or_sections is a scalar, not an array.
        Nsections = int(indices_or_sections)
        if Nsections <= 0:
            raise ValueError('number sections must be larger than 0.') from None
        Neach_section, extras = divmod(Ntotal, Nsections)
        section_sizes = ([0] +
                         extras * [Neach_section+1] +
                         (Nsections-extras) * [Neach_section])
        div_points = np.array(section_sizes, dtype=np.intp).cumsum()

    imin = div_points[r]
    imax = div_points[r + 1]
    if imin >= imax:
        i = []
    else:
        i = [imin, imax-1]

    return i

It returns i = [], which breaks the code. This seems to be the main problem.

[yradwan147]

I just used the code on Kaggle and it worked so I'll just work with that for now.

[DeaglanBartlett]

I have not tested the code on Windows (only Ubuntu and MacOS) so it's a shame that it didn't work for you. I'm glad you managed to get this working on Kaggle, but it would be nice to see why this isn't working for Windows.

Please could you let me know what the arguments passed to the split_idx function are, i.e. what are the values of Ntotal, r, and indices_or_sections? The first should be the number of equations produced (so I hope this isn't zero!), the second should be the rank number which is calling the function and the final should be the number of cores on which the code is running. My hunch is that there is some issue with the parallelisation through mpi4py which is causing this problem.

[yradwan147]

It gives 0 0 1 right before it breaks. I also wanted to ask how I can create a custom likelihood with bivariate inputs? In the documentation, it is mentioned that I have to define xvar, yvar, and yerr but what if xvar is multiple values? I tried putting them in a list named xvar but that didn't work and the results that came out were literally missing x in the functions. I'll attach the results file.

+------+---------------------------+--------+----------+--------+---------+----------+-----------+-----------+----------+----------+ | Rank | Function | L(D) | Prel | -logL | Codelen | AIFeyn | a0 | a1 | a2 | a3 | +------+---------------------------+--------+----------+--------+---------+----------+-----------+-----------+----------+----------+ | 1 | 1/(a0 + 2) | 350.32 | 6.55e-01 | 337.44 | 0.00 | 1.29e+01 | 0.00e+00 | 0.00e+00 | 0.00e+00 | 0.00e+00 | | 2 | 1/(a0 + 1/2) | 351.61 | 1.81e-01 | 337.28 | 1.45 | 1.29e+01 | 1.39e+00 | 0.00e+00 | 0.00e+00 | 0.00e+00 | | 3 | a0 + 1 - 1/a1 | 351.79 | 1.51e-01 | 337.29 | 1.62 | 1.29e+01 | 0.00e+00 | 2.17e+00 | 0.00e+00 | 0.00e+00 | | 4 | pow(Abs(a0),(1/Abs(a1))) | 352.23 | 0.00e+00 | 337.28 | 2.07 | 1.29e+01 | -1.80e-01 | 2.71e+00 | 0.00e+00 | 0.00e+00 | | 5 | 1/sqrt(Abs(a0)) | 352.68 | 0.00e+00 | 337.28 | 1.07 | 1.43e+01 | 3.55e+00 | 0.00e+00 | 0.00e+00 | 0.00e+00 | | 6 | 1/(a0(a1 + 1)) | 352.90 | 0.00e+00 | 337.28 | 2.75 | 1.29e+01 | 1.02e+00 | 8.48e-01 | 0.00e+00 | 0.00e+00 | | 7 | pow(Abs(a0),(1/(a1 + 1))) | 353.27 | 0.00e+00 | 337.28 | 1.66 | 1.43e+01 | -2.22e-01 | 1.38e+00 | 0.00e+00 | 0.00e+00 | | 8 | 1 + 1/(a0a1) | 353.43 | 0.00e+00 | 337.28 | 3.28 | 1.29e+01 | -2.96e+00 | 7.18e-01 | 0.00e+00 | 0.00e+00 | | 9 | 1/(a0 - 1/2) | 353.61 | 0.00e+00 | 337.28 | 2.00 | 1.43e+01 | 2.39e+00 | 0.00e+00 | 0.00e+00 | 0.00e+00 | | 10 | pow(Abs(1 + 1/a0),a1) | 353.91 | 0.00e+00 | 337.28 | 2.30 | 1.43e+01 | 1.07e+00 | -9.61e-01 | 0.00e+00 | 0.00e+00 | +------+---------------------------+--------+----------+--------+---------+----------+-----------+-----------+----------+----------+

[DeaglanBartlett]

Thank you - I'll look into this.

Yes so this is a currently open issue (#13) with ESR. We originally designed ESR to work with univariate functions, since our original applications required only this. So the functions expect a single value of x for each y, and therefore when you try to use multiple entries the functions cannot be evaluated. Instead of crashing, the code just assumes that this is a bad function and ignores it. As a result, the only functions which the code considers are those which didn't throw an error, i.e. the ones without any x values. This explains why these are the ones which have been returned at the end.

How many inputs do you wish to have? Given your interest and the fact that #13 has been open for a while, perhaps this is the motivation I need to address this!

[yradwan147]

I wanted to use bivariate datasets like Nikuradse1, I wish you the best in your work. Thanks alot for your help