Error in conda install of Dakota

[kbarnhart]

@mdpiper I am working on setting up the conda environment.yml for my upcoming CSDMS clinic. I plan to use Dakota in this clinic. I was planning to use the Dakota package distributed through conda.

I created a new conda environment and made sure that my prior Dakota install was no longer present.

$ conda create -n dakota python=3.6
$ conda activate Dakota
$ conda install -c csdms-stack dakota
Collecting package metadata: done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - dakota -> lapack

Current channels:

  - https://conda.anaconda.org/csdms-stack/osx-64
  - https://conda.anaconda.org/csdms-stack/noarch
  - https://repo.anaconda.com/pkgs/main/osx-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/free/osx-64
  - https://repo.anaconda.com/pkgs/free/noarch
  - https://repo.anaconda.com/pkgs/r/osx-64
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

please let me know how to fix this. Thanks!

[mdpiper]

@kbarnhart I used conda-forge to supply some dependencies for the dakota recipe. Please try:

conda create -n dakota python=3.6 -c conda-forge
conda activate dakota
conda install -c csdms-stack -c conda-forge dakota

Of course you wouldn't know about this because I didn't make install instructions! I'm sorry about that. I've created https://github.com/csdms-stack/dakota-recipe/issues/2 to address this.

Please let me know if the above works.

P.S. If you'd like, try the new CSDMS Help Desk.

[kbarnhart]

Thanks @mdpiper. Following your instructions my conda install exits successfully but then when I try and run Dakota (e.g. by just typing dakota in the command line), I get the following error:

$ dakota
dyld: Library not loaded: @rpath/libmpi_cxx.1.dylib
  Referenced from: /anaconda3/envs/dakota/bin/dakota
  Reason: image not found
Abort trap: 6

Is this expected?

[mdpiper]

@kbarnhart I think this is a case of bit rot. When I built the dakota package, I used whatever openmpi was current. Now that openmpi has gone through a few releases, if I install the current version, dakota can't find the library it was compiled against. I can get around this by revising my instructions above and installing an older version of openmpi:

conda create -n dakota python=3.6 openmpi=1.6
conda activate dakota
conda install -c csdms-stack -c conda-forge dakota

Note that this is only for macOS; everything appears to be working on Linux.

This is still my problem -- I need to learn how to be more forward-compatible when I build binary packages on conda.

[mcflugen]

@mdpiper how about building dakota on conda-forge? Maybe that would help?

[kbarnhart]

@mdpiper I can confirm that your solution worked for me. I haven't gotten to USING Dakota built in this way yet... So I'll wait to confirm that everything is working before closing the issue. thanks!

[mdpiper]

@mcflugen I think this is the right thing to do. However, in order to get what I have to build, I disabled many features and all the tests. I'd feel bad about putting my stripped-down version of dakota on the public conda-forge channel. In order to build the full version of dakota, it'll take a bit more work.

[kbarnhart]

@mdpiper @mcflugen I haven't tested this in the current build (so maybe its in there, but I suspect not). Having Queso enabled would be great https://dakota.sandia.gov/content/options-dakota-features

[kbarnhart]

@mdpiper I've run into some issues when I try and run a very simple Dakota method.(multidim_parameter_study). I'm pretty sure its a dylib issue (as outlined below) Perhaps you can help.

Context: I'm working on creating my clinic for the CSDMS annual meeting. I'd like to NOT use ~hydrosphere~ Hydroshare (stupid autocorrect) as I find it takes longer than I'd like to spend explaining to students what it is. So I want to provide a environment.yml file and deal with everything through conda environment.

I've put my working file structure in this repository. It has the following folders

• the Clow 2014 paper and datafiles
• Dakota resources
• a modified version of heat (its 1d instead of 2d and I need to ask your and @mcflugen 's recommendation as to how to attribute it properly) and
• model which has code to run Dakota.

starting from the cloned repository, to reproduce (I hope) my error start by creating an environment and compiling the heat python module.

$ git clone https://github.com/kbarnhart/calibration_with_dakota_clinic.git
$ cd calibration_with_dakota_clinic
$ conda env create -f environment.yaml
$ conda activate dakota_clinic
$ cd heat
$ make install
$ cd ../model
$ source start_01_grid.sh

This last line should fail quickly and inspecting dakota_01_grid.log should read:

Fork application failure, aborting.
System error message: No such file or directory

This typically means that a file Dakota expects is not there. To figure out where the file failed to create I did the following.

First, I looked at the end of dakota_01_grid.out as it will show the last thing Dakota did before it failed. The last line reads:

blocking fork: source run_preprocessing.sh params.in results.out; python driver.py params.in results.out

These two lines indicate that Dakota called input_filter and the analysis_driver commands with the appropriate system arguments params.in and results.out

Next, I navigated to the correct directory and ran each of these commands.

$ cd GRID_SEARCH/run.1/
$ source run_preprocessing.sh params.in results.out

This command exited without issue, so I suspect the problem is with the second command. Running it...

$ $ python driver.py params.in results.out
2019-05-06 15:40:34.900 python[3927:1093928] -[NSApplication _setup:]: unrecognized selector sent to instance 0x7f8dc7c2bfc0
2019-05-06 15:40:34.902 python[3927:1093928] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[NSApplication _setup:]: unrecognized selector sent to instance 0x7f8dc7c2bfc0'
*** First throw call stack:
(
    0   CoreFoundation                      0x00007fff3e212cf9 __exceptionPreprocess + 256
    1   libobjc.A.dylib                     0x00007fff68da6a17 objc_exception_throw + 48
    2   CoreFoundation                      0x00007fff3e28cb06 -[NSObject(NSObject) __retain_OA] + 0
    3   CoreFoundation                      0x00007fff3e1b50ef ___forwarding___ + 1485
    4   CoreFoundation                      0x00007fff3e1b4a98 _CF_forwarding_prep_0 + 120
    5   libtk8.6.dylib                      0x000000011d940154 TkpInit + 324
    6   libtk8.6.dylib                      0x000000011d8980ee Initialize + 2622
    7   _tkinter.cpython-36m-darwin.so      0x000000011a782a46 _tkinter_create + 1174
    8   python                              0x000000010d90c538 _PyCFunction_FastCallDict + 200
    9   python                              0x000000010d9e358f call_function + 143
    10  python                              0x000000010d9e1111 _PyEval_EvalFrameDefault + 46817
    11  python                              0x000000010d9d4869 _PyEval_EvalCodeWithName + 425
    12  python                              0x000000010d9e4165 _PyFunction_FastCallDict + 373
    13  python                              0x000000010d88ac80 _PyObject_FastCallDict + 320
    14  python                              0x000000010d8b23d8 method_call + 136
    15  python                              0x000000010d8922de PyObject_Call + 62
    16  python                              0x000000010d9338d5 slot_tp_init + 117
    17  python                              0x000000010d937e21 type_call + 241
    18  python                              0x000000010d88abf1 _PyObject_FastCallDict + 177
    19  python                              0x000000010d892e47 _PyObject_FastCallKeywords + 327
    20  python                              0x000000010d9e3688 call_function + 392
    21  python                              0x000000010d9e11c1 _PyEval_EvalFrameDefault + 46993
    22  python                              0x000000010d9e394c fast_function + 188
    23  python                              0x000000010d9e35ec call_function + 236
    24  python                              0x000000010d9e1111 _PyEval_EvalFrameDefault + 46817
    25  python                              0x000000010d9d4869 _PyEval_EvalCodeWithName + 425
    26  python                              0x000000010d9e4165 _PyFunction_FastCallDict + 373
    27  python                              0x000000010d88ac80 _PyObject_FastCallDict + 320
    28  python                              0x000000010d8b23d8 method_call + 136
    29  python                              0x000000010d8922de PyObject_Call + 62
    30  python                              0x000000010d9e1320 _PyEval_EvalFrameDefault + 47344
    31  python                              0x000000010d9d4869 _PyEval_EvalCodeWithName + 425
    32  python                              0x000000010d9e39fa fast_function + 362
    33  python                              0x000000010d9e35ec call_function + 236
    34  python                              0x000000010d9e1111 _PyEval_EvalFrameDefault + 46817
    35  python                              0x000000010d9d4869 _PyEval_EvalCodeWithName + 425
    36  python                              0x000000010da2d0bc PyRun_FileExFlags + 252
    37  python                              0x000000010da2c594 PyRun_SimpleFileExFlags + 372
    38  python                              0x000000010da531f6 Py_Main + 3766
    39  python                              0x000000010d882d19 main + 313
    40  libdyld.dylib                       0x00007fff6a5d43d5 start + 1
)
libc++abi.dylib: terminating with uncaught exception of type NSException
Abort trap: 6
(dakota_clinic) rgnt2-71-14-dhcp:run.1 barnhark$ ls
AWU_12AUG07.txt     input_template.yml  params.in
driver.py       inputs.yml      run_preprocessing.sh
(dakota_clinic) rgnt2-71-14-dhcp:run.1 barnhark$ python driver.py params.in results.out
2019-05-06 15:48:05.935 python[4136:1130537] -[NSApplication _setup:]: unrecognized selector sent to instance 0x7fbb90f4bd20
2019-05-06 15:48:05.939 python[4136:1130537] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[NSApplication _setup:]: unrecognized selector sent to instance 0x7fbb90f4bd20'
*** First throw call stack:
(
    0   CoreFoundation                      0x00007fff3e212cf9 __exceptionPreprocess + 256
    1   libobjc.A.dylib                     0x00007fff68da6a17 objc_exception_throw + 48
    2   CoreFoundation                      0x00007fff3e28cb06 -[NSObject(NSObject) __retain_OA] + 0
    3   CoreFoundation                      0x00007fff3e1b50ef ___forwarding___ + 1485
    4   CoreFoundation                      0x00007fff3e1b4a98 _CF_forwarding_prep_0 + 120
    5   libtk8.6.dylib                      0x000000011ef8a154 TkpInit + 324
    6   libtk8.6.dylib                      0x000000011eee20ee Initialize + 2622
    7   _tkinter.cpython-36m-darwin.so      0x000000011b7e5a46 _tkinter_create + 1174
    8   python                              0x000000010ef88538 _PyCFunction_FastCallDict + 200
    9   python                              0x000000010f05f58f call_function + 143
    10  python                              0x000000010f05d111 _PyEval_EvalFrameDefault + 46817
    11  python                              0x000000010f050869 _PyEval_EvalCodeWithName + 425
    12  python                              0x000000010f060165 _PyFunction_FastCallDict + 373
    13  python                              0x000000010ef06c80 _PyObject_FastCallDict + 320
    14  python                              0x000000010ef2e3d8 method_call + 136
    15  python                              0x000000010ef0e2de PyObject_Call + 62
    16  python                              0x000000010efaf8d5 slot_tp_init + 117
    17  python                              0x000000010efb3e21 type_call + 241
    18  python                              0x000000010ef06bf1 _PyObject_FastCallDict + 177
    19  python                              0x000000010ef0ee47 _PyObject_FastCallKeywords + 327
    20  python                              0x000000010f05f688 call_function + 392
    21  python                              0x000000010f05d1c1 _PyEval_EvalFrameDefault + 46993
    22  python                              0x000000010f05f94c fast_function + 188
    23  python                              0x000000010f05f5ec call_function + 236
    24  python                              0x000000010f05d111 _PyEval_EvalFrameDefault + 46817
    25  python                              0x000000010f050869 _PyEval_EvalCodeWithName + 425
    26  python                              0x000000010f060165 _PyFunction_FastCallDict + 373
    27  python                              0x000000010ef06c80 _PyObject_FastCallDict + 320
    28  python                              0x000000010ef2e3d8 method_call + 136
    29  python                              0x000000010ef0e2de PyObject_Call + 62
    30  python                              0x000000010f05d320 _PyEval_EvalFrameDefault + 47344
    31  python                              0x000000010f050869 _PyEval_EvalCodeWithName + 425
    32  python                              0x000000010f05f9fa fast_function + 362
    33  python                              0x000000010f05f5ec call_function + 236
    34  python                              0x000000010f05d111 _PyEval_EvalFrameDefault + 46817
    35  python                              0x000000010f050869 _PyEval_EvalCodeWithName + 425
    36  python                              0x000000010f0a90bc PyRun_FileExFlags + 252
    37  python                              0x000000010f0a8594 PyRun_SimpleFileExFlags + 372
    38  python                              0x000000010f0cf1f6 Py_Main + 3766
    39  python                              0x000000010eefed19 main + 313
    40  libdyld.dylib                       0x00007fff6a5d43d5 start + 1
)
libc++abi.dylib: terminating with uncaught exception of type NSException
Abort trap: 6

I have zero idea how to address this. Do you? Let me know if you have any recommendations or if you run into issues with reproducing these results. Thanks!

[kbarnhart]

A couple of additional comments:

1) I've started CI (lite) on the repo (travis only at present). Right now it only tests whether the env can be created and the two tests for heat can run. eventually I'll have it test that Dakota can also run.

2) When the Dakota conda build is re-visited I'd like to make a vote for enabling the python dakota.interfacing module. I've done this with my local binary build in my ~\.bash_profile with

# To use the dakota.interfacing Python module
export PYTHONPATH=$PYTHONPATH:$INSTALL_DIR/share/Dakota/Python

[mdpiper]

@kbarnhart I have an environment file for macOS that I used for testing. I think it may help to check yours against it.

[kbarnhart]

That sounds like a good idea! Let me know where to find it (or paste it here) and I’ll go from there.

Some further googleing indicates that there may be some additional issues with fork and needing to ensure the call is executable (e.g., I think the ‘source’ part of the input filter is also part of the issue). Will investigate that further when not on the bus.

On Mon, May 6, 2019 at 5:33 PM Mark Piper notifications@github.com wrote:

@kbarnhart https://github.com/kbarnhart I have an environment file https://github.com/mdpiper/conda-environments/blob/master/dakota-osx.yml for macOS that I used for testing. I think it may help to check yours against it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/csdms/help-desk/issues/4#issuecomment-489824539, or mute the thread https://github.com/notifications/unsubscribe-auth/ABSIXG6UK7J7PM4KESIID3DPUC55JANCNFSM4HKI2JEA .

[mdpiper]

It’s linked in the message.

(Unrelated: I’m currently sheltered at the SoBo rec center from copious lightning)

mp

On May 6, 2019, at 5:51 PM, Katy Barnhart notifications@github.com<mailto:notifications@github.com> wrote:

That sounds like a good idea! Let me know where to find it (or paste it here) and I’ll go from there.

Some further googleing indicates that there may be some additional issues with fork and needing to ensure the call is executable (e.g., I think the ‘source’ part of the input filter is also part of the issue). Will investigate that further when not on the bus.

On Mon, May 6, 2019 at 5:33 PM Mark Piper notifications@github.com<mailto:notifications@github.com> wrote:

@kbarnhart https://github.com/kbarnhart I have an environment file https://github.com/mdpiper/conda-environments/blob/master/dakota-osx.yml for macOS that I used for testing. I think it may help to check yours against it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/csdms/help-desk/issues/4#issuecomment-489824539, or mute the thread https://github.com/notifications/unsubscribe-auth/ABSIXG6UK7J7PM4KESIID3DPUC55JANCNFSM4HKI2JEA .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/csdms/help-desk/issues/4#issuecomment-489828052, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABLJF35XV3BYOR2FMRSMFFTPUDAAZANCNFSM4HKI2JEA.

[kbarnhart]

Oops. Sorry about that! Email formatting did not indicate that. Hope you have a safe bike ride home.

On Mon, May 6, 2019 at 6:03 PM Mark Piper notifications@github.com wrote:

It’s linked in the message.

(Unrelated: I’m currently sheltered at the SoBo rec center from copious lightning)

mp

On May 6, 2019, at 5:51 PM, Katy Barnhart <notifications@github.com mailto:notifications@github.com> wrote:

That sounds like a good idea! Let me know where to find it (or paste it here) and I’ll go from there.

Some further googleing indicates that there may be some additional issues with fork and needing to ensure the call is executable (e.g., I think the ‘source’ part of the input filter is also part of the issue). Will investigate that further when not on the bus.

On Mon, May 6, 2019 at 5:33 PM Mark Piper <notifications@github.com mailto:notifications@github.com> wrote:

@kbarnhart https://github.com/kbarnhart I have an environment file < https://github.com/mdpiper/conda-environments/blob/master/dakota-osx.yml> for macOS that I used for testing. I think it may help to check yours against it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/csdms/help-desk/issues/4#issuecomment-489824539, or mute the thread < https://github.com/notifications/unsubscribe-auth/ABSIXG6UK7J7PM4KESIID3DPUC55JANCNFSM4HKI2JEA>

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub< https://github.com/csdms/help-desk/issues/4#issuecomment-489828052>, or mute the thread< https://github.com/notifications/unsubscribe-auth/ABLJF35XV3BYOR2FMRSMFFTPUDAAZANCNFSM4HKI2JEA>.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/csdms/help-desk/issues/4#issuecomment-489830383, or mute the thread https://github.com/notifications/unsubscribe-auth/ABSIXG64WJE56I4ZJ5CHE23PUDBMTANCNFSM4HKI2JEA .

[kbarnhart]

@mdpiper local tests indicate that this works! Will keep you updated...

Also, I can confirm that I can have both (a) my normal prebuilt binaries and (b) my conda env Dakota installed and successfully toggle between them depending on which conda env is active. nice work!

[kbarnhart]

@mdpiper Alrighty! I can confirm that this works with the changes you recommended for the env file.

see these builds -- note super lame tests for Dakota (just using cat to write out the .out file and inspect, will be replaced).

@mdpiper do you expect this will only work on OSX? Or is there a glimmer of hope for Windows and/or Linux build by the meeting? If OSX only is the case, I'll just make sure that there are enough people with Macs in the room.

[mdpiper]

@kbarnhart Yes, there is a Linux binary. I don't have a comparable env file yet, but you can glean it from the run section of the recipe's meta.yaml file.

[kbarnhart]

@mdpiper thanks for pointing me to the yaml file.

I've now modified to have a environment_linux.yaml and an environment_osx.yaml.

I think this is the way I have the approach platform-specific environment files (correct me if I know better).

Right now the people who have responded to a request for pre-clinic information are clocking in at 50% OSX+Linux 50% windows... Hopefully that will stay about the same and I can have people pair up...

[kbarnhart]

@mdpiper I almost have it working...

2 of the 3 Dakota methods I’m testing work as expected. But the third (efficient global optimization) yields a segfault (core dumped) on Linux.

For example: https://travis-ci.com/kbarnhart/calibration_with_dakota_clinic/jobs/198764291#L981

Any thoughts?

[kbarnhart]

@mdpiper I have zero idea why... But these builds now all work (https://travis-ci.com/kbarnhart/calibration_with_dakota_clinic/builds/111269343).

Which is great since I was concerned that this was going to be a pain to resolve...