search

libprima / PRIMA.jl

a Julia interface to PRIMA, a Reference Implementation for Powell's methods with Modernization and Amelioration
MIT License
21 stars 5 forks source link

bobyqa crashes under Windows #19

Closed mzaffalon closed 1 month ago

mzaffalon commented 9 months ago

On Julia v1.10.0 and PRIMA v0.2.0, I get

using PRIMA
cost_func(x) = sum(abs2, x)
bobyqa(cost_func, randn(4))

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0xa8df5fbbe8 -- unknown function (ip: 000000a8df5fbbe8)
in expression starting at REPL[4]:1
unknown function (ip: 000000a8df5fbbe8)
Allocations: 748477 (Pool: 747841; Big: 636); GC: 1

EDIT:

julia> versioninfo()
Julia Version 1.10.0
Commit 3120989f39 (2023-12-25 18:01 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 6 × Intel(R) Core(TM) i5-8500 CPU @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, skylake)
  Threads: 1 on 6 virtual cores
<snip>
zaikunzhang commented 9 months ago

Hi @mzaffalon,

Thank you for raising this. Would @emmt or @amontoison have a look? Thank you.

Zaikun

amontoison commented 9 months ago

@mzaffalon What is your platform? I suspect that it's windows. I encountered similar errors on the Julia interface of GALAHAD.

Update: Sorry I didn't read the title of the issue... Do you have the same issue with Julia 1.9?

mzaffalon commented 9 months ago

@amontoison Sorry, I added the versioninfo to the initial report.

On v1.9.4 I get the same

using PRIMA
cost_func(x) = sum(abs2, x)
bobyqa(cost_func, randn(4))

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x3a85bfbab8 -- unknown function (ip: 0000003a85bfbab8)
in expression starting at REPL[4]:1
unknown function (ip: 0000003a85bfbab8)
Allocations: 1467033 (Pool: 1466079; Big: 954); GC: 2

I get the same access violation with the other methods under Julia v.1.10.0 (I did not test on v1.9.4).

emmt commented 9 months ago

@mzaffalon I have no problems with your example on Linux. Did you try the tests that come with PRIMA.jl? 

... pkg> test PRIMA

I do not have access to a machine with Windows OS but I can add your example to the tests on AppVeyor or GitHub Action for CI...

mzaffalon commented 9 months ago

@emmt Thanks for the suggestion.

(@v1.10) pkg> activate .
  Activating project at `C:\Users\michele.zaffalon\.julia\packages\PRIMA\CEs74`

(PRIMA) pkg> test
    Updating registry at `C:\Users\michele.zaffalon\.julia\registries\BrukerRegistry`
    Updating <local git_repo>
    Updating registry at `C:\Users\michele.zaffalon\.julia\registries\General.toml`
    Updating `C:\Users\michele.zaffalon\.julia\packages\PRIMA\CEs74\Project.toml`
  [c3b1956e] + TypeUtils v0.3.3
  [eead6e0c] + PRIMA_jll v0.7.1+0
    Updating `C:\Users\michele.zaffalon\.julia\packages\PRIMA\CEs74\Manifest.toml`
  [692b3bcd] + JLLWrappers v1.5.0
  [21216c6a] + Preferences v1.4.1
  [ae029012] + Requires v1.3.0
  [c3b1956e] + TypeUtils v0.3.3
  [eead6e0c] + PRIMA_jll v0.7.1+0
  [56f22d72] + Artifacts
  [ade2ca70] + Dates
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [de0858da] + Printf
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [fa267f1f] + TOML v1.0.3
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll v1.0.5+1
  [4536629a] + OpenBLAS_jll v0.3.23+2
  [8e850b90] + libblastrampoline_jll v5.8.0+1
     Testing PRIMA
      Status `C:\Users\michele.zaffalon\AppData\Local\Temp\jl_4LAuh4\Project.toml`
  [0a7d04aa] PRIMA v0.2.0 `C:\Users\michele.zaffalon\.julia\packages\PRIMA\CEs74`
  [c3b1956e] TypeUtils v0.3.3
  [eead6e0c] PRIMA_jll v0.7.1+0
  [37e2e46d] LinearAlgebra
  [8dfed614] Test
      Status `C:\Users\michele.zaffalon\AppData\Local\Temp\jl_4LAuh4\Manifest.toml`
  [692b3bcd] JLLWrappers v1.5.0
  [0a7d04aa] PRIMA v0.2.0 `C:\Users\michele.zaffalon\.julia\packages\PRIMA\CEs74`
  [21216c6a] Preferences v1.4.1
  [ae029012] Requires v1.3.0
  [c3b1956e] TypeUtils v0.3.3
  [eead6e0c] PRIMA_jll v0.7.1+0
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [b77e0a4c] InteractiveUtils
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [de0858da] Printf
  [9a3f8284] Random
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization
  [fa267f1f] TOML v1.0.3
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [e66e0078] CompilerSupportLibraries_jll v1.0.5+1
  [4536629a] OpenBLAS_jll v0.3.23+2
  [8e850b90] libblastrampoline_jll v5.8.0+1
Precompiling project...
  8 dependencies successfully precompiled in 6 seconds
     Testing Running tests...

NEWUOA:

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0xb6445ed988 -- unknown function (ip: 000000b6445ed988)
in expression starting at C:\Users\michele.zaffalon\.julia\packages\PRIMA\CEs74\test\runtests.jl:39
unknown function (ip: 000000b6445ed988)
Allocations: 1268949 (Pool: 1267200; Big: 1749); GC: 2
ERROR: Package PRIMA errored during testing

(PRIMA) pkg>
mzaffalon commented 9 months ago

I noticed that the CylancePROTECT antivirus complains: I do not know whether it is responsible to terminate the process.

emmt commented 9 months ago

I have just added your example to the test suite but now spellchecking prevent other tests to complete (@zaikunzhang it seems that some tools are not are not allowed to be used in libprima/PRIMA.jl).

I also have noticed that Appveyor is not trigerred at all in libprima/PRIMA.jl which I originally used to test the code on Windows. I can see 2 possibilities :

  1. register libprima/PRIMA.jl on Appveyor.
  2. modify GitHub action CI.yml to run the test on Windows (probably simpler). This require to solve the spellchecking issue...
zaikunzhang commented 9 months ago
  1. Appveyor.

Hi @emmt ,

Thank you for taking care of this.

  1. I have registered libprima/PRIMA.jl on Appveyor.
  2. I have enabled dcarbone/install-jq-action@* and pytooling/actions/with-post-step@* to run on any repo under libprima, so that CI.yml can be triggered now. It encounters two errors : https://github.com/libprima/PRIMA.jl/actions/runs/7693337998

Note that it was not true that "spellchecking prevent other tests to complete" --- not action can block any other. The error message you saw was the following:

Bad request - dcarbone/install-jq-action@v2.1.0 and pytooling/actions/with-post-step@adef08d3bdef092282614f3b683897cefae82ee3 are not allowed to be used in libprima/PRIMA.jl. Actions in this workflow must be: within a repository owned by libprima, created by GitHub, verified in the GitHub Marketplace, or matching the following: check-spelling/checkout-merge@, check-spelling/check-spelling@, check-spelling/spell-check-this@, jlumbroso/free-disk-space@, check-spelling/actions-checkout@, fortran-lang/setup-fortran@, juliaregistries/tagbot@, julia-actions/setup-julia@, julia-actions/cache@, julia-actions/julia-buildpkg@, julia-actions/julia-runtest@, julia-actions/julia-processcoverage@.

It means that there is a whitelist for the actions that can run on the repos of libprima --- I suppose it is understandable and indeed recommended to have such a whitelist. The old whitelist contains actions like check-spelling/checkout-merge@*, ..., julia-actions/julia-processcoverage@*, but this does not mean that such actions are blocking other actions.

To resolve the issue, I updated the whitelist to include dcarbone/install-jq-action@* and pytooling/actions/with-post-step@*

Merci beaucoup !

Zaikun

emmt commented 9 months ago

Note that it was not true that "spellchecking prevent other tests to complete" --- not action can block any other. The error message you saw was the following:

@zaikunzhang Sorry I misunderstandood the error message. Thank you very much for having fixed this. Now I can see that there are some issues with Julia 1.6 due to the different possible ways to implement extensions, I have to deal with them... Note that this is related to CUTEst not to the crash on Windows.

amontoison commented 9 months ago

@emmt We should bump the compat entry to Julia 1.9 if we use package extensions. It's a pain to use them on older Julia versions.

emmt commented 9 months ago

I agree that it is easier to deal with extensions on Julia prior to 1.9, but I managed to have something that will eventually work with Julia < 1.9 and Julia ≥ 1.9

emmt commented 9 months ago

@amontoison It seems that CUTEst fail on 32-bit architecture. Do you know whether CUTEst.jl supports these processors?

amontoison commented 9 months ago

It should but I need to compile a more recent version of CUTEst and regenerate CUTEst_jll. I'm working with Nick Gould on that but we need to switch to the Meson build system to more easily compile a shared library. Nick loves static libraries but we can't use them in Julia.

emmt commented 9 months ago

OK so let us skip testing CUTEst for 32-bit machine for now. We can re-activate this later...

emmt commented 9 months ago

I have activated testing on Windows OS by GitHub Actions but CUTEst failed to load on this platform because CUTEst_jll.artifact_dir is not defined:

ERROR: LoadError: InitError: UndefVarError: artifact_dir not defined Stacktrace: [1] getproperty @ .\Base.jl:31 [inlined] [2] init() @ CUTEst C:\Users\runneradmin.julia\packages\CUTEst\FWOg3\src\CUTEst.jl:77

@amontoison This is not directly related to the issue discussed here but prevents further tests. Are you sure that CUTEst has been tested on Windows? I can temporarily disable using CUTEst with PRIMA.

emmt commented 9 months ago

@amontoison do you have any suggestions?

amontoison commented 9 months ago

Hi @emmt, sorry for not answering earlier. I will have a look this week.

CUTEst was never tested on Windows. I also need to check if artifacts of CUTEST_jll.jl were generated for Windows. I remember that we got issues on Windows because CUTEST generates an incomplete libary (by construction) and that we can't have undefined symbols in shared library on this platform.

emmt commented 9 months ago

Hi @amontoison, no problems. We should then try to figure out how to skip CUTEst tests (and installation) on Windows.

emmt commented 7 months ago

Since no solutions have emerged, I propose to split the package in 2 one, PRIMA.jl, with the interface to libprima but without support for CUTEst, the other (say PRIMACUTEst.jl) focused on using PRIMA with CUTEst. In that way we can run the CI tests depending on the type of machine and operating system. @amontoison and @zaikunzhang what do you think?

amontoison commented 7 months ago

CUTEst is wrapped as an NLPModels so we should not modify PRIMA.jl. Note that CUTEst was never officially on Windows.

You can just use Sys.iswindows() to verify if we are on windows or not and run only a part of the tests.

emmt commented 7 months ago

The problem is not to skip the tests but that CUTEst is part of the dependencies and the CI attempts to install this package (even though it is not used) which fails on Windows. I could not figure out a way to have system-dependent dependencies. Hence my proposition to split in two packages and only testing the second one on 64-bit Unix-like machines.

amontoison commented 7 months ago

Can you open an issue in CUTEst.jl and tag me? I didn't know that the issue was that. I thought that you were just unable to use libcutest.dll on Windows wirh the artifact CUTEst_jll.jl.

emmt commented 7 months ago

@amontoison Sure I can do that (done here) but since CUTEst.jl depends on CUTEst_jll.jl and since CUTEst_jll.jl is not available for Windows, I do not see how can you deal with this in CUTEst.jl except if the Windows version of CUTEst_jll.jl is just an empty DLL or something like that.

amontoison commented 7 months ago

CUTEst_jll.jl can be compiled on Windows. We are just unable to generate a shared library because it's incomplete by construction and on Windows we can't have undefined symbols. I can just export a FileProduct instead of a LibraryProduct in build_tarballs.jl.

emmt commented 1 month ago

This should be solved by version 0.2.1 (see 395668450e14fda0eeec5d7c973ce602f843a0ae) so I close the issue.