Closed pramodk closed 1 year ago
There are two installer messagebox commands that can be commented out to see if that helps
MessageBox MB_OKCANCEL "${NEURON} already installed at $R1. Press OK to install again." IDOK ok
and
MessageBox MB_OK "The installation path is not allowed to have spaces."
but for the latter you may also need to reduce the line skip from 3 to 2 in
StrCmp $R0 "" +3 ; if no spaces then go on
I started Windose Server 2019 on AWS to test the installer:
And I am able to run it in silent mode without any issue:
Also I went to AppVeyor for quickly testing installer on their windows CI and using following configuration:
install:
- cmd: dir .
- cmd: echo "Starting ==========>"
- cmd: .\nrn8.0setup-AMD64.exe
- cmd: echo "Finished ==========>"
- cmd: dir C:\nrn
We see this log where installation finishes without any issue:
echo "Starting ==========>"
61"Starting ==========>"
62.\nrn8.0setup-AMD64.exe
63echo "Finished ==========>"
64"Finished ==========>"
65dir C:\nrn
66 Volume in drive C is Windows
67 Volume Serial Number is D4AB-4044
68
69 Directory of C:\nrn
70
7105/24/2020 12:24 PM <DIR> .
7205/24/2020 12:24 PM <DIR> ..
7305/24/2020 12:24 PM <DIR> bin
7405/24/2020 12:24 PM <DIR> demo
7505/24/2020 12:24 PM <DIR> include
7605/24/2020 12:24 PM <DIR> lib
7705/24/2020 12:24 PM <DIR> mingw
7805/23/2020 10:34 PM 6,010 notes.txt
7905/24/2020 12:24 PM <DIR> share
8005/23/2020 10:34 PM 124 test0.hoc
8105/23/2020 10:34 PM 594 test0.py
8205/24/2020 12:24 PM 44,174 uninstall.exe
83 4 File(s) 50,902 bytes
84 8 Dir(s) 43,851,468,800 bytes free
85echo "End ==========>"
86"End ==========>"
87Discovering tests...OK
88
So this issue is something very specific to Azure CI. Sometime later I will post the issue to Azure CI support. I will go ahead with finalising other parts of CI before revisiting this issue.
Nice that you got so far with the diagnosis. I wasted the morning looking into nsis logging but in my hands I could not get the nsislog plugin to work at all on my machine. I installed nsis-3.05-log.zip , a special build that enables logging support, also on my machine, but the first line in the log file is
CreateDirectory: "c:\nrn" created
which looks to be too late for any useful diagnosis.
Also I went to AppVeyor for quickly testing installer on their windows CI and using following configuration. We see this log where installation finishes without any issue.
I was wrong in above analysis! I was using an installer without any MessageBox
in .nsi
file and that's why it was successful on AppVeyor CI. I used the same installer on Azure and it also worked. On the other hand, if I use original installer with MessageBox
usage, it doesn't work on Azure as well as AppVeyor.
So, it seems like our installer is not truly
Silent. I came across an example on nsis documentation : https://nsis.sourceforge.io/Examples/silent.nsi.
I believe we need to use those extra documented flags to make our installer Silent
compatible.
I don’t know nsi script well. @nrnhines : Could you help with updating nsi script?
Above issue is fixed in #562. Installer is now working fine in Silent mode! 🎉
Bringing back unanswered question from #562 :
In this test log I see that only Python3.8 test is failing with below error:
Traceback (most recent call last):
File "C:\nrn\lib\python\neuron\__init__.py", line 124, in <module>
import hoc
ModuleNotFoundError: No module named 'hoc'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\nrn\lib\python\neuron\__init__.py", line 128, in <module>
import neuron.hoc
ImportError: DLL load failed while importing hoc: The specified module could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\nrn\lib\python\neuron\__init__.py", line 130, in <module>
exec("import neuron.hoc%d%d as hoc" % (sys.version_info[0], sys.version_info[1]))
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'neuron.hoc38'
##[error]Cmd.exe exited with code '1'.
Python 2 and 3.5/6/7 working fine. You can see in the build log, all pythons are correctly initialised.
Any hint / suggestions? Here is the installer created from Azure CI that one can install locally to reproduce the issue : https://github.com/pramodk/nrn/releases/download/nightly/nrn8.0setup-AMD64.exe
This is a puzzle and occurs also when I install from a setup.exe built on my virtualbox guest. There are many possible user environments. On mine the default python when I start the bash terminal from the NEURON program group is
bash-4.4$ which python
/cygdrive/c/Users/hines/Anaconda3/python
bash-4.4$ echo $PYTHONHOME
C:/Users/hines/Anaconda3
bash-4.4$ echo $NRN_PYLIB
C:/Users/hines/Anaconda3/python36.dll
bash-4.4$
So my first step is
bash-4.4$ export PATH=/cygdrive/e/python38:$PATH
bash-4.4$ nrnpyenv.sh
# Ignoring existing PYTHONHOME=C:/Users/hines/Anaconda3.
...
# if launch nrniv, then likely need:
export PYTHONHOME="E:/python38"
export NRN_PYLIB="E:/python38/python38.dll"
so with copy/paste I have good success when launching nrniv
bash-4.4$ export PYTHONHOME="E:/python38"
bash-4.4$ export NRN_PYLIB="E:/python38/python38.dll"
bash-4.4$ nrniv -python
NEURON -- VERSION 8.0.dev-103-g9426d754 nrnmech_fix (9426d754) 2020-05-26
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
See http://neuron.yale.edu/neuron/credits
>>> import neuron
>>> quit()
Unfortunately, I get the same problem as you when I launch python
bash-4.4$ python
Python 3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:37:50) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import neuron
Traceback (most recent call last):
File "C:\nrn\lib\python\neuron\__init__.py", line 124, in <module>
import hoc
ModuleNotFoundError: No module named 'hoc'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\nrn\lib\python\neuron\__init__.py", line 128, in <module>
import neuron.hoc
ImportError: DLL load failed while importing hoc: The specified module could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\nrn\lib\python\neuron\__init__.py", line 130, in <module>
exec("import neuron.hoc%d%d as hoc" % (sys.version_info[0], sys.version_info[1]))
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'neuron.hoc38'
>>>
Despite the existence of
bash-4.4$ ls c:/nrn/lib/python/neuron/hoc*
c:/nrn/lib/python/neuron/hoc.cp35-win_amd64.pyd c:/nrn/lib/python/neuron/hoc.cp38-win_amd64.pyd
c:/nrn/lib/python/neuron/hoc.cp36-win_amd64.pyd c:/nrn/lib/python/neuron/hoc.pyd
c:/nrn/lib/python/neuron/hoc.cp37-win_amd64.pyd
I will continue to look into this.
@nrnhines : Instead of using exe that I provided, could you check if you see same issue if you build installer with Python 3.8.2 locally and install it?
It would be helpful to identify if this is an issue with my azure CI setup or appears with local setup as well.
That is what I did. And when I install on another vbox guest that has only 3.8.3 the problem remains. I noticed the following but have not thought about it sufficiently to know if it is relevant. https://docs.python.org/3/whatsnew/3.8.html
DLL dependencies for extension modules and DLLs loaded with ctypes on Windows are now resolved more securely. Only the system paths, the directory containing the DLL or PYD file, and directories added with add_dll_directory() are searched for load-time dependencies. Specifically, PATH and the current working directory are no longer used, and modifications to these will no longer have any effect on normal DLL resolution. If your application relies on these mechanisms, you should check for add_dll_directory() and if it exists, use it to add your DLLs directory while loading your library. Note that Windows 7 users will need to ensure that Windows Update KB2533623 has been installed (this is also verified by the installer). (Contributed by Steve Dower in bpo-36085.)
Here is a big hint relevant to the 'dll dependency' quote above. When I copy all c:/nrn/bin/*.dll into the neuron module, 'import neuron' works.
Specifically, PATH and the current working directory are no longer used, and modifications to these will no longer have any effect on normal DLL resolution. If your application relies on these mechanisms, you should check for add_dll_directory() and if it exists,
Ah ok! that seems very relevant!
os.add_dll_directory fixes the issue. A pull request will be forthcoming shortly.
great that fixes the issue!
@nrnhines : could you guide me how to test nrnivmodl and nrniv on Windows?
Could not load libnrnpython...
.In a command prompt, you should be able to
C:\Users\hines>mpiexec -n 2 nrniv -mpi c:/nrn/test0.hoc
NEURON -- VERSION 8.0.dev-102-gc2265c65 dll-resolution (c2265c65) 2020-05-27
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
See http://neuron.yale.edu/neuron/credits
numprocs=2
I am 1 of 2
I am 0 of 2
The problem with running neuron.test() is that windows generates the error (@adamjhn ?)
C:\Users\hines>set PATH=e:\python38;%PATH%
C:\Users\hines>nrniv -python -c "import neuron ; neuron.test() ; quit()"
NEURON -- VERSION 8.0.dev-102-gc2265c65 dll-resolution (c2265c65) 2020-05-27
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
See http://neuron.yale.edu/neuron/credits
testBytesize (neuron.tests.test_vector.VectorTestCase)
Test that Vector.__array_interface__ returns the proper bytesize (of a double) ... ok
...
testExtendedSection (neuron.tests.test_neuron.NeuronTestCase) ... NEURON -- VERSION 8.0.dev-102-gc2265c65 dll-resolution (c2265c65) 2020-05-27
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
See http://neuron.yale.edu/neuron/credits
c:\nrn\bin\nrniv.exe: syntax error
near line 1
from multiprocessing.spawn import spawn_main; spawn_main(parent_pid=7460, pipe_handle=500)
^
c:\nrn\bin\nrniv.exe: arg not valid statement: from multiprocessing.spawn import spawn_main; spawn_main(parent_pid=7460, pipe_handle=500)
near line 0
^
0 c:\nrn\bin\nrniv.exe: can't open --multiprocessing-fork
A test of association with hoc files is possible in a command prompt with the following temp.bat file
del temp.txt
echo wopen("temp.txt") > .\temp.hoc
echo fprint("hello\n") >> .\temp.hoc
echo wopen() >> .\temp.hoc
echo quit() >> .\temp.hoc
start .\temp.hoc
timeout /T 10
cat temp.txt
I was worried about using start /wait instead of the timout since if there is a hoc failure the temp.hoc will hang.
c:\nrn\bin\nrniv.exe: arg not valid statement: from multiprocessing.spawn import spawn_main; spawn_main(parent_pid=7460, pipe_handle=500) near line 0 ^ 0 c:\nrn\bin\nrniv.exe: can't open --multiprocessing-fork
No knowledge of windows but this looks related : https://blender.stackexchange.com/questions/8530/how-to-get-python-multiprocessing-module-working-on-windows
A test of association with hoc files is possible in a command prompt with the following temp.bat file
I have to understand what this test is doing :)
What about nrnivmodl? how we can test that compiling environment is correct and nrnivmodl is working fine?
The hoc association test needs a hoc file and that is created by the echo lines. The file is launched with start which runs neuron.exe (hoc extension is associated with that) which opens a bash terminal in a new window and runs nrniv. The quit() at the end of the hoc file is needed or nrniv would just wait for user input. The temp.hoc writes to a file temp.txt in order to produce something that persists after nrniv exits and the bash terminal window closes.
A test of mknrndll just needs a folder with a mod file and a py file that creates a section and inserts the mechanism to verify that the dll is working. e.g foo.mod
NEURON {
SUFFIX foo
RANGE a
}
PARAMETER { a }
foo.py
from neuron import h
s = h.Section()
s.insert('foo')
quit()
and test with
cd folder/containing/mod/and/py/files
mknrndll
python foo.py
#check $?
Let's work around the multiprocessing bug on windows by avoiding running neuron.test() with nrniv and instead only check the python launch. eg. in a command prompt
C:\Users\hines>set PATH=e:\python38;%PATH%
C:\Users\hines>python -c "import neuron;neuron.test()"
testBytesize (neuron.tests.test_vector.VectorTestCase)
Test that Vector.__array_interface__ returns the proper bytesize (of a double) ... ok
testEndian (neuron.tests.test_vector.VectorTestCase)
...
Test section, segment, mechanism, rangevar iterators. ... ok
testRxDexistence (neuron.tests.test_neuron.NeuronTestCase) ... scipy not available
ok
testpsection (neuron.tests.test_neuron.NeuronTestCase) ... ok
----------------------------------------------------------------------
Ran 11 tests in 6.251s
OK
C:\Users\hines>
No knowledge of windows but this looks related : https://blender.stackexchange.com/questions/8530/how-to-get-python-multiprocessing-module-working-on-windows
I was not successful with
nrniv -python
import sys
sys.executable = 'e:\\python38\\python.exe'
import neuron
neuron.hoc.__file__= 'c:\\nrn\\lib\\python\\neuron\\hoc.cp38-win_amd64.pyd'
neuron.test()
But I was unclear about what they meant by __file__
.
A test of mknrndll just needs a folder with a mod file and a py file that creates a section and inserts the mechanism to verify that the dll is working. e.g foo.mod
To test mknrndll
, am I supposed to launch mknrndll
from bash and not from cmd itself?
Currently I am doing:
C:\Users\kumbhar> C:\nrn\mingw\usr\bin\bash -c "export N=/cygdrive/c/nrn && mknrndll"
I see that mknrndll
rely on N
env var and I needed to define that if I am starting bash from cmd. It works fine if launch bash shortcut by double click. (something trivial about env variables sourcing I guess).
I see that it generates dll
-rw-r--r-- 1 kumbhar None 53 May 30 01:44 test.mod
-rw-r--r-- 1 kumbhar None 389 May 30 01:48 mod_func.c
-rw-r--r-- 1 kumbhar None 1264 May 30 01:48 mod_func.o
-rw-r--r-- 1 kumbhar None 6399 May 30 01:48 test.c
-rw-r--r-- 1 kumbhar None 4815 May 30 01:48 test.o
-rwxr-xr-x 1 kumbhar None 55320 May 30 01:48 nrnmech.dll
and now I can do:
C:\Users\kumbhar>C:\Python37\python.exe
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import neuron
>>> from neuron import h
>>> s = h.Section()
>>> s.insert('foo')
__nrnsec_0000021495dfd550
So, there is no special created like linux/osx?
To test mknrndll, am I supposed to launch mknrndll from bash and not from cmd itself?
I realize I've never done this except from the mknrndll shortcut or from the bash terminal popped up from the bash shortcut. You are correct that bash with a bit of extra environment is needed. Also when I start a command prompt, I see that the %PATH% is incomplete and I needed
set PATH=%PATH%;c:\nrn\mingw\mingw64\bin
although I hate to add more things to the PATH it does not work to copy the few files from mingw/mingw64/bin into mingw/usr/bin as then gcc cannot find cc1
So, there is no special created like linux/osx?
That is correct.
Just as a note : sourceforge download of NSIS doesn't seem robust and fails sometime (3-5% builds):
nsis.install v3.05 [Approved]
nsis.install package files install completed. Performing other installation steps.
Attempt to get headers for https://svwh.dl.sourceforge.net/project/nsis/NSIS%203/3.05/nsis-3.05-setup.exe failed.
The remote file either doesn't exist, is unauthorized, or is forbidden for url 'https://svwh.dl.sourceforge.net/project/nsis/NSIS%203/3.05/nsis-3.05-setup.exe'. Exception calling "GetResponse" with "0" argument(s): "Unable to connect to the remote server"
Downloading nsis.install
from 'https://svwh.dl.sourceforge.net/project/nsis/NSIS%203/3.05/nsis-3.05-setup.exe'
ERROR: The remote file either doesn't exist, is unauthorized, or is forbidden for url 'https://svwh.dl.sourceforge.net/project/nsis/NSIS%203/3.05/nsis-3.05-setup.exe'. Exception calling "GetResponse" with "0" argument(s): "Unable to connect to the remote server"
This package is likely not broken for licensed users - see https://chocolatey.org/docs/features-private-cdn.
The install of nsis.install was NOT successful.
Error while running 'C:\ProgramData\chocolatey\lib\nsis.install\tools\chocolateyInstall.ps1'.
See log for details.
~I will upload installer to some GitHub repo from it can be fetched without failure.~ I will download from https://fossies.org/windows/misc/nsis-3.05-setup.exe/
Additional notes:
C:\Users\kumbhar\nrn>C:\nrn_test\mingw\usr\bin\bash -c "mknrndll" || goto :error
/usr/bin/bash: /cygdrive/c/nrn_test/bin/mknrndll: sh^M: bad interpreter: No such file or directory
Fixing the end line terminator.
mknrndll
gave:C:\Users\kumbhar\nrn>C:\nrn_test\mingw\usr\bin\bash -c "mknrndll" || goto :error
set: bad interpreter: No such file or directoryl: sh
Needed to do set PATH=C:\nrn_test\mingw\usr\bin;%PATH%
Needed to do set PATH=C:\nrn_test\mingw\usr\bin;%PATH%
I don't doubt this but it is strange as that path item was set during installation (by default) as was c:\nrn\bin. What was really missing is c:\nrn\mingw\mingw64\bin to get gcc. All this was done in the startup script for the bash terminal icon. ie in c:/nrn/lib/bshstart.sh
Fixing the end line terminator.
I don't understand this. Isn't the problem just with the sh being in c:/nrn/mingw/usr/bin
On my virtual box windows 10 guest with full msys2 development and an installation of neuron. From a command prompt, I see
C:\Users\hines>echo %PATH%
...;c:\nrn\bin;c:\nrn\mingw\usr\bin
Then, executing bash. I get
C:\Users\hines>bash
bash-4.4$ echo $PATH
...:/cygdrive/c/nrn/bin:/usr/bin
and this really is my neuron installed bash since ls /usr/bin only shows the items in c:\nrn\mingw\usr\bin
It turns out you can use the bshstart.sh start script but you must start in the install folder. e.g.
C:\Users\hines>cd c:\nrn
c:\nrn>bash --rcfile c:/nrn/lib/bshstart.sh
bash-4.4$ echo $PATH
/cygdrive/C/Users/hines/Anaconda3/Library/bin:/cygdrive/c/nrn/bin:/cygdrive/c/nrn/mingw/usr/bin:/cygdrive/c/nrn/mingw/mingw64/bin:/cygdrive/c/ms-mpi/Bin:/cygdrive/c/WINDOWS/system32:/cygdrive/c/WINDOWS:/cygdrive/c/WINDOWS/System32/Wbem:/cygdrive/c/WINDOWS/System32/WindowsPowerShell/v1.0:/cygdrive/c/WINDOWS/System32/OpenSSH:/cygdrive/c/Users/hines/AppData/Local/Programs/Python/Launcher:/cygdrive/c/Users/hines/Anaconda3:/cygdrive/c/Users/hines/Anaconda3/Scripts:/cygdrive/c/Users/hines/Anaconda3/Library/bin:/cygdrive/c/Users/hines/AppData/Local/Microsoft/WindowsApps:/cygdrive/c/Users/hines/AppData/Local/Programs/Microsoft VS Code/bin:/cygdrive/c/nrn/bin:/usr/bin
Notice the addition of the three paths again near the beginning of the PATH. From here you can go to a folder containing mod files and run mknrndll. e.g.
bash-4.4$ rm *.dll
bash-4.4$ mknrndll
x86_64-w64-mingw32-gcc.exe -DDLL_EXPORT -DPIC -I/cygdrive/c/nrn/include -I/cygdrive/c/nrn/src/scopmath -I/cygdrive/c/nrn/src/nrnoc -I/cygdrive/c/nrn/src/oc -c mod_func.c
nocmodl cabpump
Translating cabpump.mod into cabpump.c
Notice: Assignment to the GLOBAL variable, "vol", is not thread safe
x86_64-w64-mingw32-gcc.exe -DDLL_EXPORT -DPIC -I/cygdrive/c/nrn/include -I/cygdrive/c/nrn/src/scopmath -I/cygdrive/c/nrn/src/nrnoc -I/cygdrive/c/nrn/src/oc -c cabpump.c
...
x86_64-w64-mingw32-gcc.exe -shared mod_func.o cabpump.o cachan1.o camchan.o capump.o invlfire.o khhchan.o mcna.o nacaex.o nachan.o release.o \
-L/cygdrive/c/nrn/bin -lnrniv -lpthread -o 1464.nrnmech.dll
#rebase -b 0x64000000 -v nrnmech.dll
bash-4.4$
On Windows, starting with NEURON 7.7, you can type nrnivmodl
from the cmd prompt and that should do the right thing with telling bash to tell mknrndll
to do the right thing.
At least that doesn't seem to be the case with CMake installation on windows : I see nrnivmodl in bin directory but that is still shell script :
The file itself is in src/mswin/bin/nrnivmodl.bat
My guess is that maybe the windows-specific stuff isn't being put into the right place by the cmake version?
Here's the commit that added the bat version: https://github.com/neuronsimulator/nrn/commit/88b2fa546ef98ba00c1cc8e0f4857743362ff71b
I notice that src/mswin/batch/instalmingw.sh
had to explicitly include a copy command for the nrnivmodl.bat
file.
Ah Ok! I haven't worked on this part. I will let @nrnhines to make necessary changes if this is something we should fix.
I completely forgot about pull request #142. I will add that autotools addition to cmake. I.e nrnivmodl as a copy of mknrndll and the new (then), nrnivmodl.bat. I guess for reduced confusion, mknrndll.bat should be also added as a copy of nrnivmodl.bat .
A test of association with hoc files is possible in a command prompt with the following temp.bat file
del temp.txt
echo wopen("temp.txt") > .\temp.hoc
echo fprint("hello\n") >> .\temp.hoc
echo wopen() >> .\temp.hoc
echo quit() >> .\temp.hoc
start .\temp.hoc
timeout /T 10
cat temp.txt
I was worried about using start /wait instead of the timout since if there is a hoc failure the temp.hoc will hang.
@nrnhines : I noticed that even if I use start /wait
or start /wait /b
(as suggested here) doesn't work i.e. we resume to next command (timeout in this case) right away. This works but not 100% robust.
Since it is not robust, but perhaps usually correct, perhaps it can become a warning to us just to check it ourselves if it fails? But is never a failure from the perspective of the azure CI.
For now I have enabled that as failure. If we see this result into multiple build failures, we can change this default behaviour as non-fatal.
This issues is to document process to build Windows instructions.
Here are brief steps (WIP):
Install latest Visual Studio
Install msys2
Look into https://github.com/neuronsimulator/nrn/blob/master/mingw_files/msys2
Installed msmpisdk.msi and MsMpiSetup.exe in c:\ms-mpi (version 10.0)
version 10 Lib is already lib
Change Bin, Lib, Include to bin, lib, include
cp c:/Windows/System32/msmpi.dll c:/ms-mpi/lib/x64 cp c:/Windows/SysWoW64/msmpi.dll c:/ms-mpi/lib/x86 #optional
stdint.h now in version 10 so next not needed.
add #include in /c/ms-mpi/include/mpi.h
important. Wasted a day because I did not do this. the symptom is a
crash in libnrnmpi.dll
I don't know if this is needed anymore or not for version 10 as
libmsmpi.a does exist.
gendef msmpi.dll # creates msmpi.def
x86_64-w64-mingw32-dlltool -d msmpi.def -l libmsmpi.a -D msmpi.dll
dlltool -d msmpi.def -l libmsmpi.a -D msmpi.dll # creates libmsmpi.a
gendef can be built from mingw source via
git clone https://github.com/msys2-contrib/mingw-w64
cd mingw-w64/mingw-w64-tools/gendef
./configure --prefix=
pwd
make # may have to get rid of the -Werror
cp gendef.exe /mingw64/bin