search

volcic / motom-toolbox

GNU Affero General Public License v3.0
6 stars 0 forks source link

Installation issue - oapi.dll & oapi64.dll #1

Closed MfMartel closed 5 years ago

MfMartel commented 5 years ago

Hi there,

I have some issue with the toolbox installation. When running the RUNME script, it seems that it can not find the oapi64.dll. The problem is that I don't have such file in my Optotrak API files. I tried to reinstall oapi, hoping that there would be the possibility to also install a 64-bit version so I can have the oapi64.dll, but there is none.

The lab computer is a win64 (Windows 7), with a 32-bit Matlab 2015a. And we have the Optotrak Application Programmer's Interface v3.7.1 for 32-bit Windows platforms. But apparently it works despite having a win64 system.

Later in the script (l. 83), when it is supposed to select the version of the files (exist('use_64_bits', 'file') == 2), the result is 0 suggesting that it will go for the 32-bit stuff instead (hence the oapi.dll that I have).

My question is the following: can I safely modify the script so that it doesn't look for the oapi64.dll? Or do we need it for another function? Should I rather see with NDI directly so we can also have a 64-bit version?

Many thanks! And sorry for the maybe easy question, but I'm just starting with all this. Marie

ha5dzs commented 5 years ago

Hi, Zoltan here.

The part that checks for the API files are currently hard-coded, so right now the simplest option would be to create an empty file named oapi64.dll in the bin directory. Nothing else will use the file in a 32-bit environment, so this is safe to do so.

But, please be aware that I could not test the toolbox with a this old OAPI, so if you do run into issues, you may want to consider getting the 3.14 or 3.15 versions, which do have the 64-bit dll-s, and are being used in our lab every day.

Please do keep me updated whether the toolbox works with your old API, and then I can modify RUNME.m accordingly (effectively, the section between line 45 and 69).

MfMartel commented 5 years ago

Hi Zoltan,

Thanks for the quick reply, the trick did the work. Our version of OAPI is maximum 2-3 years old, so I hope there won't be too much changes...

When I run it, it works up to the point where "the toolbox has been added to the Matlab path. To make this permanent, click yes at the next prompt if it pops up" (it doesn't pop out though). I do have a bunch of warnings about an lcc preprocessor error and some other error in loadlibrary. Are those the warning messages you mention in the wiki that are safe to ignore?? The issue is in the end, I have an error with the mex file in the section: compile helper functions written in C. It seems that this times it finds the compiler "lcc-win32" which is not the one used at the beginning (we have Microsoft Windows SDK 7.1 (C++)). But I can not find anything in the code that suggests that it is looking for a different compiler from the one it found originally so I don't understand what is going on.

Funny thing also, if now I rerun the script without closing and opening Matlab again, this time it doesn't work and it can not find the oapi64.dll (although it's there). If I close Matlab and starts again, this error doesn't happen...

Thanks for the help!

ha5dzs commented 5 years ago

Well, I did see some changes when we upgraded from 3.14 to 3.15. For instance, the 3.14 API didn't handle many rigid bodies simultaneously loaded which was fixed in 3.15. However, the 3.15 version introduced an issue with the registration process, and NDI is looking into it for a couple of months now.

The warnings you see in loadlibrary should be about discrepancies between functions that are in the header file and the included DLL, and some weirdly defined pointers, which are also not being used. This is because the header files have been cpoied and pasted together from NDI's internal code, so we mere mortals are only allowed to see a tiny fraction of what they have. These warnings are safe to ignore, because nothing will call the missing functions anyway. But, if you are curious, please post the warnings here, and I'll do my best to justify each and every one of them. :)

With respect to compiling the C-code, based on what you've described, I think the script fails because the compiler is not recognised by the toolbox. Personally, I have not used this lcc-win32 compiler before, so I don't know its quirks. I got the toolbox to compile without errors with various versions of Visual Studio's compilers, MinGW (the one Microsoft bundles, and not the one you download from mingw.org), and g++ on Linux.

However, adding support to your compiler is not difficult. If you look at compilers.m, you can see how you can add your compiler at line 14. Additionally, there is some docs available about this too. At the very least, you need to specify the name of the compiler, and with a bit of luck it will work with the default settings. I tried to make this process as painless as possible because I anticipated people using it on totally different systems. Again, please keep me updated, so I can incorporate the compiler information in an update, so other people can suffer a bit less! :)

https://github.com/volcic/motom-toolbox/wiki/Damage-Control-FAQ#i-am-using-something-exoticold-how-to-add-my-own-c-compiler

So I would check what compiler you've got in your system with mex -setup C++, select the one you want to use if you have more than one, and then edit compilers.m accordingly.

MfMartel commented 5 years ago

Hi,

Thanks! No worries about the warnings ;)

As for the compiler, that's exactly what I did already. I changed the Name to Microsoft Windows SDK 7.1 (C++) and the version to 7.1. This compiler is the one found when I type mex -setup C++. And it works for almost the whole script. But still at the end, the one recognized is lcc-win32 and the last part of the script can not run...

Could it be linked to the compiler_flags ? It seems that it is used in the last part of the RUNME.m script, and I haven't specified anything in the compilers.m part... But maybe I do need to?

ha5dzs commented 5 years ago

Now this is a tad puzzling. So once it found the correct compiler, all RUNME.m does is that it assembles a string for each included C-file, and then executes it. In your case, this will be line 129 of RUNME.m. The compiler_flags string should be empty, and even if it is something there, it just appends the extra flags to the $COMPFLAGS variable, which, if set incorrectly or just sets nothing, are going to be ignored. For example, there are no extra flags associated with the Windows 10 SDK-bundled MinGW compiler, but there is /O2 /Wall(optimise for execution speed, and display all info/warnings) set for the Visual Studio compilers. The compilation is done with Matlab's mex command, so it's as 'stock' Matlab method as possible, like so: mex -v COMPFLAGS="$COMPFLAGS <plus custom flags, if any>" <path_to_c_file> -l../bin/oapi.lib

Same with loadlibrary, it's as per textbook. The checks in RUNME.m and compilers.m are there to make sure there is a supported C-compiler, and tries to provide meaningful error messages when something is missing or wrong.

Can you paste here some output? It might well be that you need some flag or you need to include or link something that wasn't done before, and with a bit of luck, it will be in the error message.

Also, can you verify, that Matlab's selected C-compiler is Microsoft Windows SDK 7.1 (C++) before and after the execution of RUNME.m? It is very strange that it just changes arbitrarily. Maybe there will be a clue in the output.

MfMartel commented 5 years ago

Hi,

I checked before and after running RUNME.m and mex -setup c++ still returns the right compiler Microsoft Windows SDK 7.1 (C++).

Here is my output:

This is the part making me think there is something wrong with the compiler since it doesn't select the Microsoft one and found the lcc one, but maybe I'm wrong and it's normal?

> Compiling DataGetLatest3D_as_array.c:
Verbose mode is on.
Neither -compatibleArrayDims nor -largeArrayDims is selected.
     Using -compatibleArrayDims. In the future, MATLAB will require the use of
     -largeArrayDims and remove the -compatibleArrayDims option.
     For more information:
     http://www.mathworks.com/help/matlab/matlab_external/upgrading-mex-files-to-use-64-bit-api.html.
... Looking for compiler 'lcc-win32' ...
... Looking for folder 'C:\Program Files (x86)\MATLAB\R2015a\sys\lcc' ...Yes.
Found installed compiler 'lcc-win32'.
Set PATH = C:\Program Files (x86)\MATLAB\R2015a\sys\lcc\bin;C:\Program Files (x86)\MATLAB\R2015a\sys\lcc\include;C:\Program Files (x86)\MATLAB\R2015a\sys\lcc\mex;C:\Program Files (x86)\MATLAB\R2015a\extern\lib\win32\lcc;C:\Program Files (x86)\MATLAB\R2015a\extern\include\win32\;C:\Program Files (x86)\MATLAB\R2015a\extern\include;C:\Program Files (x86)\MATLAB\R2015a\simulink\include;C:\Program Files (x86)\MATLAB\R2015a\lib\win32;C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS Client\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;c:\Program Files (x86)\ATI Technologies\ATI.ACE\Core-Static;C:\ndigital\programs;C:\ndigital\drivers;C:\Program Files\MATLAB\R2017a\bin;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files (x86)\Microsoft SDKs\TypeScript\1.0\;C:\Program Files\Microsoft SQL Server\120\Tools\Binn\;C:\Program Files (x86)\MATLAB\R2015a\runtime\win32;C:\Program Files (x86)\MATLAB\R2015a\bin;C:\Program Files\SlikSvn\bin;C:\Program Files\Microsoft Windows Performance Toolkit\
Set INCLUDE = C:\Program Files (x86)\MATLAB\R2015a\extern\include;;
Set LIB = C:\Program Files (x86)\MATLAB\R2015a\lib\win32;C:\Program Files 
(x86)\MATLAB\R2015a\extern\lib\win32\lcc;;
Set LIBPATH = C:\Program Files (x86)\MATLAB\R2015a\extern\lib\win32\lcc;C:\Program Files (x86)\MATLAB\R2015a\sys\lcc\lib\;C:\Program Files (x86)\MATLAB\R2015a\sys\lcc\mex;

This is the error:

> Error using mex
> MEX cannot find library '../bin\oapi' specified with the -l option.
>  MEX looks for a file with one of the names:
>  ../bin\oapi.lib
>  lib../bin\oapi.lib
>  Please specify the path to this library with the -L option.
> 
> Error in RUNME (line 130)
>             eval(compiler_string);

It seems to appear with these lines of command, hence the question about compiler_flags

compiler_string = sprintf('mex -v COMPFLAGS="$COMPFLAGS %s" %s -l../bin/oapi.lib', compiler_flags, file_string);
eval(compiler_string);

Thank you!

ha5dzs commented 5 years ago

This is strange, it switches back to its internal compiler. Can you compile this piece of C-code on your computer? Just copy and paste this, save it as test.c

#include "mex.h"

// matlab's gateway function.
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, mxArray *prhs[])
{
    //message.
    mexPrintf("It works!!!\n");
    mexEvalString("drawnow");
    plhs[0] = 0; //return 0
}

It should compile in Matlab by going to the directory where the file is, and executing mex -v test.c. And when you type in test, it should give the following output:

>> test
It works!!!

Can you check if it uses the Microsoft C-compiler at all? I can't see why it keeps using Matlab's internal one, when you have a different compiler selected. I noticed from the warnings that you may want to add -largeArrayDims into your compiler flags, but it's not mandatory, and the warning will probably go away when switching to a different compiler.

I think I can see the problem though: with lcc, the compilation fails because it can't find the oapi.lib file at the given path. Can you try changing the -l../bin/oapi.lib to -l..\bin\oapi.lib in RUNME.m line 139? Hopefully that will be the end of it. I haven't had this problem, but it's easily possible that this compiler doesn't like forward slashes in the path, only backslashes. If this doesn't cure the problem, you may need the toolbox' bin directory's path added into $LIBPATH, and change linking text from -l../bin/oapi.lib to -loapi

MfMartel commented 5 years ago 
Building with 'lcc-win32'.
lcc -c -noregistrylookup -DMX_COMPAT_32   -DMX_COMPAT_32  -DMATLAB_MEX_FILE -I"C:\Program Files (x86)\MATLAB\R2015a\sys\lcc\include" -I"C:\Program Files (x86)\MATLAB\R2015a\extern\include" -I"C:\Program Files (x86)\MATLAB\R2015a\simulink\include" -I"C:\Program Files (x86)\MATLAB\R2015a\sys\lcc\mex"   "D:\OPTO 2\Documents\MATLAB\motom-toolbox-master\test.c"  -DNDEBUG /Fo"D:\OPTO 2\AppData\Local\Temp\mex_33822047130656_7044\test.obj"
Error using mex
Error D:\OPTO 2\Documents\MATLAB\motom-toolbox-master\test.c: 5  redeclaration of `mexFunction' previously declared at
C:\Program Files (x86)\MATLAB\R2015a\extern\include\mex.h 138
1 errors, 0 warnings

But when coming back to the warnings I had before, if I run lines 77-90, one of the warnings is about lcc. Do you think it could be linked?

Warning: Message from C preprocessor:
lcc preprocessor error: D:\OPTO 2\Documents\MATLAB\motom-toolbox-master\generated_binaries\..\source\ndhost.h:157 D:\OPTO
2\Documents\MATLAB\motom-toolbox-master\generated_binaries\..\source\ndtypes.h:7 D:\OPTO
2\Documents\MATLAB\motom-toolbox-master\generated_binaries\..\source\ndopto.h:23 #error directive: NDI Error - Host not
defined. Options: PLATFORM_X86, PLATFORM_LINUX
lcc preprocessor error: D:\OPTO 2\Documents\MATLAB\motom-toolbox-master\generated_binaries\..\source\ndpack.h:25 D:\OPTO
2\Documents\MATLAB\motom-toolbox-master\generated_binaries\..\source\ndopto.h:24 #error directive: You must define the type
of system you are using...

I'm not so sure how to modify $LIBPATH since it is only an output, but nowhere in the script...

ha5dzs commented 5 years ago

Okay, the clue is in the second output you linked:

..\source\ndopto.h:23 #error directive: NDI Error - Host notdefined. Options: PLATFORM_X86, PLATFORM_LINUX

This happens because there is no declaration in the API header file for this compiler. (I got a similar error message when I tried to compile this on Mac, just for a laugh)

Having looked at the source files, it has defines for Microsoft Visual C++ compilers and Borland's C-compiler. Even if you go past this, and edit ndhost.h and add #define PLATFORM_X86, it will most probably fail, because it doesn't recognise the compiler type, and it can't load the platform-specific settings. I had this happening with MinGW.

I am afraid we have exhausted all the options we had. I don't think it is possible to get this compiler to work. Can you install Visual Studio 2015 with C++ support and select it in Matlab? Then you can check if test.c compiles, and if it does, so will the toolbox.

...and if you send me your relevant lines of compilers.m, I can add this to the repository, and warn other people that it will not work with this compiler. so they don't have to go through the same process you had.

MfMartel commented 5 years ago

Indeed, adding #define PLATFORM_X86 doesn't change anything...

I installed Visual Studio 2015 with C++ but it doesn't appear in my Matlab compiler list. I think this is because when you check online the list of compiler compatible with Matlab 2015a https://www.mathworks.com/content/dam/mathworks/mathworks-dot-com/support/sysreq/files/SystemRequirements-Release2015a_SupportedCompilers.pdf, you only have a few... And neither minGW nor Visual Studio are in the list. There is Microsoft Visual C++ but it has to be the pro version, which we currently don't have...

Does this mean that as long as we don't update the Matlab version or get the pro compiler, we unfortunately won't be able to use the toolbox?

And here you go for the code in compilers.m

%% Microsoft Windows SDK 7.1 (C++)
% You need the version .NET framework 4.0 and not the later ones to install this compiler
if(strcmp(compiler_info.Name, 'Microsoft Windows SDK 7.1 (C++)') && ~compiler_found)
    %Maybe only certain versions of your compiler is usable. Perhaps you will need to do a version check
    %Note that the version number is also stored as a string.
    if(strcmp(compiler_info.Version, '7.1'))
        %Note that we always APPEND to the compiler flags. The default flags are set in the $COMPFLAGS environment
        %variable and is used by Matlab's mex command.
        compiler_flags = '-largeArrayDims'; %You may not need to add any extra compiler flags, but if you do, add them here.
        compiler_found = 1;
    end
end
ha5dzs commented 5 years ago

Apologies, my bad. We use Matlab R2015b for the 32-bit windows 7 computer, and I thought R2015a is also compatible with Visual Studio 2015. I now see that it's only compatible with Visual Studio 2013. According to this, https://docs.microsoft.com/en-us/visualstudio/releasenotes/vs2013-community-vs the free version of VS can include the C++ compiler

I haven't tested it, but I am fairly confident that the 2013 will work with the toolbox. You will need to add it to compilers.m though.

And as for the code, thank you very much, will do a git push soon, and RUNME.m will throw an error if this compiler is detected. Hopefully it will save someone a lot of frustration in the future.

MfMartel commented 5 years ago

Thank you, but I downloaded the compiler and I still have the exact same issue as before...

From compiler.m:

if(strcmp(compiler_info.Name, 'Microsoft Visual C++ 2013 Professional') && ~compiler_found)
    %Maybe only certain versions of your compiler is usable. Perhaps you will need to do a version check
    %Note that the version number is also stored as a string.
    if(strcmp(compiler_info.Version, '12.0'))
        %Note that we always APPEND to the compiler flags. The default flags are set in the $COMPFLAGS environment
        %variable and is used by Matlab's mex command.
        compiler_flags = ''; %You may not need to add any extra compiler flags, but if you do, add them here.
         compiler_found = 1;
    end
end

And the warning as before in RUNME.m:

Warning: Message from C preprocessor:
lcc preprocessor error: D:\OPTO 2\Documents\MATLAB\motom-toolbox-master\generated_binaries\..\source\ndhost.h:159 D:\OPTO
2\Documents\MATLAB\motom-toolbox-master\generated_binaries\..\source\ndtypes.h:7 D:\OPTO
2\Documents\MATLAB\motom-toolbox-master\generated_binaries\..\source\ndopto.h:23 #error directive: NDI Error - Host not
defined. Options: PLATFORM_X86, PLATFORM_LINUX
lcc preprocessor error: D:\OPTO 2\Documents\MATLAB\motom-toolbox-master\generated_binaries\..\source\ndpack.h:25 D:\OPTO
2\Documents\MATLAB\motom-toolbox-master\generated_binaries\..\source\ndopto.h:24 #error directive: You must define the type
of system you are using...

And test.c doesn't run

I also tried with the Express version of VS2013 but it is not recognised in the compiler list.

ha5dzs commented 5 years ago

Darn. Okay, I am going to replicate your environment here on a computer I can scrounge, and will install your (R2015a, non-SP1) version of Matlab and a bunch of compilers. I am trying to get the test.c file to compile at first.

From what you said here, I am still not sure if something is wrong with your Matlab environment, or it's the C-code that will never run on it due to mex API changes around then.

Meanwhile, would it be possible for you to find an other computer as well, with a Matlab and compilers at the same era, and try to compile test.c?

Will get back to you when I got a computer and installed the stuff correctly.

MfMartel commented 5 years ago

Sorry for the trouble :)

test.c compiled just right with VS 2013 but a 2015a SP1 version of Matlab. I'll let you know as soon as I test it on another 2015a (non-SP1).

ha5dzs commented 5 years ago

I ended up having to use a virtual machine. Spare windows 7 computers are very difficult to find here. I installed R2015a and Visual Studio 2013 Community. Matlab detects the compiler, and test.c compiles fine.

It turned out that both the mex interface and the compiler changed from R2015a to R2015b, and from 2013 to 2015, respectively.

here is the relevant bit of compilers.m

%% Microsoft Visual C++ 2013
% Visual Studio 2013.
if(strcmp(compiler_info.Name, 'Microsoft Visual C++ 2013 Professional') && ~compiler_found)
    %if we got here, we have found our compiler.
    compiler_found = 1;
    %For each compiler and each version, different compiler flags are needed.
    compiler_flags = '/O2 /Wall'; % I was too lazy to change these. Still works fine.
end

...and you need to change line 129 in RUNME.m to the following:

compiler_string = sprintf('mex -v COMPFLAGS="$COMPFLAGS %s" %s -L..\\bin -loapi %s', compiler_flags, ['-I"', toolbox_path, '\source"'], file_string);

Now the toolbox compiles fine for me.

If you check compiler_string manually, it should be something like:

mex -v COMPFLAGS="$COMPFLAGS /O2 /Wall" -I"C:\Users\nyuad\Documents\MATLAB\motom-toolbox-master\source" -L..\bin -loapi ../source/DataReceiveLatestTransforms2_as_array.c
MfMartel commented 5 years ago

Hi,

Thanks! I was writing to you in the meantime. I got another computer, win64-bit, with a 2015a 32-bit version of Matlab. Nothing from the Optotrak is installed on this computer, but I copied pasted the files in the bin and source as we are supposed to do.

mex -setup c++ gives Microsoft Visual C++ 2013 Professional mex -v test.c compiles well. And test gets the "it worked" output.

The only difference I see is that the lab computer is not running on an Admin session (but all the MATLAB path I thought of have been opened for all the sessions). I'll try to run that on the Admin session and see if there is any difference.

I made the changes in the script, thank you for taking the time to test and fix it!

Right now I have an error saying

Error using loadlibrary (line 447)
Failed to preprocess the input file.
 Output from preprocessor is:ndopto.h
c:\users\mmartel\documents\post-doc_2019_ddl\motom-toolbox-master\source\ndhost.h(159)
: fatal error C1189: #error :  NDI Error - Host not defined. Options:
PLATFORM_X86, PLATFORM_LINUX

Error in RUNME (line 88)
        [~, warnings] = loadlibrary('../bin/oapi.dll', '../source/ndopto.h',
        'addheader', '../source/ndtypes.h', 'addheader', '../source/ndhost.h',
        'addheader', '../source/ndpack.h', 'mfilename','api_prototypes');

But maybe it comes from the fact that I'm not on the computer where all the stuffs from the Opotrak is installed. I'm gonna test it now and let you know. thx!

ha5dzs commented 5 years ago

You will have to be admin, or at least have fill access to c:\ndigital and c:\ndiopai. I tried it running as a user, and it randomly crashed during runtime. As for the error message, it seems that the preprocessor fails to detect the platform you are running it on. I don't get this with my, more recent copy of the API. With a bit of luck, you can simply add the line to ndhost.h

#define PLATFORM_X86

and ideally it should proceed. If not, I can give you additional defines, and add a sort of a backported support for 64-bit OS and 32-bit compiler set-up.

MfMartel commented 5 years ago

#define PLATFORM_X86 did the trick. It works now!

I still have troubles on the other computer, but we have a bunch of errors which are not due to the toolbox, but to some errors with NDI. It is probably the source of the new issues. So it is outside this thread ;)

Thank you very much your help!!

ha5dzs commented 5 years ago

Cool! Well, hopefully you are over the worst, but I don't know what other little surprises the old API holds. Hopefully you'l alsol hear those beautiful beeps and pausing fan noises everybody loves about this system.

I will incorporate these changes: -Detect lcc, and make compilers.m fail -Add an additional error for the SDK compiler in compilers.m -Add VS2013 to compilers.m -Throw a warning about old APIs when oapi64.* is missing but we are compiling on a 32-bit matlab in RUNME.m -Add the old-school compiler_string to the 32-bit section, for pre-2015b versions in RUNME.m

I will put these up with some extra docs. I think we can close this issue for now. Guess you can use the toolbox as it is, luckily no major overhaul is needed. :)