sccn / amica

Code for AMICA: Adaptive Mixture ICA with shared components
BSD 2-Clause "Simplified" License
23 stars 13 forks source link

AMICA not working on NSG server #24

Open moo-n opened 3 years ago

moo-n commented 3 years ago

Hello, I've been trying to run AMICA on NSG server but it doesn't work. Attached the code I used to run it and the outputs I got. the dataset I'm using is epoched. I tried both running it using one node and using multiple nodes by setting numprocs=4 and both failed. the input file:

%% test_script_amica_nsg.m
% add eeglab to path
eeglab; close;

% load dataset 
filepath = [ pwd filesep ];
filename = '23456.set';
EEG = pop_loadset(filename, filepath);

% define parameters
numprocs = 1;       % # of nodes (1-4: default 1)
max_threads = 24;   % # of threads (1-24: default = 24)
num_models = 1;     % # of models of mixture ICA 
max_iter = 3000;    % max number of learning steps

% run amica on NSG
outdir = [ pwd filesep 'amicaouttmp' filesep ];
runamica15_nsg(EEG, 'num_models',num_models, 'outdir',outdir, 'max_iter',max_iter);

output files STDOUT

                            < M A T L A B (R) >
                  Copyright 1984-2019 The MathWorks, Inc.
              R2019b Update 3 (9.7.0.1261785) 64-bit (glnxa64)
                             November 27, 2019

To get started, type doc.
For product information, visit www.mathworks.com.

[Warning: Adding folders named 'resources' to the path is not supported:
/projects/ps-nsg/home/nsguser/applications_centos7/eeglab_new/functions/resources] 
[> In path (line 109)
  In addpath (line 86)
  In eeglab>addpathifnotinlist (line 2053)
  In eeglab>myaddpath (line 2082)
  In eeglab (line 287)
  In runAmica (line 3)] 
eeglab: options file is ~/eeg_options.m
EEGLAB: adding "ANTeepimport" v1.13 (see >> help eegplugin_eepimport)
EEGLAB: adding "BCI2000import" v0.36 (see >> help eegplugin_BCI2000import)
EEGLAB: adding "BDFimport" v1.10 (see >> help eegplugin_bdfimport)
EEGLAB: adding "Biosig" to the path; subfolders (if any) might be missing from the path
EEGLAB: adding "Fieldtrip-lite" to the path; subfolders (if any) might be missing from the path
EEGLAB: adding "Fileio" to the path; subfolders (if any) might be missing from the path
EEGLAB: adding "MEF_import" v1.10 (see >> help eegplugin_mefimport)
EEGLAB: adding "NEUROPRAXimport" v1.00 (see >> help eegplugin_NEUROPRAXimport)
EEGLAB: adding "WearableSensing" v1.09 (see >> help eegplugin_pop_WearableSensing)
EEGLAB: adding "amica" v1.5.1 (see >> help eegplugin_amica)
EEGLAB: adding "bci2000legacy" v1.0 (see >> help eegplugin_bci2000legacy)
EEGLAB: adding "bdfimport" v1.1 (see >> help eegplugin_bdfimport)
EEGLAB: adding "biopac" v1.00 (see >> help eegplugin_biopac)
EEGLAB: adding "bva-io-" v1.5.13 (see >> help eegplugin_bva_io)
EEGLAB: adding "bva-io" v1.5.13 (see >> help eegplugin_bva_io)
EEGLAB: adding "cogniscan" v1.1 (see >> help eegplugin_cogniscan)
EEGLAB: adding "corrmap" v2.02.1 (see >> help eegplugin_corrmap)
EEGLAB: adding "ctfimport" v1.04 (see >> help eegplugin_ctfimport)
EEGLAB: adding "dipfit" v2.3 (see >> help eegplugin_dipfit)
EEGLAB: adding "egilegacy" v1.0 (see >> help eegplugin_egilegacy)
EEGLAB: adding "erpssimport" v1.01 (see >> help eegplugin_erpssimport)
EEGLAB: adding "firfilt" v1.6.2 (see >> help eegplugin_firfilt)
EEGLAB: adding "gtecimport" v3.15.01 (see >> help eegplugin_gtecimport)
EEGLAB: adding "import_NihonKodenM00files" v1.11 (see >> help eegplugin_importNihonKodenM00)
EEGLAB: adding "instepascimport" v1.00 (see >> help eegplugin_ascinstep)
EEGLAB: adding "loadcurry" v2.0 (see >> help eegplugin_loadcurry)
EEGLAB: adding "loadhdf5v" v1.1 (see >> help eegplugin_loadhdf5)
EEGLAB: error while adding plugin "eegplugin_mffimport"
   Error using error
Function is not defined for 'cell' inputs.
EEGLAB: adding "mffmatlabio" v2.01 (see >> help eegplugin_mffmatlabio)
EEGLAB: adding "musemonitor" v3.1 (see >> help eegplugin_musemonitor)
EEGLAB: adding "neuroimaging4d" v1.00 (see >> help eegplugin_4dneuroimaging)
EEGLAB: adding "neuroneimport" v1.0.3.4 (see >> help eegplugin_neurone)
EEGLAB: adding "neuroscanio" v1.1 (see >> help eegplugin_neuroscanio)
EEGLAB: adding "procominfinity" v1.00 (see >> help eegplugin_procom)
EEGLAB: adding "snapmaster" v1.0 (see >> help eegplugin_snapmaster)
EEGLAB: adding "xdf-EEGLAB" v1.14 (see >> help eegplugin_xdfimport)
[Warning: Adding folders named 'resources' to the path is not supported:
/projects/ps-nsg/home/nsguser/applications_centos7/eeglab_new/functions/resources] 
[> In path (line 109)
  In addpath (line 86)
  In eeglab>addpathifnotinlist (line 2053)
  In eeglab>myaddpath (line 2082)
  In eeglab (line 287)
  In eeglab (line 1084)
  In runAmica (line 3)] 
pop_loadset(): loading file /projects/ps-nsg/home/nsguser/ngbw/workspace/NGBW-JOB-EEGLAB_TG-985C5F6F32174E75A1931EDA1AD2E76B/12456/12456.set ...
Running AMICA on single node ...
INTERNAL ERROR: invalid error code ffffffff (Ring Index out of range) in rdma_cm_get_hostnames:1267
Warning: RDMA CM Initialization failed. Continuing without RDMA CM support. Please set MV2_USE_RDMA_CM=0 to disable RDMA CM.
           1 processor name = comet-18-44.sdsc.edu
           1 host_num =   1457401042
 This is MPI process           1 of           1 ; I am process           1 of
           1 on node: comet-18-44.sdsc.edu
           1  : node root process           1 of           1
Processing arguments ...
forrtl: severe (66): output statement overflows record, unit -5, file Internal Formatted Write
Image              PC                Routine            Line        Source             
amica15c           0000000000E88B8B  Unknown               Unknown  Unknown
amica15c           0000000000E86B97  Unknown               Unknown  Unknown
amica15c           000000000044D799  Unknown               Unknown  Unknown
amica15c           000000000040CD6C  Unknown               Unknown  Unknown
amica15c           000000000040A6B6  Unknown               Unknown  Unknown
libc.so.6          00002B5285D823D5  Unknown               Unknown  Unknown
amica15c           000000000040A529  Unknown               Unknown  Unknown
Oops! AMICA did not run succesfully...
`

STDERR `

{Unrecognized function or variable 'filename'.

Error in runamica15_nsg (line 922)
if exist(filename,'file')

Error in runAmica (line 18)
runamica15_nsg(EEG, 'num_models',num_models, 'outdir',outdir,
'max_iter',max_iter);
} 

`

japalmer29 commented 3 years ago

Are you sure you're using pop_loadset correctly? I think you need keywords, like EEG=pop_loadset('filename',filename,'filepath',filepath);

On Sun, Oct 25, 2020, 6:44 PM moo-n notifications@github.com wrote:

Hello, I've been trying to run AMICA on NSG server but it doesn't work. Attached the code I used to run it and the outputs I got. the dataset I'm using is epoched. I tried both running it using one node and using multiple nodes by setting numprocs=4 and both failed. the input file:

%% test_script_amica_nsg.m

% add eeglab to path

eeglab; close;

% load dataset

filepath = [ pwd filesep ];

filename = '23456.set';

EEG = pop_loadset(filename, filepath);

% define parameters

numprocs = 1; % # of nodes (1-4: default 1)

max_threads = 24; % # of threads (1-24: default = 24)

num_models = 1; % # of models of mixture ICA

max_iter = 3000; % max number of learning steps

% run amica on NSG

outdir = [ pwd filesep 'amicaouttmp' filesep ];

runamica15_nsg(EEG, 'num_models',num_models, 'outdir',outdir, 'max_iter',max_iter);

output files STDOUT

                        < M A T L A B (R) >

              Copyright 1984-2019 The MathWorks, Inc.

          R2019b Update 3 (9.7.0.1261785) 64-bit (glnxa64)

                         November 27, 2019

To get started, type doc.

For product information, visit www.mathworks.com.

[�Warning: Adding folders named 'resources' to the path is not supported:

/projects/ps-nsg/home/nsguser/applications_centos7/eeglab_new/functions/resources]�

[�> In path (line 109)

In addpath (line 86)

In eeglab>addpathifnotinlist (line 2053)

In eeglab>myaddpath (line 2082)

In eeglab (line 287)

In runAmica (line 3)]�

eeglab: options file is ~/eeg_options.m

EEGLAB: adding "ANTeepimport" v1.13 (see >> help eegplugin_eepimport)

EEGLAB: adding "BCI2000import" v0.36 (see >> help eegplugin_BCI2000import)

EEGLAB: adding "BDFimport" v1.10 (see >> help eegplugin_bdfimport)

EEGLAB: adding "Biosig" to the path; subfolders (if any) might be missing from the path

EEGLAB: adding "Fieldtrip-lite" to the path; subfolders (if any) might be missing from the path

EEGLAB: adding "Fileio" to the path; subfolders (if any) might be missing from the path

EEGLAB: adding "MEF_import" v1.10 (see >> help eegplugin_mefimport)

EEGLAB: adding "NEUROPRAXimport" v1.00 (see >> help eegplugin_NEUROPRAXimport)

EEGLAB: adding "WearableSensing" v1.09 (see >> help eegplugin_pop_WearableSensing)

EEGLAB: adding "amica" v1.5.1 (see >> help eegplugin_amica)

EEGLAB: adding "bci2000legacy" v1.0 (see >> help eegplugin_bci2000legacy)

EEGLAB: adding "bdfimport" v1.1 (see >> help eegplugin_bdfimport)

EEGLAB: adding "biopac" v1.00 (see >> help eegplugin_biopac)

EEGLAB: adding "bva-io-" v1.5.13 (see >> help eegplugin_bva_io)

EEGLAB: adding "bva-io" v1.5.13 (see >> help eegplugin_bva_io)

EEGLAB: adding "cogniscan" v1.1 (see >> help eegplugin_cogniscan)

EEGLAB: adding "corrmap" v2.02.1 (see >> help eegplugin_corrmap)

EEGLAB: adding "ctfimport" v1.04 (see >> help eegplugin_ctfimport)

EEGLAB: adding "dipfit" v2.3 (see >> help eegplugin_dipfit)

EEGLAB: adding "egilegacy" v1.0 (see >> help eegplugin_egilegacy)

EEGLAB: adding "erpssimport" v1.01 (see >> help eegplugin_erpssimport)

EEGLAB: adding "firfilt" v1.6.2 (see >> help eegplugin_firfilt)

EEGLAB: adding "gtecimport" v3.15.01 (see >> help eegplugin_gtecimport)

EEGLAB: adding "import_NihonKodenM00files" v1.11 (see >> help eegplugin_importNihonKodenM00)

EEGLAB: adding "instepascimport" v1.00 (see >> help eegplugin_ascinstep)

EEGLAB: adding "loadcurry" v2.0 (see >> help eegplugin_loadcurry)

EEGLAB: adding "loadhdf5v" v1.1 (see >> help eegplugin_loadhdf5)

EEGLAB: error while adding plugin "eegplugin_mffimport"

Error using error

Function is not defined for 'cell' inputs.

EEGLAB: adding "mffmatlabio" v2.01 (see >> help eegplugin_mffmatlabio)

EEGLAB: adding "musemonitor" v3.1 (see >> help eegplugin_musemonitor)

EEGLAB: adding "neuroimaging4d" v1.00 (see >> help eegplugin_4dneuroimaging)

EEGLAB: adding "neuroneimport" v1.0.3.4 (see >> help eegplugin_neurone)

EEGLAB: adding "neuroscanio" v1.1 (see >> help eegplugin_neuroscanio)

EEGLAB: adding "procominfinity" v1.00 (see >> help eegplugin_procom)

EEGLAB: adding "snapmaster" v1.0 (see >> help eegplugin_snapmaster)

EEGLAB: adding "xdf-EEGLAB" v1.14 (see >> help eegplugin_xdfimport)

[�Warning: Adding folders named 'resources' to the path is not supported:

/projects/ps-nsg/home/nsguser/applications_centos7/eeglab_new/functions/resources]�

[�> In path (line 109)

In addpath (line 86)

In eeglab>addpathifnotinlist (line 2053)

In eeglab>myaddpath (line 2082)

In eeglab (line 287)

In eeglab (line 1084)

In runAmica (line 3)]�

pop_loadset(): loading file /projects/ps-nsg/home/nsguser/ngbw/workspace/NGBW-JOB-EEGLAB_TG-985C5F6F32174E75A1931EDA1AD2E76B/12456/12456.set ...

Running AMICA on single node ...

INTERNAL ERROR: invalid error code ffffffff (Ring Index out of range) in rdma_cm_get_hostnames:1267

Warning: RDMA CM Initialization failed. Continuing without RDMA CM support. Please set MV2_USE_RDMA_CM=0 to disable RDMA CM.

       1 processor name = comet-18-44.sdsc.edu

       1 host_num =   1457401042

This is MPI process 1 of 1 ; I am process 1 of

       1 on node: comet-18-44.sdsc.edu

       1  : node root process           1 of           1

Processing arguments ...

forrtl: severe (66): output statement overflows record, unit -5, file Internal Formatted Write

Image PC Routine Line Source

amica15c 0000000000E88B8B Unknown Unknown Unknown

amica15c 0000000000E86B97 Unknown Unknown Unknown

amica15c 000000000044D799 Unknown Unknown Unknown

amica15c 000000000040CD6C Unknown Unknown Unknown

amica15c 000000000040A6B6 Unknown Unknown Unknown

libc.so.6 00002B5285D823D5 Unknown Unknown Unknown

amica15c 000000000040A529 Unknown Unknown Unknown

Oops! AMICA did not run succesfully...

`

STDERR `

{�Unrecognized function or variable 'filename'.

Error in runamica15_nsg (line 922)

if exist(filename,'file')

Error in runAmica (line 18)

runamica15_nsg(EEG, 'num_models',num_models, 'outdir',outdir,

'max_iter',max_iter);

}�

`

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/japalmer29/amica/issues/24, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRBESUL2VJROMY66LYQUZLSMTIAJANCNFSM4S6YAJHQ .

moo-n commented 3 years ago

Are you sure you're using pop_loadset correctly? I think you need keywords, like EEG=pop_loadset('filename',filename,'filepath',filepath); .

I just tried the command and it loads the set just fine.

japalmer29 commented 3 years ago

Did you save the dataset as separate .set and .fdt files? If it is single file it won't work. This can be changed in options in Eeglab gui.

On Sun, Oct 25, 2020, 10:31 PM moo-n notifications@github.com wrote:

Are you sure you're using pop_loadset correctly? I think you need keywords, like EEG=poploadset('filename',filename,'filepath',filepath); … <#m-2793330629098404438_> On Sun, Oct 25, 2020, 6:44 PM moo-n @.***> wrote: Hello, I've been trying to run AMICA on NSG server but it doesn't work. Attached the code I used to run it and the outputs I got. the dataset I'm using is epoched. I tried both running it using one node and using multiple nodes by setting numprocs=4 and both failed. the input file: %% test_script_amica_nsg.m % add eeglab to path eeglab; close; % load dataset filepath = [ pwd filesep ]; filename = '23456.set'; EEG = pop_loadset(filename, filepath); % define parameters numprocs = 1; % # of nodes (1-4: default 1) max_threads = 24; %

of threads (1-24: default = 24) num_models = 1; % # of models of mixture

ICA max_iter = 3000; % max number of learning steps % run amica on NSG outdir = [ pwd filesep 'amicaouttmp' filesep ]; runamica15_nsg(EEG, 'num_models',num_models, 'outdir',outdir, 'max_iter',max_iter); output files STDOUT < M A T L A B (R) > Copyright 1984-2019 The MathWorks, Inc. R2019b Update 3 (9.7.0.1261785) 64-bit (glnxa64) November 27, 2019 To get started, type doc. For product information, visit www.mathworks.com. [�Warning: Adding folders named 'resources' to the path is not supported: /projects/ps-nsg/home/nsguser/applications_centos7/eeglab_new/functions/resources]� [�> In path (line 109) In addpath (line 86) In eeglab>addpathifnotinlist (line 2053) In eeglab>myaddpath (line 2082) In eeglab (line 287) In runAmica (line 3)]� eeglab: options file is ~/eeg_options.m EEGLAB: adding "ANTeepimport" v1.13 (see >> help eegplugin_eepimport) EEGLAB: adding "BCI2000import" v0.36 (see >> help eegplugin_BCI2000import) EEGLAB: adding "BDFimport" v1.10 (see >> help eegplugin_bdfimport) EEGLAB: adding "Biosig" to the path; subfolders (if any) might be missing from the path EEGLAB: adding "Fieldtrip-lite" to the path; subfolders (if any) might be missing from the path EEGLAB: adding "Fileio" to the path; subfolders (if any) might be missing from the path EEGLAB: adding "MEF_import" v1.10 (see >> help eegplugin_mefimport) EEGLAB: adding "NEUROPRAXimport" v1.00 (see >> help eegplugin_NEUROPRAXimport) EEGLAB: adding "WearableSensing" v1.09 (see

help eegplugin_pop_WearableSensing) EEGLAB: adding "amica" v1.5.1 (see help eegplugin_amica) EEGLAB: adding "bci2000legacy" v1.0 (see >> help eegplugin_bci2000legacy) EEGLAB: adding "bdfimport" v1.1 (see >> help eegplugin_bdfimport) EEGLAB: adding "biopac" v1.00 (see >> help eegplugin_biopac) EEGLAB: adding "bva-io-" v1.5.13 (see >> help eegplugin_bva_io) EEGLAB: adding "bva-io" v1.5.13 (see >> help eegplugin_bva_io) EEGLAB: adding "cogniscan" v1.1 (see >> help eegplugin_cogniscan) EEGLAB: adding "corrmap" v2.02.1 (see >> help eegplugin_corrmap) EEGLAB: adding "ctfimport" v1.04 (see >> help eegplugin_ctfimport) EEGLAB: adding "dipfit" v2.3 (see >> help eegplugin_dipfit) EEGLAB: adding "egilegacy" v1.0 (see >> help eegplugin_egilegacy) EEGLAB: adding "erpssimport" v1.01 (see >> help eegplugin_erpssimport) EEGLAB: adding "firfilt" v1.6.2 (see >> help eegplugin_firfilt) EEGLAB: adding "gtecimport" v3.15.01 (see >> help eegplugin_gtecimport) EEGLAB: adding "import_NihonKodenM00files" v1.11 (see help eegplugin_importNihonKodenM00) EEGLAB: adding "instepascimport" v1.00 (see >> help eegplugin_ascinstep) EEGLAB: adding "loadcurry" v2.0 (see >> help eegplugin_loadcurry) EEGLAB: adding "loadhdf5v" v1.1 (see >> help eegplugin_loadhdf5) EEGLAB: error while adding plugin "eegplugin_mffimport" Error using error Function is not defined for 'cell' inputs. EEGLAB: adding "mffmatlabio" v2.01 (see >> help eegplugin_mffmatlabio) EEGLAB: adding "musemonitor" v3.1 (see >> help eegplugin_musemonitor) EEGLAB: adding "neuroimaging4d" v1.00 (see >> help eegplugin_4dneuroimaging) EEGLAB: adding "neuroneimport" v1.0.3.4 (see >> help eegplugin_neurone) EEGLAB: adding "neuroscanio" v1.1 (see >> help eegplugin_neuroscanio) EEGLAB: adding "procominfinity" v1.00 (see >> help eegplugin_procom) EEGLAB: adding "snapmaster" v1.0 (see >> help eegplugin_snapmaster) EEGLAB: adding "xdf-EEGLAB" v1.14 (see >> help eegplugin_xdfimport) [�Warning: Adding folders named 'resources' to the path is not supported: /projects/ps-nsg/home/nsguser/applications_centos7/eeglab_new/functions/resources]� [�> In path (line 109) In addpath (line 86) In eeglab>addpathifnotinlist (line 2053) In eeglab>myaddpath (line 2082) In eeglab (line 287) In eeglab (line 1084) In runAmica (line 3)]� pop_loadset(): loading file /projects/ps-nsg/home/nsguser/ngbw/workspace/NGBW-JOB-EEGLAB_TG-985C5F6F32174E75A1931EDA1AD2E76B/12456/12456.set ... Running AMICA on single node ... INTERNAL ERROR: invalid error code ffffffff (Ring Index out of range) in rdma_cm_get_hostnames:1267 Warning: RDMA CM Initialization failed. Continuing without RDMA CM support. Please set MV2_USE_RDMA_CM=0 to disable RDMA CM. 1 processor name = comet-18-44.sdsc.edu 1 host_num = 1457401042 This is MPI process 1 of 1 ; I am process 1 of 1 on node: comet-18-44.sdsc.edu 1 : node root process 1 of 1 Processing arguments ... forrtl: severe (66): output statement overflows record, unit -5, file Internal Formatted Write Image PC Routine Line Source amica15c 0000000000E88B8B Unknown Unknown Unknown amica15c 0000000000E86B97 Unknown Unknown Unknown amica15c 000000000044D799 Unknown Unknown Unknown amica15c 000000000040CD6C Unknown Unknown Unknown amica15c 000000000040A6B6 Unknown Unknown Unknown libc.so.6 00002B5285D823D5 Unknown Unknown Unknown amica15c 000000000040A529 Unknown Unknown Unknown Oops! AMICA did not run succesfully... STDERR {�Unrecognized function or variable 'filename'. Error in runamica15_nsg (line 922) if exist(filename,'file') Error in runAmica (line 18) runamica15_nsg(EEG, 'num_models',num_models, 'outdir',outdir, 'max_iter',max_iter); }� ` — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#24 https://github.com/japalmer29/amica/issues/24>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRBESUL2VJROMY66LYQUZLSMTIAJANCNFSM4S6YAJHQ .

I just tried the command and it loads the set just fine.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/japalmer29/amica/issues/24#issuecomment-716313415, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRBESRMHVT7EWKN22BX7LLSMUCTPANCNFSM4S6YAJHQ .

moo-n commented 3 years ago

This time it was one file .set. I will use two files and let you know.

moo-n commented 3 years ago

An update: this is the code I used:

%% test_script_amica_nsg.m
% add eeglab to path
eeglab; close;

% load dataset 
filepath = [ pwd filesep ];
filename = '23456.set';
EEG = pop_loadset('filename',filename,'filepath',filepath);

% define parameters
numprocs = 4;       % # of nodes (1-4: default 1)
max_threads = 24;   % # of threads (1-24: default = 24)
num_models = 1;     % # of models of mixture ICA 
max_iter = 3000;    % max number of learning steps

% run amica on NSG
outdir = [ pwd filesep 'amicaouttmp' filesep ];
runamica15_nsg(EEG, 'numprocs', numprocs,'num_models',num_models, 'outdir',outdir, 'max_iter',max_iter);

the output from the cluster machine: stdout.txt


                            < M A T L A B (R) >
                  Copyright 1984-2019 The MathWorks, Inc.
              R2019b Update 3 (9.7.0.1261785) 64-bit (glnxa64)
                             November 27, 2019

To get started, type doc.
For product information, visit www.mathworks.com.

[Warning: Adding folders named 'resources' to the path is not supported:
/projects/ps-nsg/home/nsguser/applications_centos7/eeglab_new/functions/resources] 
[> In path (line 109)
  In addpath (line 86)
  In eeglab>addpathifnotinlist (line 2053)
  In eeglab>myaddpath (line 2082)
  In eeglab (line 287)
  In runAmicanodes (line 3)] 
eeglab: options file is ~/eeg_options.m
EEGLAB: adding "ANTeepimport" v1.13 (see >> help eegplugin_eepimport)
EEGLAB: adding "BCI2000import" v0.36 (see >> help eegplugin_BCI2000import)
EEGLAB: adding "BDFimport" v1.10 (see >> help eegplugin_bdfimport)
EEGLAB: adding "Biosig" to the path; subfolders (if any) might be missing from the path
EEGLAB: adding "Fieldtrip-lite" to the path; subfolders (if any) might be missing from the path
EEGLAB: adding "Fileio" to the path; subfolders (if any) might be missing from the path
EEGLAB: adding "MEF_import" v1.10 (see >> help eegplugin_mefimport)
EEGLAB: adding "NEUROPRAXimport" v1.00 (see >> help eegplugin_NEUROPRAXimport)
EEGLAB: adding "WearableSensing" v1.09 (see >> help eegplugin_pop_WearableSensing)
EEGLAB: adding "amica" v1.5.1 (see >> help eegplugin_amica)
EEGLAB: adding "bci2000legacy" v1.0 (see >> help eegplugin_bci2000legacy)
EEGLAB: adding "bdfimport" v1.1 (see >> help eegplugin_bdfimport)
EEGLAB: adding "biopac" v1.00 (see >> help eegplugin_biopac)
EEGLAB: adding "bva-io-" v1.5.13 (see >> help eegplugin_bva_io)
EEGLAB: adding "bva-io" v1.5.13 (see >> help eegplugin_bva_io)
EEGLAB: adding "cogniscan" v1.1 (see >> help eegplugin_cogniscan)
EEGLAB: adding "corrmap" v2.02.1 (see >> help eegplugin_corrmap)
EEGLAB: adding "ctfimport" v1.04 (see >> help eegplugin_ctfimport)
EEGLAB: adding "dipfit" v2.3 (see >> help eegplugin_dipfit)
EEGLAB: adding "egilegacy" v1.0 (see >> help eegplugin_egilegacy)
EEGLAB: adding "erpssimport" v1.01 (see >> help eegplugin_erpssimport)
EEGLAB: adding "firfilt" v1.6.2 (see >> help eegplugin_firfilt)
EEGLAB: adding "gtecimport" v3.15.01 (see >> help eegplugin_gtecimport)
EEGLAB: adding "import_NihonKodenM00files" v1.11 (see >> help eegplugin_importNihonKodenM00)
EEGLAB: adding "instepascimport" v1.00 (see >> help eegplugin_ascinstep)
EEGLAB: adding "loadcurry" v2.0 (see >> help eegplugin_loadcurry)
EEGLAB: adding "loadhdf5v" v1.1 (see >> help eegplugin_loadhdf5)
EEGLAB: error while adding plugin "eegplugin_mffimport"
   Error using error
Function is not defined for 'cell' inputs.
EEGLAB: adding "mffmatlabio" v2.01 (see >> help eegplugin_mffmatlabio)
EEGLAB: adding "musemonitor" v3.1 (see >> help eegplugin_musemonitor)
EEGLAB: adding "neuroimaging4d" v1.00 (see >> help eegplugin_4dneuroimaging)
EEGLAB: adding "neuroneimport" v1.0.3.4 (see >> help eegplugin_neurone)
EEGLAB: adding "neuroscanio" v1.1 (see >> help eegplugin_neuroscanio)
EEGLAB: adding "procominfinity" v1.00 (see >> help eegplugin_procom)
EEGLAB: adding "snapmaster" v1.0 (see >> help eegplugin_snapmaster)
EEGLAB: adding "xdf-EEGLAB" v1.14 (see >> help eegplugin_xdfimport)
[Warning: Adding folders named 'resources' to the path is not supported:
/projects/ps-nsg/home/nsguser/applications_centos7/eeglab_new/functions/resources] 
[> In path (line 109)
  In addpath (line 86)
  In eeglab>addpathifnotinlist (line 2053)
  In eeglab>myaddpath (line 2082)
  In eeglab (line 287)
  In eeglab (line 1084)
  In runAmicanodes (line 3)] 
pop_loadset(): loading file /projects/ps-nsg/home/nsguser/ngbw/workspace/NGBW-JOB-EEGLAB_TG-DCA1920B6CF94C7F9E354186B73EBEDF/23456/23456.set ...
Reading float file '/projects/ps-nsg/home/nsguser/ngbw/workspace/NGBW-JOB-EEGLAB_TG-DCA1920B6CF94C7F9E354186B73EBEDF/23456/23456.fdt'...
Running AMICA across multiple nodes ...
IBRUN WARNING: Your job geometry appears to suggest that you want 1 threads per MPI rank, but your OMP_NUM_THREADS is 24.  This will cause thread binding to be disabled.
           1 processor name = comet-21-47.sdsc.edu
           1 host_num =    498679133
           2 processor name = comet-21-48.sdsc.edu
           2 host_num =    695192638
 This is MPI process           1 of           4 ; I am process           1 of
           1 on node: comet-21-47.sdsc.edu
           1  : node root process           1 of           4
           4 processor name = comet-21-52.sdsc.edu
           4 host_num =   1313062967
 This is MPI process           4 of           4 ; I am process           1 of
           1 on node: comet-21-52.sdsc.edu
Processing arguments ...
 This is MPI process           2 of           4 ; I am process           1 of
           1 on node: comet-21-48.sdsc.edu
           2  : node root process           2 of           4
           4  : node root process           4 of           4
           3 processor name = comet-21-50.sdsc.edu
           3 host_num =    920035957
 This is MPI process           3 of           4 ; I am process           1 of
           1 on node: comet-21-50.sdsc.edu
           3  : node root process           3 of           4
 num_files =            1
 FILES: 
 /projects/ps-nsg/home/nsguser/ngbw/workspace/NGBW-JOB-EEGLAB_TG-DCA1920B6CF94C7
 F9E354186B73EBEDF/23456//23456.fdt
 num_dir_files =            1
 initial matrix block_size =          128
 do_opt_block =            0
 blk_min =          256
 blk_step =          256
 blk_max =         1024
 number of models =            1
 max_thrds =           24
 use_min_dll =            1
 min dll =   1.000000000000000E-009
 use_grad_norm =            1
 min grad norm =   1.000000000000000E-007
 number of density mixture components =            3
 pdf type =            0
 max_iter =         3000
 num_samples =            1
 data_dim =          127
 field_dim =      4915200
 do_history =            0
 histstep =           10
 share_comps =            0
 share_start =          100
 comp_thresh =   0.990000000000000     
 share_int =          100
 initial lrate =   5.000000000000000E-002
 minimum lrate =   1.000000000000000E-008
 minimum data covariance eigenvalue =   1.000000000000000E-012
 lrate factor =   0.500000000000000     
 initial rholrate =   5.000000000000000E-002
 rho0 =    1.50000000000000     
 min rho =    1.00000000000000     
 max rho =    2.00000000000000     
 rho lrate factor =   0.500000000000000     
 kurt_start =            3
 num kurt =            5
 kurt interval =            1
 do_newton =            1
 newt_start =           50
 newt_ramp =           10
 initial newton lrate =    1.00000000000000     
 do_reject =            0
 num reject =            3
 reject sigma =    3.00000000000000     
 reject start =            2
 reject interval =            3
 write step =           20
 write_nd =            0
 write_LLt =            1
 dec window =            1
 max_decs =            3
 fix_init =            0
 update_A =            1
 update_c =            1
 update_gm =            1
 update_alpha =            1
 update_mu =            1
 update_beta =            1
 invsigmax =    100.000000000000     
 invsigmin =   0.000000000000000E+000
 do_rho =            1
 load_rej =            0
 load_c =            0
 load_gm =            0
 load_alpha =            0
 load_mu =            0
 load_beta =            0
 load_rho =            0
 load_comp_list =            0
 do_mean =            1
 do_sphere =            1
 pcakeep =          127
 pcadb =    30.0000000000000     
 byte_size =            4
 doscaling =            1
 scalestep =            1
mkdir: cannot create directory ‘/projects/ps-nsg/home/nsguser/ngbw/workspace/NGBW-JOB-EEGLAB_TG-DCA1920B6CF94C7F9E354186B73EBEDF/23456/amicaouttmp/’: File exists
 output directory = 
 /projects/ps-nsg/home/nsguser/ngbw/workspace/NGBW-JOB-EEGLAB_TG-DCA1920B6CF94C7
 F9E354186B73EBEDF/23456/amicaouttmp/
           2 : setting num_thrds to           24  ...
           3 : setting num_thrds to           24  ...
           1 : setting num_thrds to           24  ...
           4 : setting num_thrds to           24  ...
           4 : using          24 threads.
           1 : using          24 threads.
           2 : using          24 threads.
           3 : using          24 threads.
           1 : node_thrds =           24          24          24          24
 bytes in real =            1
           1 : REAL nbyte =            1
 getting segment list ...
 blocks in sample =      4915200
 total blocks =      4915200
 node blocks =      1228800     1228800     1228800     1228800
 node            1  start: file            1  sample            1  index 
           1
 node            1  stop : file            1  sample            1  index 
     1228800
 node            2  start: file            1  sample            1  index 
     1228801
 node            2  stop : file            1  sample            1  index 
     2457600
 node            3  start: file            1  sample            1  index 
     2457601
 node            3  stop : file            1  sample            1  index 
     3686400
 node            4  start: file            1  sample            1  index 
     3686401
 node            4  stop : file            1  sample            1  index 
     4915200
           1 : data =    11.9744024276733        4.24514341354370     
           3 : data =   -7.78323650360107      -0.299957871437073     
           2 : data =    1.66699647903442       -26.0488491058350     
           4 : data =   -2.59526801109314       -6.62773466110229     
 getting the mean ...
  mean =   -1.36149215149254      -0.697963101435924     
 -0.908440884302629     
 subtracting the mean ...
 getting the covariance matrix ...
 cnt =      4915200
 doing eig nx =          127  lwork =       161290
 doing eig nx =          127  lwork =       161290
 doing eig nx =          127  lwork =       161290
 doing eig nx =          127  lwork =       161290
 minimum eigenvalues =   2.439452986291257E-002   2.24693689752528     
   2.29249564029623     
 minimum eigenvalues =   2.439452986291257E-002   2.24693689752528     
   2.29249564029623     
 maximum eigenvalues =    13003.3767440951        1393.52042042707     
   1052.95147913640     
 num eigs kept =            0
 maximum eigenvalues =    13003.3767440951        1393.52042042707     
   1052.95147913640     
 num eigs kept =          127
 getting the sphering matrix ...
 minimum eigenvalues =   2.439452986291257E-002   2.24693689752528     
   2.29249564029623     
 maximum eigenvalues =    13003.3767440951        1393.52042042707     
   1052.95147913640     
 num eigs kept =          127
 sphering the data ...
 minimum eigenvalues =   2.439452986291257E-002   2.24693689752528     
   2.29249564029623     
 maximum eigenvalues =    13003.3767440951        1393.52042042707     
   1052.95147913640     
 num eigs kept =            0
 numeigs =          127
japalmer29 commented 3 years ago

It seems that the sphering is being done on all the nodes, but the minimum eigenvalue is not being broadcast or something. The code may have been changed so that the sphering is done redundantly instead of at root node and broadcast, but mineig is not broadcast first.

I’m not sure where the version of amica15.f90 is that was compiled for NSG.

Arno, or other collaborator, can you tell me the working code directory on Comet?

-Jason

From: moo-n [mailto:notifications@github.com] Sent: Wednesday, October 28, 2020 10:02 AM To: japalmer29/amica amica@noreply.github.com Cc: Jason Palmer japalmer29@gmail.com; Comment comment@noreply.github.com Subject: Re: [japalmer29/amica] AMICA not working on NSG server (#24)

An update: this is the code I used:

%% test_script_amica_nsg.m

% add eeglab to path

eeglab; close;

% load dataset

filepath = [ pwd filesep ];

filename = '23456.set';

EEG = pop_loadset('filename',filename,'filepath',filepath);

% define parameters

numprocs = 4; % # of nodes (1-4: default 1)

max_threads = 24; % # of threads (1-24: default = 24)

num_models = 1; % # of models of mixture ICA

max_iter = 3000; % max number of learning steps

% run amica on NSG

outdir = [ pwd filesep 'amicaouttmp' filesep ];

runamica15_nsg(EEG, 'numprocs', numprocs,'num_models',num_models, 'outdir',outdir, 'max_iter',max_iter);

the output from the cluster machine: stdout.txt

                        < M A T L A B (R) >

              Copyright 1984-2019 The MathWorks, Inc.

          R2019b Update 3 (9.7.0.1261785) 64-bit (glnxa64)

                         November 27, 2019

To get started, type doc.

For product information, visit www.mathworks.com http://www.mathworks.com .

[�Warning: Adding folders named 'resources' to the path is not supported:

/projects/ps-nsg/home/nsguser/applications_centos7/eeglab_new/functions/resources]�

[�> In path (line 109)

In addpath (line 86)

In eeglab>addpathifnotinlist (line 2053)

In eeglab>myaddpath (line 2082)

In eeglab (line 287)

In runAmicanodes (line 3)]�

eeglab: options file is ~/eeg_options.m

EEGLAB: adding "ANTeepimport" v1.13 (see >> help eegplugin_eepimport)

EEGLAB: adding "BCI2000import" v0.36 (see >> help eegplugin_BCI2000import)

EEGLAB: adding "BDFimport" v1.10 (see >> help eegplugin_bdfimport)

EEGLAB: adding "Biosig" to the path; subfolders (if any) might be missing from the path

EEGLAB: adding "Fieldtrip-lite" to the path; subfolders (if any) might be missing from the path

EEGLAB: adding "Fileio" to the path; subfolders (if any) might be missing from the path

EEGLAB: adding "MEF_import" v1.10 (see >> help eegplugin_mefimport)

EEGLAB: adding "NEUROPRAXimport" v1.00 (see >> help eegplugin_NEUROPRAXimport)

EEGLAB: adding "WearableSensing" v1.09 (see >> help eegplugin_pop_WearableSensing)

EEGLAB: adding "amica" v1.5.1 (see >> help eegplugin_amica)

EEGLAB: adding "bci2000legacy" v1.0 (see >> help eegplugin_bci2000legacy)

EEGLAB: adding "bdfimport" v1.1 (see >> help eegplugin_bdfimport)

EEGLAB: adding "biopac" v1.00 (see >> help eegplugin_biopac)

EEGLAB: adding "bva-io-" v1.5.13 (see >> help eegplugin_bva_io)

EEGLAB: adding "bva-io" v1.5.13 (see >> help eegplugin_bva_io)

EEGLAB: adding "cogniscan" v1.1 (see >> help eegplugin_cogniscan)

EEGLAB: adding "corrmap" v2.02.1 (see >> help eegplugin_corrmap)

EEGLAB: adding "ctfimport" v1.04 (see >> help eegplugin_ctfimport)

EEGLAB: adding "dipfit" v2.3 (see >> help eegplugin_dipfit)

EEGLAB: adding "egilegacy" v1.0 (see >> help eegplugin_egilegacy)

EEGLAB: adding "erpssimport" v1.01 (see >> help eegplugin_erpssimport)

EEGLAB: adding "firfilt" v1.6.2 (see >> help eegplugin_firfilt)

EEGLAB: adding "gtecimport" v3.15.01 (see >> help eegplugin_gtecimport)

EEGLAB: adding "import_NihonKodenM00files" v1.11 (see >> help eegplugin_importNihonKodenM00)

EEGLAB: adding "instepascimport" v1.00 (see >> help eegplugin_ascinstep)

EEGLAB: adding "loadcurry" v2.0 (see >> help eegplugin_loadcurry)

EEGLAB: adding "loadhdf5v" v1.1 (see >> help eegplugin_loadhdf5)

EEGLAB: error while adding plugin "eegplugin_mffimport"

Error using error

Function is not defined for 'cell' inputs.

EEGLAB: adding "mffmatlabio" v2.01 (see >> help eegplugin_mffmatlabio)

EEGLAB: adding "musemonitor" v3.1 (see >> help eegplugin_musemonitor)

EEGLAB: adding "neuroimaging4d" v1.00 (see >> help eegplugin_4dneuroimaging)

EEGLAB: adding "neuroneimport" v1.0.3.4 (see >> help eegplugin_neurone)

EEGLAB: adding "neuroscanio" v1.1 (see >> help eegplugin_neuroscanio)

EEGLAB: adding "procominfinity" v1.00 (see >> help eegplugin_procom)

EEGLAB: adding "snapmaster" v1.0 (see >> help eegplugin_snapmaster)

EEGLAB: adding "xdf-EEGLAB" v1.14 (see >> help eegplugin_xdfimport)

[�Warning: Adding folders named 'resources' to the path is not supported:

/projects/ps-nsg/home/nsguser/applications_centos7/eeglab_new/functions/resources]�

[�> In path (line 109)

In addpath (line 86)

In eeglab>addpathifnotinlist (line 2053)

In eeglab>myaddpath (line 2082)

In eeglab (line 287)

In eeglab (line 1084)

In runAmicanodes (line 3)]�

pop_loadset(): loading file /projects/ps-nsg/home/nsguser/ngbw/workspace/NGBW-JOB-EEGLAB_TG-DCA1920B6CF94C7F9E354186B73EBEDF/23456/23456.set ...

Reading float file '/projects/ps-nsg/home/nsguser/ngbw/workspace/NGBW-JOB-EEGLAB_TG-DCA1920B6CF94C7F9E354186B73EBEDF/23456/23456.fdt'...

Running AMICA across multiple nodes ...

IBRUN WARNING: Your job geometry appears to suggest that you want 1 threads per MPI rank, but your OMP_NUM_THREADS is 24. This will cause thread binding to be disabled.

       1 processor name = comet-21-47.sdsc.edu

       1 host_num =    498679133

       2 processor name = comet-21-48.sdsc.edu

       2 host_num =    695192638

This is MPI process 1 of 4 ; I am process 1 of

       1 on node: comet-21-47.sdsc.edu

       1  : node root process           1 of           4

       4 processor name = comet-21-52.sdsc.edu

       4 host_num =   1313062967

This is MPI process 4 of 4 ; I am process 1 of

       1 on node: comet-21-52.sdsc.edu

Processing arguments ...

This is MPI process 2 of 4 ; I am process 1 of

       1 on node: comet-21-48.sdsc.edu

       2  : node root process           2 of           4

       4  : node root process           4 of           4

       3 processor name = comet-21-50.sdsc.edu

       3 host_num =    920035957

This is MPI process 3 of 4 ; I am process 1 of

       1 on node: comet-21-50.sdsc.edu

       3  : node root process           3 of           4

num_files = 1

FILES:

/projects/ps-nsg/home/nsguser/ngbw/workspace/NGBW-JOB-EEGLAB_TG-DCA1920B6CF94C7

F9E354186B73EBEDF/23456//23456.fdt

num_dir_files = 1

initial matrix block_size = 128

do_opt_block = 0

blk_min = 256

blk_step = 256

blk_max = 1024

number of models = 1

max_thrds = 24

use_min_dll = 1

min dll = 1.000000000000000E-009

use_grad_norm = 1

min grad norm = 1.000000000000000E-007

number of density mixture components = 3

pdf type = 0

max_iter = 3000

num_samples = 1

data_dim = 127

field_dim = 4915200

do_history = 0

histstep = 10

share_comps = 0

share_start = 100

comp_thresh = 0.990000000000000

share_int = 100

initial lrate = 5.000000000000000E-002

minimum lrate = 1.000000000000000E-008

minimum data covariance eigenvalue = 1.000000000000000E-012

lrate factor = 0.500000000000000

initial rholrate = 5.000000000000000E-002

rho0 = 1.50000000000000

min rho = 1.00000000000000

max rho = 2.00000000000000

rho lrate factor = 0.500000000000000

kurt_start = 3

num kurt = 5

kurt interval = 1

do_newton = 1

newt_start = 50

newt_ramp = 10

initial newton lrate = 1.00000000000000

do_reject = 0

num reject = 3

reject sigma = 3.00000000000000

reject start = 2

reject interval = 3

write step = 20

write_nd = 0

write_LLt = 1

dec window = 1

max_decs = 3

fix_init = 0

update_A = 1

update_c = 1

update_gm = 1

update_alpha = 1

update_mu = 1

update_beta = 1

invsigmax = 100.000000000000

invsigmin = 0.000000000000000E+000

do_rho = 1

load_rej = 0

load_c = 0

load_gm = 0

load_alpha = 0

load_mu = 0

load_beta = 0

load_rho = 0

load_comp_list = 0

do_mean = 1

do_sphere = 1

pcakeep = 127

pcadb = 30.0000000000000

byte_size = 4

doscaling = 1

scalestep = 1

mkdir: cannot create directory ‘/projects/ps-nsg/home/nsguser/ngbw/workspace/NGBW-JOB-EEGLAB_TG-DCA1920B6CF94C7F9E354186B73EBEDF/23456/amicaouttmp/’: File exists

output directory =

/projects/ps-nsg/home/nsguser/ngbw/workspace/NGBW-JOB-EEGLAB_TG-DCA1920B6CF94C7

F9E354186B73EBEDF/23456/amicaouttmp/

       2 : setting num_thrds to           24  ...

       3 : setting num_thrds to           24  ...

       1 : setting num_thrds to           24  ...

       4 : setting num_thrds to           24  ...

       4 : using          24 threads.

       1 : using          24 threads.

       2 : using          24 threads.

       3 : using          24 threads.

       1 : node_thrds =           24          24          24          24

bytes in real = 1

       1 : REAL nbyte =            1

getting segment list ...

blocks in sample = 4915200

total blocks = 4915200

node blocks = 1228800 1228800 1228800 1228800

node 1 start: file 1 sample 1 index

       1

node 1 stop : file 1 sample 1 index

 1228800

node 2 start: file 1 sample 1 index

 1228801

node 2 stop : file 1 sample 1 index

 2457600

node 3 start: file 1 sample 1 index

 2457601

node 3 stop : file 1 sample 1 index

 3686400

node 4 start: file 1 sample 1 index

 3686401

node 4 stop : file 1 sample 1 index

 4915200

       1 : data =    11.9744024276733        4.24514341354370     

       3 : data =   -7.78323650360107      -0.299957871437073     

       2 : data =    1.66699647903442       -26.0488491058350     

       4 : data =   -2.59526801109314       -6.62773466110229     

getting the mean ...

mean = -1.36149215149254 -0.697963101435924

-0.908440884302629

subtracting the mean ...

getting the covariance matrix ...

cnt = 4915200

doing eig nx = 127 lwork = 161290

doing eig nx = 127 lwork = 161290

doing eig nx = 127 lwork = 161290

doing eig nx = 127 lwork = 161290

minimum eigenvalues = 2.439452986291257E-002 2.24693689752528

2.29249564029623

minimum eigenvalues = 2.439452986291257E-002 2.24693689752528

2.29249564029623

maximum eigenvalues = 13003.3767440951 1393.52042042707

1052.95147913640

num eigs kept = 0

maximum eigenvalues = 13003.3767440951 1393.52042042707

1052.95147913640

num eigs kept = 127

getting the sphering matrix ...

minimum eigenvalues = 2.439452986291257E-002 2.24693689752528

2.29249564029623

maximum eigenvalues = 13003.3767440951 1393.52042042707

1052.95147913640

num eigs kept = 127

sphering the data ...

minimum eigenvalues = 2.439452986291257E-002 2.24693689752528

2.29249564029623

maximum eigenvalues = 13003.3767440951 1393.52042042707

1052.95147913640

num eigs kept = 0

numeigs = 127

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/japalmer29/amica/issues/24#issuecomment-717630151 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRBESRKYZAFWNWWGIO3UULSM5UPLANCNFSM4S6YAJHQ . https://github.com/notifications/beacon/ACRBESU3P5R6EGRAMPG3L3DSM5UPLA5CNFSM4S6YAJH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFLDCVRY.gif