mfromano
commented
9 years ago Going to try to recreate it now. I'll let you know how it goes. I do think that it would be a good idea to do one giant merge at some point. It will suck but I think that it would be a good idea.
Best, Mike
On Thu, Jul 23, 2015 at 3:36 PM, asoplata notifications@github.com wrote:
This is a really weird one, since I can't figure out how I haven't run across it before, nor why it hasn't occurred for anyone else. Note that everything I say will apply to trying to run a cluster job; local sims are a different story. I first noticed this when trying to run my own cells, but I gradually simplified it down to even occurring on the Example Batch Script that Jason has sent some people (sent to me on 2015/03/27).
Can other people PLEASE try recreating this bug (e.g. just running the script below, only needing to change codepath) using up-to-date copies of the GitHub version of DNSIM or the other altered versions of DNSIM (e.g. ones they got from Jason directly), if you use those versions on the cluster? Basically, whichever version you use on the cluster.
If I take this minimal/extremely simple MATLAB script, where the only simulation run is the one sent to the cluster but not using coder, and I try to run it with multiplicity set to > 1, then I get an Artifex Ghostscript system error in which case the system is unable to save any voltage trace plots (and other plots).
%% Demo: DNSim batch simulations% Purpose: demonstrate how to run simulation batches varying model parameters% on local machine or cluster with or without codegen.% Created by Jason Sherfey (27-Mar-2015)% -------------------------------------------------------------------------% model specification spec=[]; spec.nodes(1).label = 'HH'; % name of this node population spec.nodes(1).multiplicity = 2; % size of this node population spec.nodes(1).dynamics = {'V''=current/c'}; % node dynamics for state variables that can be "seen" by other nodes spec.nodes(1).mechanisms = {'iNa','iK','itonic'}; % note: corresponding mechanism files (e.g., iNa.txt) should exist in matlab path spec.nodes(1).parameters = {'stim',7,'c',1}; % user-specified parameter values (note: these override default values set in mechanism files) % simulation controls tspan=[0 3000]; % [beg end], ms, simulation time limits SOLVER='euler'; % numerical integration method dt=.01; % integration time step [ms] dsfact=10; % downsample factor (applied to simulated data)% THIS IS THE ONLY THING YOU NEED TO CHANGE% THIS IS THE ONLY THING YOU NEED TO CHANGE codepath='~/x010-dnsim/dnsim'; % path to dnsim toolbox% THIS IS THE ONLY THING YOU NEED TO CHANGE% THIS IS THE ONLY THING YOU NEED TO CHANGE % %% SINGLE SIMULATION% % local simulation with codegen% data = runsim(spec,'timelimits',tspan,'dt',dt,'SOLVER',SOLVER,'dsfact',dsfact,...% 'coder',1,'verbose',0);% plotv(data,spec,'varlabel','V'); %% SIMULATION BATCHES % output controls rootdir = pwd; % where to save outputs savedata_flag = 0; % 0 or 1, whether to save the simulated data (after downsampling) saveplot_flag = 1; % 0 or 1, whether to save plots plotvars_flag = 1; % 0 or 1, whether to plot state variables % what to vary across batch scope = 'HH'; % node or connection label whose parameter you want to vary variable = 'c'; % parameter to vary (e.g., capacitance) values = '[1]'; % values for each simulation in batch (varied across batches) %{ Prerequisites for submitting jobs to cluster: 1. add path-to-dnsim/csh to your environment. e.g., add to .bashrc: "export PATH=$PATH:$HOME/dnsim/csh" 2. run matlab on cluster node that recognizes the command 'qsub'%} % % -------------------------------------------------------------------------% % LOCAL BATCHES (run serial jobs on current machine)% cluster_flag=0; % 0 or 1, whether to submit jobs to a cluster (requires: running on a cluster node that recognizes the command 'qsub')% % % without codegen% values = '[1 1.5]';% [~,~,outdir]=simstudy(spec,{scope},{variable},{values},...% 'dt',dt,'SOLVER',SOLVER,'rootdir',rootdir,'timelimits',tspan,'dsfact',dsfact,...% 'savedata_flag',savedata_flag,'saveplot_flag',saveplot_flag,'plotvars_flag',plotvars_flag,'addpath',codepath,...% 'cluster_flag',cluster_flag,'coder',0);% % % with codegen% values = '[2 2.5]';% [~,~,outdir]=simstudy(spec,{scope},{variable},{values},...% 'dt',dt,'SOLVER',SOLVER,'rootdir',rootdir,'timelimits',tspan,'dsfact',dsfact,...% 'savedata_flag',savedata_flag,'saveplot_flag',saveplot_flag,'plotvars_flag',plotvars_flag,'addpath',codepath,...% 'cluster_flag',cluster_flag,'coder',1); % -------------------------------------------------------------------------% CLUSTER BATCHES (submit to queue for running parallel jobs on cluster nodes) cluster_flag=1; % 0 or 1, whether to submit jobs to a cluster (requires: running on a cluster node that recognizes the command 'qsub') % without codegen% values = '[3 3.5]'; [~,~,outdir]=simstudy(spec,{scope},{variable},{values},... 'dt',dt,'SOLVER',SOLVER,'rootdir',rootdir,'timelimits',tspan,'dsfact',dsfact,... 'savedata_flag',savedata_flag,'saveplot_flag',saveplot_flag,'plotvars_flag',plotvars_flag,'addpath',codepath,... 'cluster_flag',cluster_flag,'coder',0); % % with codegen% values = '[4 4.5]';% [~,~,outdir]=simstudy(spec,{scope},{variable},{values},...% 'dt',dt,'SOLVER',SOLVER,'rootdir',rootdir,'timelimits',tspan,'dsfact',dsfact,...% 'savedata_flag',savedata_flag,'saveplot_flag',saveplot_flag,'plotvars_flag',plotvars_flag,'addpath',codepath,...% 'cluster_flag',cluster_flag,'coder',1);
This is the job1.out output log, which shows the error message
command = job1 cwd = /usr3/graduate/asoplata/batchdirs/B20150723-145627 HOSTNAME = scc-ba3.scc.bu.edu JOB_NAME = 5880217 JOB_NAME = B20150723-145627_job1 < M A T L A B (R) > Copyright 1984-2013 The MathWorks, Inc. R2013a (8.1.0.604) 64-bit (glnxa64) February 15, 2013
To get started, type one of these: helpwin, helpdesk, or demo. For product information, visit www.mathworks.com.
Looking in known mech list for iNa Found mech file. Using: ~/research/modeling/database/iNa.txt Looking in known mech list for iK Found mech file. Using: ~/research/modeling/database/iK.txt Looking in known mech list for itonic Found mech file. Using: ~/research/modeling/database/itonic.txt Model Description----------------------------------
Specification files:
Populations: HH (n=2): dynamics: V' = ((-HH_iNa_INaf(HH_V,HH_iNa_mNaf,HH_iNa_hNaf))+((-HH_iK_IKf(HH_V,HH_iK_nKf))+((HH_itonic_Itonic(t))+0)))/(1) mechanisms: iNa, iK, itonic parameters: stim=7, c=1
Connections: HH HH -
Connection parameters: Model Equations:---------------------------------- ODEs: HH_V' = ((-((120)._HH_iNa_mNaf.^3._HH_iNahNaf.(HH_V-(50))))+((-((36)._HH_iKnKf.^4.(HHV-(-77))))+((((7)(t>(0) & t<HH_itonic_offset)))+0)))/(1) HH_iNamNaf' = ((2.5-.1(HHV+65))./(exp(2.5-.1(HHV+65))-1)).(1-HH_iNa_mNaf)-(4_exp(-(HH_V+65)/18))._HH_iNa_mNaf; HH_iNa_hNaf' = (.07_exp(-(HHV+65)/20)).(1-HH_iNahNaf)-(1./(exp(3-.1(HH_V+65))+1))._HH_iNa_hNaf; HH_iKnKf' = ((.1-.01(HHV+65))./(exp(1-.1(HHV+65))-1)).(1-HH_iK_nKf)-(.125_exp(-(HH_V+65)/80))._HH_iK_nKf;
Initial Conditions: HH_V(0) =[0, 0, ] HH_iNa_mNaf(0) =[0, 0, ] HH_iNa_hNaf(0) =[0, 0, ] HH_iK_nKf(0) =[0, 0, ]
Matlab-formatted (copy and paste to repeat simulation):%-----------------------------------------------------------% Auxiliary variables: HH_itonic_offset = inf; % Anonymous functions: HH_iNaaM = @(IN) (2.5-.1(IN+65))./(exp(2.5-.1_(IN+65))-1); HH_iNa_bM = @(IN) 4_exp(-(IN+65)/18); HH_iNa_aH = @(IN) .07_exp(-(IN+65)/20); HH_iNabH = @(IN) 1./(exp(3-.1(IN+65))+1); HH_iNa_INaf = @(IN,m,h) (120)._m.^3.h.(IN-(50)); HH_iKaN = @(IN) (.1-.01(IN+65))./(exp(1-.1_(IN+65))-1); HH_iK_bN = @(IN) .125_exp(-(IN+65)/80); HH_iK_IKf = @(IN,n) (36).n.^4.(IN-(-77)); HH_itonicItonic = @(t) (7)(t>(0) & t<HH_itonic_offset); % ODE Handle, ICs, integration, and plotting: ODEFUN = @(t,X) [((-((120).X(3:4).^3.X(5:6).(X(1:2)-(50))))+((-((36).X(7:8).^4.(X(1:2)-(-77))))+((((7 )(t>(0) & t<HH_itonicoffset)))+0)))/(1);((2.5-.1(X(1:2)+65))./(exp(2.5-.1(X(1:2)+65))-1)).(1-X(3:4))-(4_exp(-(X(1:2)+65)/18))._X(3:4);(.07exp(-(X(1:2)+65)/20)).(1-X(5:6))-(1./(exp(3-.1_(X(1:2)+65))+1)).X(5:6);((.1-.01(X(1:2)+65))./(exp(1-.1(X(1:2)+65))-1)).(1-X(7:8))-(.125_exp(-(X(1:2)+65)/80))._X(7:8);]; IC = [0 0 0 0 0 0 0 0];
[t,y]=ode23(ODEFUN,[0 100],IC); % numerical integrationfigure; plot(t,y); % plot all variables/functionstry legend('HH_V','HH_iNa_mNaf','HH_iNa_hNaf','HH_iK_nKf'); end%-----------------------------------------------------------
odefun_file =
odefun_20150723_145703_364_job0001
Simulation interval: 0-3000 Starting integration (euler, dt=0.01) Processed 600 of 3000 ms (elapsed time: 0.715 s) Processed 1200 of 3000 ms (elapsed time: 1.798 s) Processed 1800 of 3000 ms (elapsed time: 2.876 s) Processed 2400 of 3000 ms (elapsed time: 3.956 s) Processed 3000 of 3000 ms (elapsed time: 5.034 s) saving plots - voltage traces: /usr3/graduate/asoplata/x010-dnsim/dnsim/HH-c/20150723/images/rawV/job0001_HH-c1_time0-3000_rawV Error: Problem converting PostScript. System returned error: -1.Failed to convert to output format; Ghostscript status: -100.Error: /rangecheck in --string--heck Operand stack: picstr 90003 Execution stack: %interp_exit .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- --nostringval-- --nostringval-- false 1 %stopped_push 1 3 %oparray_pop 1 3 %oparray_pop 1 3 %oparray_pop 1 3 %oparray_pop .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- Dictionary stack: --dict:1121/1686(ro)(G)-- --dict:0/20(G)-- --dict:71/200(L)-- --dict:94/160(L)-- --dict:8/76(L)-- Current allocation mode is local Last OS error: 2 Current file position is 657900 Artifex Ghostscript 8.54: Unrecoverable error, exit code 1
in ghostscript (line 186) in LocalPrint (line 314) in print (line 240) in biosimdriver (line 196) in job1 (line 1)saving cluster log (/usr3/graduate/asoplata/batchdirs/B20150723-145627/pbsout/job1.out) to /usr3/graduate/asoplata/x010-dnsim/dnsim/HH-c/20150723/logs/job0001_HH-c1_time0-3000_job1.out. sim_data and spec assigned to base workspace. finished
I have tried submitting the jobs/running the above script both through the CLI with/without X-Forwarding, and even in the MATLAB command window inside a VNC session, and the error always occurs, so I'm pretty sure it's not due to 'how' I'm submitting it to the cluster. I've run it with both the regular MATLAB 2013a and also the MATLAB 2014b available on the cluster (Thanks for showing me that, Mike), and so if it's a MATLAB version problem...then it's been around for a long and it's still around.
What I'm most concerned about is that no one else has reported/made mention of this issue, even though Jason, and AFAIK at least Julia and Ben, have been running cluster batches where some celltype multiplicites are greater than 1. While yes, this does in part sound like a screw-up on the cluster's (or MATLAB's) part in that MATLAB's cluster stuff isn't handling the graphics printing/saving correctly (apparently MATLAB first prints figures into a PostScript thing, which apparently requires live graphics support by the OS, before printing that to, say, a PNG), judging from the previous sentence, we have working DNSIM code where this error is not encountered. Therefore, it is possible that there is some difference between the DNSIM code that is on the GitHub and the altered DNSIM code people are using in the 'wild' that may be responsible for this bug. If that is the cause of the bug, then it's going to be one NASTY fix.
It could also just be a difference between my 'setup' somehow and someone else's which is why I really need someone else to test at least the GitHub-default ('dev' branch) version of the code for this bug on their end! Based on the simplicity of the simulation I'm running, I REALLY don't think it's an issue with my specific pipeline, though.
Just to reiterate, until we figure this out, or I switch to a 'wild/non-GitHub' version of the DNSIM code that doesn't experience this problem, I/people using the GitHub version CANNOT run sims on the cluster that use a multiplicity >1, aka a simulation that uses more than 1 of a certain celltype.
Thanks for any help.
— Reply to this email directly or view it on GitHub https://github.com/jsherfey/dnsim/issues/56.
Michael Romano
M.D./Ph.D. candidate Boston University School of Medicine 203-521-5575
kupiqu
asoplata
This is a really weird one, since I can't figure out how I haven't run across it before, nor why it hasn't occurred for anyone else. Note that everything I say will apply to trying to run a cluster job; local sims are a different story. I first noticed this when trying to run my own cells, but I gradually simplified it down to even occurring on the Example Batch Script that Jason has sent some people (sent to me on 2015/03/27).
Can other people PLEASE try recreating this bug (e.g. just running the script below, only needing to change
codepath) using up-to-date copies of the GitHub version of DNSIM or the other altered versions of DNSIM (e.g. ones they got from Jason directly), if you use those versions on the cluster? Basically, whichever version you use on the cluster.If I take this minimal/extremely simple MATLAB script, where the only simulation run is the one sent to the cluster but not using coder, and I try to run it with
multiplicityset to > 1, then I get anArtifex Ghostscriptsystem error in which case the system is unable to save any voltage trace plots (and other plots). Edit: Note that you have to change where your DNSIM code is installed, but other than that, if you've ever run a sim on the cluster successfully before, then this should work without you having to change anything else.This is the
job1.outoutput log, which shows the error messageI have tried submitting the jobs/running the above script both through the CLI with/without X-Forwarding, and even in the MATLAB command window inside a VNC session, and the error always occurs, so I'm pretty sure it's not due to 'how' I'm submitting it to the cluster. I've run it with both the regular MATLAB 2013a and also the MATLAB 2014b available on the cluster (Thanks for showing me that, Mike), and so if it's a MATLAB version problem...then it's been around for a long and it's still around.
What I'm most concerned about is that no one else has reported/made mention of this issue, even though Jason, and AFAIK at least Julia and Ben, have been running cluster batches where some celltype multiplicites are greater than 1. While yes, this does in part sound like a screw-up on the cluster's (or MATLAB's) part in that MATLAB's cluster stuff isn't handling the graphics printing/saving correctly (apparently MATLAB first prints figures into a PostScript thing, which apparently requires live graphics support by the OS, before printing that to, say, a PNG), judging from the previous sentence, we have working DNSIM code where this error is not encountered. Therefore, it is possible that there is some difference between the DNSIM code that is on the GitHub and the altered DNSIM code people are using in the 'wild' that may be responsible for this bug. If that is the cause of the bug, then it's going to be one NASTY fix.
It could also just be a difference between my 'setup' somehow and someone else's which is why I really need someone else to test at least the GitHub-default ('dev' branch) version of the code for this bug on their end! Based on the simplicity of the simulation I'm running, I REALLY don't think it's an issue with my specific pipeline, though.
Just to reiterate, until we figure this out, or I switch to a 'wild/non-GitHub' version of the DNSIM code that doesn't experience this problem, I/people using the GitHub version CANNOT run sims on the cluster that use a multiplicity >1, aka a simulation that uses more than 1 of a certain celltype.
Thanks for any help.