Simulations with differen corner-library (`.lib ../.../cornerMOSlv_psp.lib mos_tt and ...ss ...ff ...fs ...sf`), what would You do?

[StefanSchippers]

@olisnr

if You want to make some simulations with differen corner-library (.lib ../.../cornerMOSlv_psp.lib mos_tt and ...ss ...ff ...fs ...sf), what would You do? i can use a launcher.sym to set a lib-variable and then xschem netlist; xschem simulate but i didnt figured out, how to wait of the end of the simulation before update the lib-variable, and simulate again. or i could use TCL to write different spice-netlists and run them in parallel (also without knowing when it ended)...

now it looks like:

and the possibility to load different .raw-files is REALLY great!

Originally posted by @olisnr in https://github.com/StefanSchippers/xschem/issues/141#issuecomment-1816409510

[StefanSchippers]

@olisnr, Opened a new issue-thread since the other grew too long.

Waiting for a simulation (or some )to finish is not difficult. If you want to run more in parallel (this is ideal for corner sims since these runs can go independently) you need to ensure output files go in different places. You can define some tcl variables in the code blocks (and use tcleval(...) ).

Then to run some sims and wait do the following:

#set tcl variables for 1st run
xschem netlist
set id1 [simulate]

#set tcl variables for 2nd run
xschem netlist
set id2 [simulate]

#set tcl variables for 3rd run
xschem netlist
set id3 [simulate]

# wait for completion of all runs.If they are different corners they should finish in similar times

vwait execute(cmd,$id1)
vwait execute(cmd,$id2)
vwait execute(cmd,$id3)

execute(cmd,id) is the command line of the running process. This variable (and others related to this simulation) get deleted at the end (except the new execute(exitcode,id) ) . so we can vwait for them (vwait blocks until something happens with the variable).

PS: simulate is the recommended way to launch simulations instead of the legacy xschem simulate. the simulate procedure returns an integer id to use with the execute variables:

# $execute(id) is an integer identifying the last running pipeline
#       set id $execute(id) to get the current subprocess id in variable 'id'
# $execute(status,$id) contains the status argument as given in proc execute
# $execute(pipe,$id) contains the channel descriptor to read or write as specified in status
# $execute(data,$id) contains the stdout of the pipeline (output data)
# $execute(cmd,$id) contains the pipeline command
# $execute(win_path,$id) contains the xctx->current_win_path that started the command
# $execute(exitcode,$id) contains the exit code (this is a new addition in xschem)

[StefanSchippers]

Following an old request where you were asking to use tcl variables in node names / datasets / whatever in the graph dialog, the new approach is the following. Instead of wrapping some parts with tcleval() just wrap the entire list of nodes inside a tcleval(....) as shown below. The second node is completely specified by variables (alias name, signal, dataset, raw file):

[olisnr]

many thanks! i was trying with the $execute(...) but it wasnt working. the parallel code looks super, i will test it.

and im very happy with plotting different simulations together.

do You have an idea, how one could annotate an already open sub-sheet? or go back to the simulation controlling top-sheet without closing the sub-sheet?

[StefanSchippers]

many thanks! i was trying with the $execute(...) but it wasnt working. the parallel code looks super, i will test it.

and im very happy with plotting different simulations together.

do You have an idea, how one could annotate an already open sub-sheet? or go back to the simulation controlling top-sheet without closing the sub-sheet?

May be in the parallel code above you need to add a delay (after 1000) . If ngspice has not yet read the netlist and you generate the netlist for the second run you get a mess. Another solution is to use different netlist names (xschem set netlist_name xxxx.spice) for the runs, so you avoid such race conditions.

[StefanSchippers]

do You have an idea, how one could annotate an already open sub-sheet? or go back to the simulation controlling top-sheet without closing the sub-sheet?

Some additional work needs to be done, but you can annotate now a subsheet without leaving the subsheet. See video clip.

https://github.com/StefanSchippers/xschem/assets/69359491/847b1068-153c-4877-9b6f-1594b8da9dcb

[olisnr]

thats great. is there a way to get the name of the top-sheet from the child? then i could make an universal annotate launcher.sym

in the doc i didnt found something.

[olisnr]

i tried the child annotation, but something didnt work correct:

https://github.com/StefanSchippers/xschem/assets/120017377/78174102-0804-44e6-a3c7-6a97aefaae27

[StefanSchippers]

Yes you can get the name (full path) of parent levels with xschem get schname n where n is the hierarchy level (0= top)

[StefanSchippers]

i tried the child annotation, but something didnt work correct:

It did not work since you opened the sub sheet with Alt-E. This just opens a new top level. Do the following steps.

• open the top schematic .
• descend with 'e' into the subsheet.
• open a new empty tab
• open again the top level. You can use this top level to run / change simulations.
• the tab with the subsheet should be able to annotate the OP data with your launcher.

My plan is to make this work also with Alt-E, of course... may be in the following commits

[olisnr]

the launcher works after e using: tclcommand="xschem annotate_op ${netlist_dir}/[file rootname [file tail [xschem get schname 0]]].raw 0"

thanks!

[olisnr]

maybe the doc for

raw_clear
   Delete loaded simulation raw file

should be unload instead of delete

[olisnr]

some other thing i would like in the documentation:

_in a tcleval() we do a tcl command and variable substitution. This means inside the tcleval() there is a string. The result is the string with tcl variable substitution (any $var or ${var} (better) instances will be replaced with the var content) and command substitution (any [command] instances will be replaced with the return value of the tcl command). Of course tcl commands can do various side effects, like setting variables, printing stuff, even creating new procedures. The tcleval() is processed by removing the tcleval( and the ending ), then passing the string to the tcl subst command.

A tclcommand attribute in the launcher is just taken as is (no tcleval() needed) and passed to the eval tcl function. So for example in a launcher you don't need to wrap the command into [...].

The tcleval() construct is done also for efficiency reasons. If an attribute does not begin wth tcleval() no call to the tcl interpreter is done. A C strstr() check on the tcleval( string is super fast.

The difference between eval and subst is that any string is valid for subst (if no commands or variables are found it is returned as is, and this is what we want for string attributes. tclcommand attributes are directly passed to the eval function and must be valid tcl programs just as you are typing them at the tcl prompt. All the executions mentioned are done in the global scope._

[olisnr]

im trying the multithreading code, but i have problems with it. now it looks like:

tclcommand="
    set corners [list tt ss ff sf fs]

    foreach corner $corners { ### debug
        puts $corner
        puts [string cat {$id_} ${corner}]
    }

    foreach corner $corners {
        set hv_corner $corner; 
        xschem netlist; 
        delay 111;
        set [string cat {id_} ${corner}] [simulate]; 
    }

    foreach corner $corners {
        puts [string cat {$execute(cmd, } [string cat {$id_} ${corner}] \)]; 
        vwait [string cat {$execute(cmd, } [string cat {$id_} ${corner}] \)]; 
    }

about ### debug

    xschem raw_clear;
    xschem raw_read [rawfilestart].raw; 
    xschem annotate_op;
"

it says:

tt
$id_tt
ss
$id_ss
ff
$id_ff
sf
$id_sf
fs
$id_fs
Simulation started: execution ID: 178
Simulation started: execution ID: 179
Simulation started: execution ID: 180
Simulation started: execution ID: 181
Simulation started: execution ID: 182
$execute(cmd, $id_tt)

it simulates something, but hangs vor ever in vwait [string cat {$execute(cmd, } [string cat {$id_} ${corner}] \)]; ( vwait $execute(cmd, $id_tt) )...

i didnt found vwait in the doc, and im not really sure how it should work

[StefanSchippers]

vwait is a tcl command so get the info with man vwait. also remove the space after cmd, in these lines:

        puts [string cat {$execute(cmd, } [string cat {$id_} ${corner}] \)]; 
        vwait [string cat {$execute(cmd, } [string cat {$id_} ${corner}] \)]; 

as execute(cmd, n) and execute(cmd,n) (where n is an integer) are 2 different variables. In the first case you are waiting fore a variable that does not exists and so forever.

[StefanSchippers]

I think also the:

vwait [string cat {$execute(cmd, } [string cat {$id_} ${corner}] \)]

should be:

vwait [string cat {execute(cmd,} [string cat {$id_} ${corner}] {)}]

[olisnr]

yes i changed it, now it produces execute(cmd,$id_tt) but still stalls. there is also something other wrong.

[StefanSchippers]

I was wrong, it seems you need to remove the $ before execute ad the space after cmd,

[StefanSchippers]

also do a test with one corner only

[olisnr]

i did this also. it produces vwait execute(cmd,$id_tt) now, and the variable $id_tt has the id of the simulation.

[StefanSchippers]

Try this command on a circuit that takes some seconds to simulate. Does this work? (to me it does):

set id [simulate]; vwait [string cat {execute(cmd,} $id {)}]

[olisnr]

Your line is working, and my code is also working for only one thread iF i use the id as number and not as variable:

        vwait [string cat {execute(cmd,} [string cat {$id_} ${corner}] \)]; # blocks vor ever
        vwait [string cat {execute(cmd,} [set [string cat {id_} ${corner}]] \)]; # works...

if i start all simulations, then mostly it stalls after the 2nd vwait, but sometimes it event riches the 4th. may be its a question of the ending times of the different threads

[StefanSchippers]

Yes I think something must be implemented here... if a vwait on one variable loses a write on another variable the second vwait stalls forever. Will think for a simpler solution. May be you can test a polling loop:

while {1} {
  delay 1000 ;# check every second
  if { ![info exists var1] && ![info exists var2] } {
    # all variables removed, jobs finished
    break
}

something like that. var1 and var2 are the execute(cmd,n) variables.

[StefanSchippers]

I have managed this script to work for 2 parallel simulations:

xschem load poweramp.sch

xschem netlist
set id1 [simulate]
delay 1000

# change VPP and output file name
set commands [xschem getprop instance NGSPICE value]
regsub {\.param VPP=50} $commands {.param VPP=30} commands
regsub -all {poweramp\.raw} $commands {poweramp1.raw} commands
xschem setprop instance NGSPICE value $commands

xschem netlist
set id2 [simulate]
delay 1000

while { 1} {
  delay 1000
  if { ![info exists execute(cmd,$id1)] && ![info exists execute(cmd,$id2)]} {break}
}

puts done

see video clip:

https://github.com/StefanSchippers/xschem/assets/69359491/60e0c3c0-2a7f-415c-b7e6-62bc40f39172

[olisnr]

it wanted it in a loop:

    while {true} {
        set info_exists false;
        foreach corner $corners {
            set info_exists expr {$info_exists || [info exists [set [string cat {id_} ${corner}]]]}
        }
        if {!info_exists} {
            break;
        }
        delay 333
    }

but the set info_exists expr {$info_exists || [info exists [set [string cat {id_} ${corner}]]]} dont work. even set info_exists expr {$info_exists} isnt possible. why? expr {$info_exists} returns a false, but i cant load the variable whit it...

[StefanSchippers]

what is the output of

string cat {id_} ${corner}

? I don't see the execute(....

set info_exists expr {$info_exists} ? may be you meant: set info_exists [expr {$info_exists}]

[StefanSchippers]

I would do something like this:

# run simulations
set id_list {}
foreach corner $corners {
  # set corner variables
  xschem netlist
  set id [simulate]
  lappend id_list $id
  delay 1000 ;# let simulator the time to read netlist
}

# poll loop
while {1} {
  delay 1000
  set busy 0
  foreach id $id_list {
    set var "execute(cmd,$id)"
    if {[info exists $var]} { set busy 1}
  }
  if {!$busy} { break }
}
puts Done

the poll loop works, tested with 3 parallel simulations.

[StefanSchippers]

I have implemented the "true descend sub schematic in new tab/window" (the Alt-E key). See video.

https://github.com/StefanSchippers/xschem/assets/69359491/b6254710-e256-4ec1-ace8-8290f1c37b35

[olisnr]

super! i love this!!!

when make a new simulation, then i guess i have to reload the sub-sheets individual, or are the updated data used for all plots automatic?

[olisnr]

Xschem enjoys the address range of 32bit :)

[StefanSchippers]

super! i love this!!!

when make a new simulation, then i guess i have to reload the sub-sheets individual, or are the updated data used for all plots automatic?

Currently you have to reload the data in the subsheets. It could be possible in principle to share a single raw file between the top level and the subsheets, but this is a complex thing to handle. In general additional tabs or windows (in the same xschem process) are designed to be independent. What happens in one schematic on tab 'x' does not cause any side effect on another schematic in tab 'y'. This will be the next step. When the master deletes the raw file all sub schematics that share the same raw file must get notiified otherwise big segfaults will happen.

[StefanSchippers]

Xschem enjoys the address range of 32bit :)

how much raw files did you load?

[olisnr]

15

[StefanSchippers]

15

yes but total file size?

ngspice stores data as doubles (8 bytes / node/point) xschem converts to floats (4 bytes/ node/point) so the total size in memory should be 50%of the total file size. 7 significant digits of floats is more than enough in 99.995% of all simulations. There is though a compile option in xschem to keep data as doubles in memory.

[olisnr]

the data are 153MB on disk, and a new openened Xschem needs 143MB RAM for it. i was trying to get the TCL-scripts working, and at some time there i saw the memory consumption, i had many errors in my scripts, and may be there where data loaded or generated and never unloaded of may be there is a memory hole or two :)

i played a bit with loading/unloading and simulating, and i think there is a problem with unloading data: unloading does mostly not free memory, and loading needs more.

also the plots get strange peaks (that seems to be not real):

if You want to try Your self: aa_opamp_bias_test_corner_05.sch.zip

[olisnr]

for the float / double question: my experience is, if You really want to do something whit the data: integrating, IQ demodulation, filtering or the like, float produces very fast big errors. i would prefer double, if You want to do something more then plotting in future.

[StefanSchippers]

for the float / double question: my experience is, if You really want to do something whit the data: integrating, IQ demodulation, filtering or the like, float produces very fast big errors. i would prefer double, if You want to do something more then plotting in future.

you can change this line:

#define SPICE_DATA float

to double in xschem.h. Consider however that simulation data has 3-4 significant digits.

[StefanSchippers]

also the plots get strange peaks (that seems to be not real):

the AC plots are broken because all 5 simulation processes write into the same [rawfilestart]_ac.raw file. Add corner info like you did for the other files. also op data is written by 5 processes into the same [rawfilestart].raw file. also there is a write done into [rawfilestart]_dc.raw you must differentiate these output files per single process.

[StefanSchippers]

I have changed the schematic to avoid concurrent ngspice writes into the same raw files. With all data loaded the xschem process uses 100MB of resident memory, and plots seem to be correct (no spikes) stef.zip

[olisnr]

thanks for fixing!

this was stupid! i didnt think about it, i was only trying learn TCL...

what do You think about the memory? do You think there is a problem? if i repeat the same simulation many times, i would expect that the memory-usage stays at some point constant. may be 2 times the sim-data or so.

[StefanSchippers]

after the fix the raw files are no more corrupt and I don't see memory issues. you can run xschem in full debug:

xschem stef.sch -d 3 -l ~/log

load and unload the sim data several tines, then quit the program, then go into the xschem sources src/ directory and run: ./track_memory.awk ~/log you get:

schippes@asus:~/xschem-repo/trunk/src$ ./track_memory.awk ~/log
peak allocated memory = 76508366
Total allocated memory after exit = 0
# of malloc's = 192731
# of realloc's = 367867
# of free's = 257441
Number of unfreed pointers = 0
Total leaked memory = 0

so a maximum of 76.5MB was allocated (process size is bigger since it includes all the code and library data), all the data was released at the end and no dangling pointer was left.

https://github.com/StefanSchippers/xschem/assets/69359491/5708e4f0-9b15-4e9c-90fa-528d4fa83927

[olisnr]

if im correct, then You write about loosed handles or pointers to allocated memory. i was writing about that with every simulation, Xschem allocate additional memory. and that even if i reload the schematic, the memory remains allocated (the last version is better!). i wonder why.

i did a boring video to demonstrate it:

https://github.com/StefanSchippers/xschem/assets/120017377/009a8b08-f097-400f-8ee3-690f3ceca965

after a some more simulations:

https://github.com/StefanSchippers/xschem/assets/120017377/95663f33-2c6f-49d6-b752-7a7096d9af4d

[olisnr]

now i found that the iHPs openPDK is updated, and i wanted to run some of the test-schematics. i had to change the library-paths a little bit, and was able to simulate and plot. but after loading the plot-data Xschem dosent accept any mouse selection in the schematic any more. also tab-switching dont work. but the menue is ok and also the graphdialog.

i never saw this before. i tried with the openPDK xschemrc and with my own (+ and path-append for the symbols in a click-arrow)

https://github.com/StefanSchippers/xschem/assets/120017377/c0209929-1d56-467b-8fad-cc9e2c599166

tell me if should try something.

zooming is still working. for me it feels like there is an invisible open dialog...

[StefanSchippers]

Yes this usually happen if a dialog has been left open somewhere, may be obscured... What does the Load sym launcher do?

[StefanSchippers]

can you please check this? xschem get semaphore this should normally return 0 if it is not 0 xschem is half locked, meaning you can not change the circuit, select, change tabs etc. You can force this to be zero and xschem should then work as usual. xschem set semaphore 0 Problem is understanding what caused the semaphore to remain "red" (1)

[olisnr]

name=h1
descr="load sym Ctrl + left click" 
tclcommand="
append XSCHEM_LIBRARY_PATH :/ALL/Xschem/IHP-Open-PDK-main/ihp-sg13g2/libs.tech/xschem
"

[olisnr]

% xschem get semaphore
1

yes somebody blocks it after loading the dc-sim. it says:

% xschem my_strdup2(): WARNING: src == *dest == 0x562377db9a20, id=0
Simulation started: execution ID: 23
free_rawfile(): clearing data
free_rawfile(): clearing data
free_rawfile(): clearing data
free_rawfile(): clearing data
Raw file data read: /ALL/Xschem/IHP-Open-PDK-main/ihp-sg13g2/libs.tech/xschem/sg13g2_tests/simulation/dc_hv_nmos.raw
points=3010, vars=2, datasets=1 sim_type=dc
xschem get semaphore
% 
% xschem get semaphore
1
% xschem set semaphore 0
% 

it seems that only the "Wave" dialog blocks. and something other strange is: i tried to load the same file using two different launchers. one is working, the other not, but both try to load the same file:

working:

 name=h5
descr="load waves Ctrl + left click" 
tclcommand="xschem raw_read $netlist_dir/dc_hv_nmos.raw dc" 

->

% Raw file data read: /ALL/Xschem/IHP-Open-PDK-main/ihp-sg13g2/libs.tech/xschem/sg13g2_tests/simulation/dc_hv_nmos.raw
points=3010, vars=2, datasets=1 sim_type=dc

not working:

name=h3 
descr="netlist simulate wait load" 
tclcommand="

    xschem netlist; 
    set id [simulate]; 

    puts sim_started; #debug

    while {true} {
        if {![info exists execute(cmd,[set id])]} {

            puts sim_ended; #debug

            break;
        }
        puts -nonewline .; flush stdout; #debug

        delay 99;
    }

    if {true} {
        xschem raw_clear;
        xschem raw_read ${netlist_dir}/[file rootname [file tail [xschem get current_name]]].raw;
        xschem annotate_op;
    }

    puts done; #debug
"

->

Simulation started: execution ID: 36
sim_started
.....................sim_ended
free_rawfile(): clearing data
Raw file data read: /ALL/Xschem/IHP-Open-PDK-main/ihp-sg13g2/libs.tech/xschem/sg13g2_tests/simulation/dc_hv_nmos.raw
points=3010, vars=2, datasets=1 sim_type=dc
free_rawfile(): clearing data
free_rawfile(): clearing data
raw_read(): no useful data found
done

it loads the same file but...

[StefanSchippers]

Can you prepare a minimal test case? I have prepared a schematic with the identical issues as yours, a missing symbol, a launcher that fixes the path, descending and returing from a vsource to reload the schematic, double click on the netlist/simulate/wait/load launcher, closing the dialog box. --> everything works.

Can not get it to fail.

[StefanSchippers]

if im correct, then You write about loosed handles or pointers to allocated memory. i was writing about that with every simulation, Xschem allocate additional memory. and that even if i reload the schematic, the memory remains allocated (the last version is better!). i wonder why.

Operating systems are designed to deal with these situations. In many cases if a program frees memory it is not returned to the OS, because it is assumed it will need it again. Subsequent allocations will be faster because the memory pages are already there. No need to search for free memory pages.

Quoting a comment from someone that surely knows things better than me: free often doesn't return memory to the operating system because it assumes that your program will need it again in the future and it's faster to just keep the memory around inside the program. – Art

It is fair to assume that if the OS runs short on memory the freed chunks will be reclaimed from processes.

[olisnr]

would it be good to start the xschem file with #!/usr/local/bin/xschem ?

[olisnr]

i will try tomorrow to make this test-case