Qucs / qucs-test

Qucs Project test suite and scripts repository
5 stars 14 forks source link

Added qucs print report generator, with comparison output #7

Closed in3otd closed 9 years ago

in3otd commented 9 years ago

I have expanded the print function to generate a small report, similar to the one used for qucsator, in order to have a summary of the results and check easily if some of the print failed. Similarly, two qucs versions can be compared for the print output. At present there is no reference result for printing, but only the comparison table is generated. Not sure how we could compare the print output, as we should probably allow for some "fuzzy factor". I have also fixed a couple of small things around.

The comparison report for print looks like this:

*****************************
* Qucs printing test report *
*****************************
---------------------------------------------------------------------------------------------
Project                        | Schem. Version   |   Print Runtime     |   Print Runtime    
---------------------------------------------------------------------------------------------
AC_SW_resonance_prj            |                  |                     |                     
  resonance.sch                | 0.0.4            | 0.216293            | 0.164690            
AC_SW_swr_meter_prj            |                  |                     |                     
  swr_meter.sch                | 0.0.4            | 0.213962            | 0.165291            
AC_bandpass_prj                |                  |                     |                     
  bandpass.sch                 | 0.0.17           | FAIL (-11)          | FAIL (-11)          
  opa227.sch                   | 0.0.17           | 0.114141            | 0.113669            
  bandpass.dpl                 | 0.0.17           | 0.113685            | 0.113746            
AC_groupdelay_ac_prj           |                  |                     |                     
  groupdelay_ac.sch            | 0.0.12           | 0.165728            | 0.164440            
DC_AC_LM358_spice_prj          |                  |                     |                     
  LM358_spice.sch              | 0.0.17           | 0.215737            | 0.164068            
DC_AC_SP_stab_prj              |                  |                     |                     
  stab.sch                     | 0.0.4            | 0.215052            | 0.164092            
DC_AC_active_bp_prj            |                  |                     |                     
  active_bp.sch                | 0.0.5            | 0.214313            | 0.164478            
DC_AC_active_lp_prj            |                  |                     |                     
  active_lp.sch                | 0.0.5            | 0.215293            | 0.164474            
DC_AC_bbv_prj                  |                  |                     |                     
  bbv.sch                      | 0.0.5            | 0.113781            | 0.113616            
DC_AC_gain_prj                 |                  |                     |                     
  gain.sch                     | 0.0.5            | FAIL (-11)          | FAIL (-11)          
  singleOPV.sch                | 0.0.5            | 0.113681            | 0.113683            
DC_AC_gyrator_prj              |                  |                     |                     
  gyrator.sch                  | 0.0.4            | 0.115023            | 0.113613            
DC_AC_notch_prj                |                  |                     |                     
  notch.sch                    | 0.0.5            | 0.114538            | 0.113627            
DC_AC_selective_amp_prj        |                  |                     |                     
  selective_amp.sch            | 0.0.5            | 0.113716            | 0.113720            
DC_SP_HICUM-fig10_prj          |                  |                     |                     
  HICUM-fig10.sch              | 0.0.11           | 0.113677            | 0.163706            
DC_SP_LPF-Balun2_prj           |                  |                     |                     
  IdealBalun.sch               | 0.0.15           | 0.113839            | 0.113671            
  LPF-Balun2.sch               | 0.0.15           | FAIL (-11)          | FAIL (-11)          
DC_SP_LPF-Balun3_prj           |                  |                     |                     
  LPF-Balun3.sch               | 0.0.15           | 0.113920            | 0.113636            
DC_SP_opamp_gyrator_prj        |                  |                     |                     
  opamp_gyrator.sch            | 0.0.5            | 0.113689            | 0.113649            
DC_SW_bridge_prj               |                  |                     |                     
  bridge.sch                   | 0.0.3            | 0.114054            | 0.113627            
DC_SW_bsim4v30nMOS_Ids_Vgs_prj |                  |                     |                     
  bsim4v30nMOS_Ids_Vgs.sch     | 0.0.18           | 0.163736            | 0.164028            
DC_SW_bsim4v30pMOS_Ids_Vgs_prj |                  |                     |                     
  bsim4v30pMOS_Ids_Vgs.sch     | 0.0.18           | 0.163739            | 0.163938            
DC_SW_charac_prj               |                  |                     |                     
  charac.sch                   | 0.0.4            | 0.113873            | 0.113716            
DC_SW_curtice_1_tb1_prj        |                  |                     |                     
  curtice_1_tb1.sch            | 0.0.18           | FAIL (-11)          | FAIL (-11)          
DC_SW_diff1_prj                |                  |                     |                     
  diff1.sch                    | 0.0.5            | 0.113760            | 0.113660            
DC_SW_diode_prj                |                  |                     |                     
  diode.sch                    | 0.0.18           | 0.165447            | 0.163946            
DC_SW_fgummel_prj              |                  |                     |                     
  fgummel.sch                  | 0.0.10           | 0.113723            | 0.113653            
DC_SW_preregulator_prj         |                  |                     |                     
  preregulator.sch             | 0.0.15           | 0.113710            | 0.113663            
DC_SW_rgummel_prj              |                  |                     |                     
  rgummel.sch                  | 0.0.10           | 0.113635            | 0.113772            
DC_TR_LM358_spice_prj          |                  |                     |                     
  LM358_spice.sch              | 0.0.17           | 0.113698            | 0.113687            
DC_TR_SW_TestHBProbe_prj       |                  |                     |                     
  TestHBProbe.sch              | 0.0.18           | 0.113705            | 0.113965            
DC_TR_SW_spice_BFR520_prj      |                  |                     |                     
  BFR520_sub.sch               | 0.0.17           | 0.113797            | 0.113722            
  spice_BFR520.sch             | 0.0.18           | FAIL (-11)          | FAIL (-11)          
DC_TR_active_mixer_prj         |                  |                     |                     
  active_mixer.sch             | 0.0.4            | 0.114606            | 0.114044            
SP_SmithChartTest_prj          |                  |                     |                     
  SmithChartTest.sch           | 0.0.12           | 0.113842            | 0.113734            
SP_Sparam_diagrams_prj         |                  |                     |                     
  Sparam_diagrams.sch          | 0.0.18           | FAIL (-6)           | 0.164026            
SP_bpf_10Ghz_prj               |                  |                     |                     
  bpf_10Ghz.sch                | 0.0.4            | 0.114995            | 0.113786            
SP_chebyshev1_5th_prj          |                  |                     |                     
  chebyshev1_5th.sch           | 0.0.3            | 0.114964            | 0.121085            
SP_elliptic_5th_prj            |                  |                     |                     
  elliptic_5th.sch             | 0.0.3            | 0.114908            | 0.116207            
SP_fet_noise_prj               |                  |                     |                     
  fet.sch                      | 0.0.5            | 0.113714            | 0.113770            
  fet_noise.sch                | 0.0.5            | FAIL (-11)          | FAIL (-11)          
SP_fhr01fh_prj                 |                  |                     |                     
  fhr01fh.sch                  | 0.0.3            | 0.113657            | 0.115050            
SP_giacoletto_prj              |                  |                     |                     
  giacoletto.sch               | 0.0.3            | 0.113726            | 0.114350            
SP_groupdelay_sp_prj           |                  |                     |                     
  groupdelay_sp.sch            | 0.0.12           | 0.113748            | 0.114636            
SP_microstrip_prj              |                  |                     |                     
  microstrip.sch               | 0.0.4            | 0.114486            | 0.113719            
SP_mscoupler_prj               |                  |                     |                     
  mscoupler.sch                | 0.0.7            | 0.113625            | 0.114647            
SP_sparam_prj                  |                  |                     |                     
  sparam.sch                   | 0.0.17           | FAIL (-6)           | 0.113730            
SP_wilkinson_prj               |                  |                     |                     
  wilkinson.sch                | 0.0.5            | 0.113731            | 0.114660            
TR_555_Fig7_prj                |                  |                     |                     
  555_Fig7.sch                 | 0.0.17           | FAIL (-11)          | FAIL (-11)          
  timer_555.sch                | 0.0.17           | FAIL (-11)          | FAIL (-11)          
  timer_Discharge.sch          | 0.0.17           | 0.113693            | 0.113772            
  timer_amp.sch                | 0.0.17           | 0.113866            | 0.113792            
  timer_digital_comb.sch       | 0.0.17           | 0.114453            | 0.114752            
  timer_thresh.sch             | 0.0.17           | 0.064304            | 0.063676            
  timer_trig.sch               | 0.0.17           | 0.113798            | 0.064353            
  555_Fig7.dpl                 | 0.0.17           | 0.063717            | 0.063740            
TR_555_Fig9_prj                |                  |                     |                     
  555_Fig9.sch                 | 0.0.17           | FAIL (-11)          | FAIL (-11)          
  timer_555.sch                | 0.0.17           | FAIL (-11)          | FAIL (-11)          
  timer_Discharge.sch          | 0.0.17           | 0.113738            | 0.115738            
  timer_amp.sch                | 0.0.17           | 0.115106            | 0.113687            
  timer_digital_comb.sch       | 0.0.17           | 0.113700            | 0.113741            
  timer_thresh.sch             | 0.0.17           | 0.113728            | 0.063718            
  timer_trig.sch               | 0.0.17           | 0.063707            | 0.063702            
TR_Puls3b_prj                  |                  |                     |                     
  Puls3b.sch                   | 0.0.17           | 0.114708            | 0.114399            
TR_boostconverter_prj          |                  |                     |                     
  boostconverter.sch           | 0.0.15           | 0.113649            | 0.113696            
TR_buckboost_prj               |                  |                     |                     
  buckboost.sch                | 0.0.15           | 0.113570            | 0.113769            
TR_buckconverter_prj           |                  |                     |                     
  buckconverter.sch            | 0.0.15           | 0.113858            | 0.113658            
TR_chargepump_prj              |                  |                     |                     
  chargepump.sch               | 0.0.5            | 0.113694            | 0.114405            
TR_classic_osci_prj            |                  |                     |                     
  classic_osci.sch             | 0.0.4            | 0.113704            | 0.113786            
TR_colpitts_base_prj           |                  |                     |                     
  colpitts_base.sch            | 0.0.5            | 0.113730            | 0.114393            
TR_colpitts_prj                |                  |                     |                     
  colpitts.sch                 | 0.0.4            | 0.113705            | 0.114057            
TR_diode_hb_prj                |                  |                     |                     
  diode_hb.sch                 | 0.0.9            | 0.113827            | 0.113792            
TR_fullwaverectifier_1_prj     |                  |                     |                     
  fullwaverectifier_1.sch      | 0.0.15           | 0.113704            | 0.113701            
TR_fullwaverectifier_2_prj     |                  |                     |                     
  fullwaverectifier_2.sch      | 0.0.15           | 0.115469            | 0.115593            
TR_gilbert_prj                 |                  |                     |                     
  gilbert.sch                  | 0.0.4            | 0.113721            | 0.113780            
TR_lc_osc_prj                  |                  |                     |                     
  lc_osc.sch                   | 0.0.12           | 0.114060            | 0.113789            
TR_lf_osci_prj                 |                  |                     |                     
  lf_osci.sch                  | 0.0.4            | 0.163798            | 0.113619            
TR_mixer_prj                   |                  |                     |                     
  mixer.sch                    | 0.0.4            | 0.114487            | 0.114094            
TR_multiplier_prj              |                  |                     |                     
  multiplier.sch               | 0.0.4            | 0.113682            | 0.113708            
TR_resistor_prj                |                  |                     |                     
  resistor.sch                 | 0.0.18           | 0.113733            | 0.113698            
TR_rf_osci_prj                 |                  |                     |                     
  rf_osci.sch                  | 0.0.4            | 0.113665            | 0.114854            
TR_sawtooth-1_prj              |                  |                     |                     
  sawtooth-1.sch               | 0.0.10           | 0.113874            | 0.114424            
TR_sawtooth-2_prj              |                  |                     |                     
  sawtooth-2.sch               | 0.0.10           | 0.113684            | 0.113768            
TR_sawtooth-3_prj              |                  |                     |                     
  sawtooth-3.sch               | 0.0.17           | 0.113944            | 0.113764            
TR_sawtooth-discreet_prj       |                  |                     |                     
  sawtooth-discreet.sch        | 0.0.17           | 0.163789            | 0.113799            
TR_schmitt_prj                 |                  |                     |                     
  schmitt.sch                  | 0.0.5            | 0.113740            | 0.114072            
TR_single_balanced_prj         |                  |                     |                     
  single_balanced.sch          | 0.0.12           | 0.113740            | 0.115923            
TR_supply_prj                  |                  |                     |                     
  supply.sch                   | 0.0.5            | 0.113737            | 0.115015            
TR_sym_osci_prj                |                  |                     |                     
  sym_osci.sch                 | 0.0.4            | 0.120292            | 0.113762            
---------------------------------------------------------------------------------------------
Qucs versions:   Qucs 0.0.19 (8055078) : Qucs 0.0.19 (a16372c) : 

Binary Locations:
/usr/bin
/home/user/qucs-git/qucs/qucs

Report produced on: 2015-07-05 22:06:14

---------------------------------------------------------------------------------------------
Saved simulation report: qucs_comparison_150705_220614_print_results.txt 

the comparison is between the current qucs and the one before https://github.com/felix-salfelder/qucs/commit/dbd3376652d815bea86bb0a142f8012386d6db7f and you can see the failing S-parameters diagram printing there.

The overall code is not really nice but seems to work. It seems I didn't break anything in the existing code, but please double check.

felix-salfelder commented 9 years ago

nice.

would it be possible to compare the print results against a reference? i suspect a glitch in axis ranges in dbS11 of sparam.sch (still in dbd3376). it would be of great advantage to catch things like that immediately...

guitorri commented 9 years ago

This looks good! Allow me some more time to test it. Some of the print failures are crashes in qucs...

I will try to get a better backtrace than this:

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   org.qucs                        0x000000010ad14047 QHash<QString, QString>::value(QString const&) const + 23
1   org.qucs                        0x000000010ad13c8c 0x10ac0b000 + 1084556

Some remarks:

If we default to .png (or allow to pass a suffix as parameter) we could use some sort of visual regression test against reference images. We could use .pdf as well, but it might need to be rasterized to be able to get pixel differences.

Something like this: http://twolfson.com/2014-02-25-visual-regression-testing-in-travis-ci http://www.imagemagick.org/script/compare.php http://tex.stackexchange.com/questions/36351/comparing-the-output-of-two-pdfs

I don't know what changed since then with the qucs --print but I recall (a9dd434) an issue that older schematics would not print. Doing a load-print was giving different results than load-save-print.

in3otd commented 9 years ago

I noticed some trailing white-space in run.py

oops! forgot to clean that up...

It does not abort (ctrl+c) after script is fired up?

yes, it does not. AFAIU, I think it's due to the fact that it always uses multiprocessing (even for a single process), so you have to manually kill the main process to terminate all the threads, at present. I saw there are several proposed solutions around, but apparently no one really working in all cases, need to try some and see.

I didn't know about the visual regression testing with Travis, looks quite interesting. It will make sense to move to .png anyway, I wanted to add a command-line option to select the output format... later.

As you saw, many of the schematics in skip-print.txt can actually be printed now, but I didn't check if the output is correct, though. A question I had since some time is whether the visual appearance of the schematics is actually the same on all platforms; some of the test schematics have components (simulations, equations, text) which partly superposes, as if they were smaller when the schematics was created.

guitorri commented 9 years ago

Printing (this PR) or netlisting raises the gives crash reported by @ra3xdh in https://github.com/Qucs/qucs/issues/333.

guitorri commented 9 years ago

The crashes should be fixed with https://github.com/Qucs/qucs/pull/351

Adding the file format options sound like a nice idea. I was thinking about using subparser with argparse to improve the CLI commands.

The schematic and the printed version are slightly different... not sure why.