Execute runAllTests.py, debug a little.

[dpinney]

I tried giving it a run last week and it didn't complete even after 1 hour.

Could you figure out if it even still works?

[cshjin]

One quick shot is the time consuming of weather.py, the makeClimateCsv needs to download a whole year data. Is that necessary? Can we change it to a month period, such as "2010-01-01" to "2010-02-01"?

[dpinney]

Yeah a couple months should be fine.

[cshjin]

Another quick shot: the url in the code: address = "http://www.wunderground.com/history/airport/{}/{:d}/{:d}/{:d}/DailyHistory.html?req_city=NA&reqstate=NA&req_statename=NA&format=1".format(airport, year, month, day) seems taking much more time to get response. I did explore weather underground service this morning. And I suggest use its API to retrieve data, for example: "http://api.wunderground.com/api/2f060cf5d6061a63/history_20140701/q/VA/Arlington.json" for city weather data or http://api.wunderground.com/api/2f060cf5d6061a63/history_20140701/q/22203.json for zip code weather data or http://api.wunderground.com/api/2f060cf5d6061a63/history_20140701/q/KDCA.json for airport weather data. It seems it get history data much faster than through url.

And also we can get the geo location by http://api.wunderground.com/api/2f060cf5d6061a63/geolookup/q/KDCA.json

The API is free for developer.

[cshjin]

And output with three month weather data:

C:\Users\jxh10\Documents\GitHub\omf\omf>python runAllTests.py
********** TESTING calibrate.py ************
Beginning to test calibrate.py
Currently working in:  c:\users\jxh10\appdata\local\temp\tmpq5oj_7
********** TESTING feeder.py ************
Parsed tokens into object of type: <type 'dict'>
All the objects after recorder attach:  set([u'node', '', u'transformer', u'tran
sformer_configuration', u'overhead_line', u'line_spacing', u'triplex_meter', u'c
limate', u'fuse', u'underground_line', u'switch', u'triplex_node', u'capacitor',
 u'line_configuration', u'underground_line_conductor', 'recorder', u'overhead_li
ne_conductor'])
Number of objects still embedded: 0
Swing collector: {'property': 'sum(power_in.real),sum(power_in.imag)', 'interval
': '1', 'group': u'class=transformer AND groupid=swingKids', 'file': u'SwingKids
_transformer.csv', 'limit': '0', 'object': 'collector'}
Time-adjusted collector: {'property': 'sum(power_in.real),sum(power_in.imag)', '
interval': '3600', 'group': u'class=transformer AND groupid=swingKids', 'file':
u'SwingKids_transformer.csv', 'limit': '0', 'object': 'collector'}
********** TESTING weather.py ************
Beginning to test weather.py
DCA lat/lon = (38.852222, -77.037778)
Weather downloading to c:\users\jxh10\appdata\local\temp\tmpaskhdv
Peak solar extraction in c:\users\jxh10\appdata\local\temp\tmpaskhdv
Pull weather and solar data together in c:\users\jxh10\appdata\local\temp\tmpask
hdv
Testing the full process together.
Working in c:\users\jxh10\appdata\local\temp\tmpugf7up
Output CSV available at c:\users\jxh10\appdata\local\temp\tmpsyddoh\weatherDCA.c
sv
Cleanup true, deleted c:\users\jxh10\appdata\local\temp\tmpugf7up
********** TESTING cvrStatic.py ************
SENT TO BACKGROUND C:\Users\jxh10\Documents\GitHub\omf\omf\data\Model\admin\Auto
mated staticCVR Testing
StartRTING TO RUN C:\Users\jxh10\Documents\GitHub\omf\omf\data\Model\admin\Autom
ated staticCVR Testing
DONE RUNNING C:\Users\jxh10\Documents\GitHub\omf\omf\data\Model\admin\Automated
staticCVR Testing
********** TESTING gridlabMulti.py ************
SENT TO BACKGROUND C:\Users\jxh10\Documents\GitHub\omf\omf\data\Model\admin\Auto
mated Multiple GridlabD Testing
STARTING TO RUN C:\Users\jxh10\Documents\GitHub\omf\omf\data\Model\admin\Automat
ed Multiple GridlabD Testing
DONE RUNNING C:\Users\jxh10\Documents\GitHub\omf\omf\data\Model\admin\Automated
Multiple GridlabD Testing Simple Market System
********** TESTING pvWatts.py ************
********** TESTING voltageDrop.py ************
********** TESTING __init__.py ************
Full path to Gridlab executable we're using: gridlabd
Testing string cleaning.
Testing GridlabD solver.
gridlabD runInFilesystem with no specified workDir. Working in c:\users\jxh10\ap
pdata\local\temp\tmpwok9zb
Processing 2000-01-09 08:51:00 PST...000-01-01 02:27:00 PST...
GridlabD standard output: Core profiler results
======================

Total objects                 31 objects
Parallelism                    1 thread
Total time                  28.0 seconds
  Core time                  0.8 seconds (2.8%)
    Compiler                34.2 seconds (122.1%)
    Instances                0.0 seconds (0.0%)
    Random variables         0.0 seconds (0.0%)
    Schedules                0.0 seconds (0.1%)
    Loadshapes               0.0 seconds (0.0%)
    Enduses                  0.0 seconds (0.0%)
    Transforms               0.0 seconds (0.0%)
  Model time                27.2 seconds/thread (97.2%)
Simulation time                9 days
Simulation speed             239 object.hours/second
Passes completed            4570 passes
Time steps completed        4570 timesteps
Convergence efficiency      1.00 passes/timestep
Read lock contention        0.0%
Write lock contention       0.0%
Average timestep            170 seconds/timestep
Simulation rate           27771 x realtime

Model profiler results
======================

Class            Time (s) Time (%) msec/obj
---------------- -------- -------- --------
node              23.206     85.3%  11603.0
climate            3.769     13.8%   3769.0
house              0.108      0.4%     54.0
triplex_meter      0.039      0.1%     13.0
ZIPload            0.020      0.1%      2.5
transformer        0.019      0.1%      9.5
waterheater        0.018      0.1%      9.0
triplex_node       0.015      0.1%     15.0
regulator          0.012      0.0%     12.0
triplex_line       0.008      0.0%      4.0
================ ======== ======== ========
Total             27.214    100.0%    877.9

+------------------------+

TEST RESULTS

+------------------------+
1 tests failed:

C:\Users\jxh10\Documents\GitHub\omf\omf\calibrate.py
Traceback (most recent call last):
  File "calibrate.py", line 122, in <module>
    _tests()
  File "calibrate.py", line 119, in _tests
    assert None == omfCalibrate(workDir, feederPath, scadaPath), "feeder calibra
tion failed"
  File "calibrate.py", line 13, in omfCalibrate
    scadaSubPower = _processScadaData(workDir,scadaPath)
  File "calibrate.py", line 99, in _processScadaData
    with open(scadaPath,"r") as scadaFile:
IOError: [Errno 2] No such file or directory: 'uploads\\colScada.tsv'

Check code in calibrate.py, please sync colScada.tsv to uploads folder.

[cshjin]

13b2efa4091208e9ab50d003db0a9323aa073a64 uploads colScada.tsv to uploads folder. And test succeeded on calibrate.py.

[dpinney]

With 3 months of weather data what was the running time?

[cshjin]

About half an hour.

[dpinney]

Not too bad. We should probably add milToGridlab.py back to the tests. If NRECA IT comes back with continuous integration I figure 30 min + 30 min (milToGridlab) is an acceptable daily test volume.

[srivats0603]

just to add here... @jinhw1989 had problem in calibrate.py due to a nomenclature mismatch.

The tests in the calibrate.py are running the Frank feeder and the scada file it searches for in the upload folder is called colScada.tsv. (I did not change it when i was writing the tests) Frank feeder should point to FrankScada.tsv and columbia feeder should point to colScada.tsv.

[dpinney]

It's better for the tests to be correct than for them to be complete.

History: http://en.wikipedia.org/wiki/Worse_is_better

I fixed the scada reference in ed46c662715f1f164ea8d7828e8032005c1a546b

[srivats0603]

Thanks, nice share on wiki!