Add TMY3 Data to our climate data store.

[dpinney]

TODO

• Run the TMY3 to TMY2 converter on the TMY3 data, and add it to the store. We should make sure that we have a naming convention that makes it clear what data is 2 versus 3. Maybe "DC-Washington-v3.tmy2" versus "CA-Los Angeles-v2.tmy2".
• Write a short Python function that will graph the data in a TMY2 file. This is to help us debug.

[cshjin]

A new beta release of SAM(2014-09-30) already support TMY3 data.

[dpinney]

That's great.

But I think we we should still store everything as tmy2. Since Gridlab-D doesn't support tmy3.

[jcfuller1]

We could look at updating to TMY3. The format is pretty straightforward, so I think the upgrade would be relatively simple.

[dpinney]

Let's try the TMY3->2 converter first then re-asses whether GLD support for TMY3 is valuable.

[cshjin]

A quick shot here: run TMY3ToTMY2_CMD tool, (write it to python)

C:\Users\User\Desktop>TMY3toTMY2_CMD.exe 725340TY.csv > 725340TY.tmy2

It will convert to TMY2 file, and then run it in gridlabd __init__.py, it may get the following error:

GridlabD standard error: ERROR    [INIT] : climate::init() -- weather file header improperly formed
ERROR    [INIT] : init_by_deferral(): object climate initialization failed
ERROR    [INIT] : model initialization failed
FATAL    [INIT] : shutdown after simulation stopped prematurely
FATAL    [INIT] : environment startup failed: Invalid argument
GridlabD standard output: dump to 'gridlabd.xml' complete
Model profiler results

Header in my TMY2 file is

725340 CHICAGO MIDWAY AP      IL  -6 N 41 47 W  87 45   186

And the problem is the name of the station, which is NOT allowed for any spaces between "CHICAGO MIDWAY AP" Rename it to string without spaces could pass gridlabD test.

Besides, new TMY2 works fine in SAM. Ref: header format of TMY2: http://rredc.nrel.gov/solar/pubs/tmy2/tab3-1.html, latest version of GridlabD climate.cpp: http://www.gridlabd.org/documents/doxygen/latest_dev/climate_8cpp-source.html

[dpinney]

Nice job catching that bug with the space.

Should be easy to fix with a little python:

import os, csv, re, shutil

pathToExe ='"C:\\Program Files (x86)\\TMY3toTMY2_CMD\\TMY3toTMY2_CMD.exe"'
testFolder = "./tmy3 full dataset/"
os.chdir(testFolder)

def renameTmy2s():
    allNames = [x for x in os.listdir('.') if x.endswith('.txt')]
    for fName in allNames:
        with open(fName,'r') as inFile:
            code, city, state = inFile.readline().split()[0:3]
            print city, state
        shutil.copyfile(fName, state + '-' + city + '.tmy2')

def convertTmy3(csvPath):
    readFile = open(csvPath, "rb")
    reader = csv.reader(readFile)
    firstRow = reader.next()
    origName = str(firstRow[1])
    newName = re.sub(r'\W', '', origName.replace(" ", "_"))
    firstRow[1] = '"' + newName + '"'
    outName = firstRow[2] + "-" + newName + "-V3.tmy2.txt"
    writeFile = open(outName, "wb")
    writeFile.write(",".join(firstRow))     
    writer = csv.writer(writeFile, quotechar="'")
    for row in reader:
        writer.writerow(row)
    # Now convert.
    readFile.close()
    writeFile.close()
    os.system(pathToExe + " " + outName + " >" + outName[0:-4])

def convertAll():
    # Now for everything:
    allPaths = [x for x in os.listdir(".") if x.endswith(".csv")]
    for fName in allPaths:
        convertTmy3(fName)

convertAll()

I forgot the TMY3 data set is 1.6 GB or whatever. Probably not the best thing to check in to Github.

[dpinney]

There was a bunch of junk in there besides spaces: slashes in both directions, backticks (huh?), parens, etc. Code updated to remove those too.

[dpinney]

Oh man. TMY3ToTMY2_CMD can't handle underscores. Ironic choice of name!

[dpinney]

Okay, now the code above converts all the TMY3 csv files to TMY2 files (caveat: on my machine).

We should probably hold off on putting them all into production until we have a better location chooser.

[dpinney]

Every TMY3 file converted to TMY2 format:

https://www.dropbox.com/s/xys6yrpbgubo4ls/tmy3%20full%20dataset%20conversion%20to%20tmy2%20format.zip?dl=0

[drdanley]

It is important for that we note whether the data comes from TMY2 or TMY3, even if they are both in the same format. The TMY3 data is from different datasets and may be incomplete or have larger error margins. Marking the data as either TMY2, TMY3 or some other source will allow people to verify for themselves how much they trust the data.

[dpinney]

Good point. They're all labeled e.g DC-Washington-TMY3.tm2.

[dpinney]

Conversion works. Waiting on database upgrade for push to production.