Identify a way to fix .epw data in a way BEopt can still load it

[carlosparadis]

Currently proposed methods:

• Identify a way to convert a fixed .csv into epw (see #17 for epw->csv conversion) OR
• Identify a way to correct the .epw file while still keeping it loadable by BEopt.

[carlosparadis]

@kathrynparadis This may help:

• https://unmethours.com/question/14109/use-weather-converter-to-convert-csv-to-epw/
• https://bigladdersoftware.com/projects/elements/

Also for reference:

• https://beopt.nrel.gov/home

[kathrynparadis]

Conclusion

Agreed solution will be to convert EPW file to CSV, edit the CSV, then convert the CSV back to EPW, and use the resulting EPW in BEopt

I ended up using the EnergyPlus Weather Statistics and Conversion tool. I found a way to "Override the Default Type" to convert the CSV to EPW without needing the .def file (I just changed the input to the Energy Plus CSV file):

[carlosparadis]

Follow-Up

According to Riley,

A new error is appearing in the converted file. As such, the issue remains open.

[carlosparadis]

For the record: This issue was resurrected again a few days ago, as we are trying to reach NREL for a solution.

Some reference links on EPW:

• Q&A Forum on .epw creation (https://unmethours.com/question/2393/energy-plus-how-to-create-and-use-epw-and-ddy-files/)

[carlosparadis]

@kathrynparadis @eileenpeppard

The method to generate the TMY

The process for formatting a csv file a TMY3 file according to that PDF seems easy, however, creating is likely impossible for us, or would at least be worth the effort of all we have done up to this point.

• A TMY3 file is basically a .csv file with 2 headers from what I understand in the document.
• Creating a TMY3 file requires several years of data + a list of special variables to use as input to the algorithm that selects the "most typical month" among all years. Here is a list of them, from the document provided by Riley:

And this doesn't even include the math behind the method of each indicator or the algorithm using them to choose.

What I propose

I think this is a lot of time investment for this, and we would be better off trying to exhaust other possibilities first. For instance, I found this tool:

https://beopt.nrel.gov/downloadDView

That serves to visualize both TMY3 and EPW. I think @kathrynparadis could use it, or maybe someone on the architecture end who knows BeOPT more than we do, and compare the modified TMY3 to the working .epw to see if we can diagnose why it doesn't work.

@eileenpeppard What do you think?

[kathrynparadis]

I tested the tool mentioned in the previous post:

https://beopt.nrel.gov/downloadDView

However, I didn't see any differences (aside from the ones we intentionally created) in the two epw files (the original one that works with BEopt, and the one we tinkered with that didn't work in BEopt).

Continuing to look into this issue, and other options while awaiting a response from Scott (email informing him of the problem was sent July 16th).

[carlosparadis]

Here are some stuff I noticed you missed by comparing in a notepad the original .csv with the revised .csv (after you save from opening in excel and making the necessary edits:

1. Date Format Changes

Before	After

2. Unecessary number of commas (columns) are added

Before	After

3. Decimals have 0. added as a prefix and remove 0

(e.g. 24.50 in original becomes 24.5 ; .5 in original becomes 0.5)

Before

After

Notes

There may be more. I recommend changes to the data are made either in Tableau, R, Python or a text editor like Notepad++ on Windows (dont use the default Notepad).

[carlosparadis]

More notes comparing .epw:

Third column from right to left, on the example files provided, the 0.13 value repeated over several rows in the original is replaced by 999.000 in the revised. The same value, slighty different format, from the second column also on right to left on both files (i.e. it feels the second column got copy pasted in the third, both from right to left).

[carlosparadis]

Editing by notepad didnt work even from the original.

Try using this tool to edit the .epw file and see if it works:

https://bigladdersoftware.com/projects/elements/

Dont try to convert to csv ; just edit the epw directly

[kathrynparadis]

I used the Elements program, and here is what I found:

1. I am able to edit up to 4 rows (just the Dry Bulb Temperature, and Wet Bulb temperature columns - relative humidity will correct itself), and it will run in BEopt just fine.
2. Any more than 4 rows, and it gives the error again (I tried copying the data from the previous day for the 10 lines of bad data to no avail, and tried duplicating the 1st line of data on each end of the bad data to meet halfway)

The error we get on the new BEopt runs is a bit different than the old (the number on second hers.py line error changed):

[kathrynparadis]

Hello all,

I have spoken with Riley and Ben, and it seems that the consensus is that ERI is not required for the BEopt simulations.

As the error with the edited Barbers point TMY file was related to the ERI, we have decided to just run the simulations without the ERI option marked.

Should we want to use ERI in the future we can look into this error again, however, since this issue is not a priority at the moment we have decided to just run all the simulations without ERI.

Please let us know if you would like us to use the ERI again.

Thank you,

[kathrynparadis]

Just to summarize this:

We ended up editing the tmy file in notepad++.

At row 4620 (the 1991, 7, 12, 4, 0 row near the middle of the document which represents a July 12th at 4am reading) through row 4629, there were 10 rows (2 columns) of data that had erroneous values of -9.3. The 2 columns in the first 5 rows of bad data were changed to the values of the row directly before the series of bad data (row 4619). The 2 columns in the last 5 rows were replaced with values from the row directly after the bad data (row 4630).

Changing any more than 4 rows while running BEopt with the ERI selected caused it to throw errors and not complete the run, so every file was run without the ERI selected.