C:\Users\JKIM4\Documents\GitHub\edv_synthetic_smart_meter_data>rake format_data['bdgp']
C:/Ruby25-x64/bin/ruby.exe scripts/format_data_bdgp.rb
###############################################
Verifying latitude and longitude in location 1 out of 2 locations
zipcode: 32806
city: Orlando
state: Florida
###############################################
Verifying latitude and longitude in location 2 out of 2 locations
zipcode: 32806
city: Orlando
state: Florida
###############################################
Adding standard labels (header names) into metadata.csv file
Adding values of standard labels into metadata.csv file
Copying electricity timeseries data into timeseriesdata.csv file
header: ["timestamp", "fuel_type", "Panther_office_Danica", "Panther_office_Christin"]
row[0] = 1/1/2016 0:00
row[0].split(' ') = ["1/1/2016", "0:00"]
row[0].split(' ')[0] = 1/1/2016
split_date = ["1", "1", "2016"]
date = 202016/1/1
time = 0:00
row[0] = 202016/1/1 0:00
SUMMARY
○ issue here is, string splitting scheme is not compatible with the format row[0] grabbed from csv
○ theory: maybe when down-sampled csv file was created (by me) with two buildings, it might have done something weird in the timestamp column.
TEST2: testing with BDGP2 original data (saved in here) & without opening/closing all three (elec, gas, meta) files. outputs when format_data['bdgp'] gets executed:
row[0] = 2016-01-01 00:00:00
row[0].split(' ') = ["2016-01-01", "00:00:00"]
row[0].split(' ')[0] = 2016-01-01
split_date = ["2016-01-01"]
Traceback (most recent call last):
6: from scripts/format_data_bdgp.rb:96:in <main>' 5: from C:/Ruby25-x64/lib/ruby/2.5.0/csv.rb:1289:inopen'
4: from scripts/format_data_bdgp.rb:98:in block in <main>' 3: from scripts/format_data_bdgp.rb:98:ineach_with_index'
2: from scripts/format_data_bdgp.rb:98:in each' 1: from scripts/format_data_bdgp.rb:113:inblock (2 levels) in '
scripts/format_data_bdgp.rb:113:in `+': no implicit conversion of nil into String (TypeError)
Usage: rake format_data[data_option] path/to/metadata path/to/timeseriesdata
SUMMARY
○ looks like row[0] string's date format is different from my down-sample file
○ but even with the original file, splitting with "/" string is not working since numeric values are parsed with "-"
TEST3: opening original files (used in TEST2) and deleting all other buildings besides "Panther_lodging_Hattie". and not touching anything on the timestamp column. outputs when format_data['bdgp'] gets executed:
C:\Users\JKIM4\Documents\GitHub\edv_synthetic_smart_meter_data>rake format_data['bdgp']
C:/Ruby25-x64/bin/ruby.exe scripts/format_data_bdgp.rb
###############################################
Verifying latitude and longitude in location 1 out of 1 locations
zipcode: 32806
city: Orlando
state: Florida
###############################################
Adding standard labels (header names) into metadata.csv file
Adding values of standard labels into metadata.csv file
Copying electricity timeseries data into timeseriesdata.csv file
header: ["timestamp", "fuel_type", "Panther_lodging_Hattie"]
row[0] = 1/1/2016 0:00
row[0].split(' ') = ["1/1/2016", "0:00"]
row[0].split(' ')[0] = 1/1/2016
split_date = ["1", "1", "2016"]
date = 202016/1/1
time = 0:00
row[0] = 202016/1/1 0:00
SUMMARY
○ confirmed that opening/modifying/closing is making the timestamp format differently.
SUGGESTION
○ not sure this is coming from difference between MAC and Windows
○ but should find solutions for TEST2 case.
○ maybe changing .split('/') (shown below in currrent ruby_upgrade branch) to .split('-') might solve the issue. not testing since out of time.
TEST1: testing with this sample file: ○ electricity_bdgp.csv ○ gas_bdgp.csv ○ metadata_bdgp.csv
outputs when format_data['bdgp'] gets executed:
C:\Users\JKIM4\Documents\GitHub\edv_synthetic_smart_meter_data>rake format_data['bdgp'] C:/Ruby25-x64/bin/ruby.exe scripts/format_data_bdgp.rb ############################################### Verifying latitude and longitude in location 1 out of 2 locations zipcode: 32806 city: Orlando state: Florida ############################################### Verifying latitude and longitude in location 2 out of 2 locations zipcode: 32806 city: Orlando state: Florida ############################################### Adding standard labels (header names) into metadata.csv file Adding values of standard labels into metadata.csv file Copying electricity timeseries data into timeseriesdata.csv file header: ["timestamp", "fuel_type", "Panther_office_Danica", "Panther_office_Christin"] row[0] = 1/1/2016 0:00 row[0].split(' ') = ["1/1/2016", "0:00"] row[0].split(' ')[0] = 1/1/2016 split_date = ["1", "1", "2016"] date = 202016/1/1 time = 0:00 row[0] = 202016/1/1 0:00
row[0] = 2016-01-01 00:00:00 row[0].split(' ') = ["2016-01-01", "00:00:00"] row[0].split(' ')[0] = 2016-01-01 split_date = ["2016-01-01"] Traceback (most recent call last): 6: from scripts/format_data_bdgp.rb:96:in'
scripts/format_data_bdgp.rb:113:in `+': no implicit conversion of nil into String (TypeError)
Usage: rake format_data[data_option] path/to/metadata path/to/timeseriesdata
<main>' 5: from C:/Ruby25-x64/lib/ruby/2.5.0/csv.rb:1289:inopen' 4: from scripts/format_data_bdgp.rb:98:inblock in <main>' 3: from scripts/format_data_bdgp.rb:98:ineach_with_index' 2: from scripts/format_data_bdgp.rb:98:ineach' 1: from scripts/format_data_bdgp.rb:113:inblock (2 levels) inC:\Users\JKIM4\Documents\GitHub\edv_synthetic_smart_meter_data>rake format_data['bdgp'] C:/Ruby25-x64/bin/ruby.exe scripts/format_data_bdgp.rb ############################################### Verifying latitude and longitude in location 1 out of 1 locations zipcode: 32806 city: Orlando state: Florida ############################################### Adding standard labels (header names) into metadata.csv file Adding values of standard labels into metadata.csv file Copying electricity timeseries data into timeseriesdata.csv file header: ["timestamp", "fuel_type", "Panther_lodging_Hattie"] row[0] = 1/1/2016 0:00 row[0].split(' ') = ["1/1/2016", "0:00"] row[0].split(' ')[0] = 1/1/2016 split_date = ["1", "1", "2016"] date = 202016/1/1 time = 0:00 row[0] = 202016/1/1 0:00
SUMMARY ○ confirmed that opening/modifying/closing is making the timestamp format differently.
SUGGESTION ○ not sure this is coming from difference between MAC and Windows ○ but should find solutions for TEST2 case. ○ maybe changing .split('/') (shown below in currrent ruby_upgrade branch) to .split('-') might solve the issue. not testing since out of time.