Open arriflex opened 6 years ago
sankerust
commented
6 years ago @arriflex Hello! Thank you for the suggestion!
Could you please specify what do you mean,
Do you have the values set to NULL in the table?
Or the table cells are empty and their value is considered as NULL?
arriflex
commented
6 years ago Good question! I should have specified. The csv I am being provided has no value between the commas rather than any value (including NULL) or zeroes. It has other issues as well, so I'm already having to massage it a fair amount. I've put the first raw 24 hours into a code snippet below for reference. Note also that there's also an extra line between each legitimate line.
Very unique to this csv is that it represents a year of hourly data from historically appropriate months so January might be from 1985 whereas March might be from 2003. Because I'm using this data to compare performance I use a regex find replace all the years with the current one. This isn't really something I would expect csv2influx to handle for me although it would be cool if it did.
Also, it looks like there's a potential feature request elsewhere for timezone support so I'll just mention here that I'd love the ability to set the timezone in the config file because this csv is in a local tz, but the timestamp isn't labeled as such.
` hour_index,timestamp,global_horizontal_irradiance,direct_normal_irradiance,diffuse_horizontal_irradiance,dry_bulb_temperature,windspeed,albedo,solar_altitude_angle,solar_azimuth_angle,solar_incident_angle,horizon_elevation_angle,adjusted_ghi,poa_direct_irradiance,poa_diffuse_irradiance,poa_reflected_irradiance,shaded_direct_irradiance,shaded_diffuse_irradiance,shaded_reflected_irradiance,effective_direct_irradiance,effective_diffuse_irradiance,effective_reflected_irradiance,soiled_irradiance,total_irradiance,nameplate_power,avg_cell_temp,module_irradiance_derated_power,module_mpp_power,module_power,optimizer_input_power,optimizer_output_power,optimal_dc_power,optimal_dc_voltage,inverter_overpower_loss,inverter_underpower_loss,inverter_overvoltage_loss,inverter_undervoltage_loss,actual_dc_power,actual_dc_voltage,ac_power,grid_power
1,1991-01-01 00:00:00,0.0,0.0,0.0,1.1,0.0,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
2,1991-01-01 01:00:00,0.0,0.0,0.0,2.8,2.6,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
3,1991-01-01 02:00:00,0.0,0.0,0.0,2.2,2.6,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
4,1991-01-01 03:00:00,0.0,0.0,0.0,1.1,2.1,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
5,1991-01-01 04:00:00,0.0,0.0,0.0,0.0,3.1,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
6,1991-01-01 05:00:00,0.0,0.0,0.0,0.6,3.1,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
7,1991-01-01 06:00:00,0.0,0.0,0.0,-0.6,3.1,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
8,1991-01-01 07:00:00,28.0,10.0,27.0,0.0,0.0,0.19,3.51481,121.626,81.3,0.0,28.0,1.73864,26.5608,0.0425383,1.46448,26.4061,0.0425383,1.05361,25.2367,0.0135541,25.7778,25.7742,15072.0,0.81031,13027.5,14658.2,14192.4,0.0,0.0,14191.4,583.929,0.0,0.0,0.0,0.0,14191.4,583.929,13143.2,13142.3095703125
9,1991-01-01 08:00:00,153.0,112.0,127.0,3.3,2.1,0.19,13.3392,131.431,70.183,0.0,153.0,38.2048,139.307,0.232441,38.2048,138.495,0.232441,34.4804,132.362,0.0740635,163.578,163.572,95653.0,7.76465,93967.4,101795.0,97948.4,0.0,0.0,97905.2,609.16,0.0,0.0,0.0,0.0,97905.2,609.16,95125.1,95076.609375
10,1991-01-01 09:00:00,245.0,197.0,173.0,5.0,0.0,0.19,21.6531,142.991,60.5107,0.0,245.0,96.0543,185.184,0.37221,96.0543,184.105,0.37221,91.1005,175.952,0.118598,261.828,261.819,153104.0,13.2313,152874.0,161382.0,155995.0,0.0,0.0,155883.0,602.776,0.0,0.0,0.0,0.0,155883.0,602.776,152424.0,152299.734375
11,1991-01-01 10:00:00,300.0,177.0,217.0,7.2,2.6,0.19,27.8205,156.596,53.0904,0.0,300.0,106.805,231.528,0.455767,106.805,230.179,0.455767,103.253,219.986,0.145222,316.916,316.904,185314.0,15.5664,185934.0,194162.0,187695.0,0.0,0.0,187531.0,598.911,0.0,0.0,0.0,0.0,187531.0,598.911,183590.0,183409.265625
12,1991-01-01 11:00:00,462.0,466.0,221.0,10.0,2.1,0.19,31.1452,171.994,48.9649,0.0,462.0,305.914,246.128,0.701882,305.914,244.695,0.701882,297.912,233.859,0.223643,521.355,521.342,304868.0,24.2298,308051.0,309098.0,298573.0,0.0,0.0,298132.0,581.303,0.0,0.0,0.0,0.0,298132.0,581.303,292580.0,292120.71875
13,1991-01-01 12:00:00,465.0,464.0,226.0,11.7,2.1,0.19,31.1354,188.13,48.9781,0.0,465.0,303.381,251.4,0.706439,303.381,249.936,0.706439,295.439,238.868,0.225095,523.841,523.829,306318.0,25.9977,309525.0,308044.0,298668.0,0.0,0.0,298218.0,575.099,0.0,0.0,0.0,0.0,298218.0,575.099,292624.0,292164.65625
14,1991-01-01 13:00:00,400.0,446.0,192.0,12.8,4.1,0.19,27.7927,203.521,53.127,0.0,400.0,267.673,214.346,0.60769,267.673,213.097,0.60769,258.752,203.661,0.19363,453.354,453.344,265102.0,23.6397,267587.0,269252.0,260554.0,0.0,0.0,260220.0,582.146,0.0,0.0,0.0,0.0,260220.0,582.146,255257.0,254908.4375
15,1991-01-01 14:00:00,263.0,264.0,166.0,12.2,4.1,0.19,21.611,217.116,60.5645,0.0,263.0,129.431,181.628,0.399556,129.431,180.57,0.399556,122.734,172.574,0.127312,289.527,289.517,169300.0,19.1225,169501.0,174193.0,168375.0,0.0,0.0,168239.0,589.816,0.0,0.0,0.0,0.0,168239.0,589.816,164542.0,164396.484375
16,1991-01-01 15:00:00,132.0,208.0,84.0,11.7,3.1,0.19,13.2866,228.667,70.2484,0.0,132.0,70.5819,92.9706,0.200538,70.5819,92.4291,0.200538,63.6681,88.3361,0.0638979,149.027,149.022,87143.3,15.5059,85272.4,89212.2,86292.0,0.0,0.0,86255.9,588.028,0.0,0.0,0.0,0.0,86255.9,588.028,83284.0,83246.8125
17,1991-01-01 16:00:00,28.0,15.0,27.0,10.0,2.6,0.19,3.455,238.467,81.3731,0.0,28.0,1.72413,26.627,0.0425383,1.45242,26.4719,0.0425383,1.0409,25.2996,0.0135541,25.827,25.8236,15101.0,10.6818,13055.3,14027.5,13582.0,0.0,0.0,13581.0,557.457,0.0,0.0,0.0,0.0,13581.0,557.457,12547.2,12546.3203125
18,1991-01-01 17:00:00,0.0,0.0,0.0,6.7,2.1,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
19,1991-01-01 18:00:00,0.0,0.0,0.0,4.4,2.6,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
20,1991-01-01 19:00:00,0.0,0.0,0.0,3.9,2.6,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
21,1991-01-01 20:00:00,0.0,0.0,0.0,2.8,3.1,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
22,1991-01-01 21:00:00,0.0,0.0,0.0,2.2,2.6,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
23,1991-01-01 22:00:00,0.0,0.0,0.0,3.9,2.6,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
24,1991-01-01 23:00:00,0.0,0.0,0.0,2.8,3.1,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
25,1991-01-02 00:00:00,0.0,0.0,0.0,1.1,0.0,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
`
It would be pretty cool if there were an option to replace NULL values with a predefined one such as 0 in the config file. One of the csv files I have to deal with is full of nulls instead of zeros and I have no control over the generator.