search

WikiWatershed / model-my-watershed

The web application front end for Model My Watershed.
https://modelmywatershed.org
Apache License 2.0
57 stars 31 forks source link

Change filenames for CSVs downloaded from Analyze UI #2575

Closed rajadain closed 6 years ago

rajadain commented 6 years ago

Currently the CSVs for Analyze tabs do not match those downloaded from the UI. This is explained in the commit message here: https://github.com/WikiWatershed/model-my-watershed/commit/7b2b1125d158f7287c81ac6e497c5141f7aa1052

Currently, the CSV Download in the UI is handled via the tableExport library which converts a rendered HTML table to CSV and downloads the file. Since when we're exporting from the MultiShareModal we will not have the renderings at hand to reference, we must work off the internal data each Analyze Task has.

~Make it so that the CSV exported to HydroShare matches the formatting of the downloaded one.~ See https://github.com/WikiWatershed/model-my-watershed/issues/2575#issuecomment-367437360

ajrobbins commented 6 years ago

~Stroud would like the HS-exported CSVs to exactly match the ones downloaded from the UI, in schema and filename.~

See https://github.com/WikiWatershed/model-my-watershed/issues/2575#issuecomment-367437360

kellyi commented 6 years ago

For reference here are the downloadable values for this AOI:

screen shot 2018-02-20 at 2 55 54 pm

Streams ->

streams2018-02-20T19_55_22.470Z.csv

"Stream Order","Total Length (km)","Mean Channel Slope (%)"
1st,26.96,0.68%
2nd,24.06,0.10%
3rd,3.70,0.42%
4th,4.61,0.01%
5th,0.00,"No Data"
6th,45.06,0.01%
7th,0.00,"No Data"
8th,0.00,"No Data"
9th,0.00,"No Data"
10th,0.00,"No Data"
Other,7.51,"No Data"
Combined,111.90,0.20%

Land ->

nlcd_land_cover_2018-02-20T19_58_37.199Z.csv

Type,"NLCD Code","Area (km2)","Coverage (%)"
"Open Water",11,18.53,8.0
"Perennial Ice/Snow",12,0.00,0.0
"Developed, Open Space",21,28.25,12.2
"Developed, Low Intensity",22,30.03,13.0
"Developed, Medium Intensity",23,69.17,29.9
"Developed, High Intensity",24,69.68,30.1
"Barren Land (Rock/Sand/Clay)",31,0.12,0.1
"Deciduous Forest",41,7.86,3.4
"Evergreen Forest",42,0.06,0.0
"Mixed Forest",43,0.33,0.1
Shrub/Scrub,52,0.70,0.3
Grassland/Herbaceous,71,0.65,0.3
Pasture/Hay,81,0.54,0.2
"Cultivated Crops",82,0.16,0.1
"Woody Wetlands",90,2.90,1.3
"Emergent Herbaceous Wetlands",95,2.25,1.0

Soil ->

nlcd_soils_2018-02-20T19_59_50.340Z.csv

Type,"Area (km2)","Coverage (%)"
"A - High Infiltration",1.45,0.6
"A/D - High/Very Slow Infiltration",0.00,0.0
"B - Moderate Infiltration",8.72,3.8
"B/D - Medium/Very Slow Infiltration",1.87,0.8
"C - Slow Infiltration",213.72,92.4
"C/D - Medium/Very Slow Infiltration",1.52,0.7
"D - Very Slow Infiltration",3.94,1.7

Terrain ->

terrain2018-02-20T20_00_23.442Z.csv

,"Elevation (m)","Slope (%)"
Average,23.2,2.4
Minimum,-0.4,0.0
Maximum,112.1,75.4

Climate ->

climate2018-02-20T20_01_09.243Z.csv

Month,"Mean Precip. (cm)","Mean Temp. (°C)"
January,8.6,0.5
February,7.6,1.2
March,9.8,5.9
April,9.2,11.7
May,9.3,17.5
June,9.8,22.4
July,11.4,25.1
August,11.7,24.2
September,9.4,20.5
October,8.3,14.3
November,8.0,8.3
December,9.2,2.6
Annual,112.2,12.9

Pt Sources ->

pointsource_2018-02-20T20_01_45.395Z.csv

"NPDES Code",City,"Discharge (MGD)","TN Load (kg/yr)","TP Load (kg/yr)"
0063355,"City Of Camden","No Data","No Data","No Data"
0136441,"City Of Camden","No Data","No Data","No Data"
0169285,"City Of Camden","No Data","No Data","No Data"
0004090,"City Of Camden",0.113,"5,469.280",468.800
0195707,"City Of Camden","No Data","No Data","No Data"
0129917,"City Of Camden","No Data","No Data","No Data"
0026182,"City Of Camden",53.100,"1,727,960.840","83,823.090"
G0137561,"City Of Gloucester City","No Data","No Data","No Data"
0026620,"City Of Gloucester City","No Data","No Data","No Data"
0108847,"City Of Gloucester City","No Data","No Data","No Data"
G050080,"Lower Merion Township",0.001,33.580,2.880
0011533,Philadelphia,4.083,"1,160.760","16,937.800"
0026662,Philadelphia,81.000,"1,521,327.620","37,577.450"
0050202,Philadelphia,"No Data","No Data","No Data"
0012629,Philadelphia,5.640,"5,306.350","23,393.950"
0058955,Philadelphia,"No Data","No Data","No Data"
0036447,Philadelphia,0.060,"2,921.410",250.410
0057690,Philadelphia,"No Data","No Data","No Data"
R900024,Philadelphia,"No Data","No Data","No Data"
0011088,Philadelphia,"No Data","No Data","No Data"
0011428,Philadelphia,0.000,3.390,0.290
0057479,Philadelphia,0.800,"38,713.390","3,318.290"
0011657,Philadelphia,41.060,"No Data","No Data"
0021440,"Township Of Haddon","No Data","No Data","No Data"
0062391,"Township Of West Deptford",11.750,"568,602.890","48,737.390"
0005401,"Township Of West Deptford",0.272,73.050,"1,126.140"
"Total For Area of Interest",,197.879,"3,871,572.560","215,636.490"

Animals ->

animal_estimate_2018-02-20T20_02_28.153Z.csv

Animal,Count
"Chickens, Broilers",215
"Chickens, Layers",245
"Cows, Beef",14
"Cows, Dairy",14
Horses,112
Pigs/Hogs/Swine,40
Sheep,41
Turkeys,1

Water Qual ->

catchment_water_quality_2018-02-20T20_03_05.181Z.csv

Id,"Area (ha)","Total N (kg/ha)","Total P (kg/ha)","Total SS (kg/ha)","Avg TN (mg/l)","Avg TP (mg/l)","Avg TSS (mg/l)"
1991,142.630,3.140,0.235,558.301,3.209,0.266,63.627
1992,"1,325.910",10.242,0.839,825.188,3.209,0.268,63.606
1993,5.850,4.507,0.345,502.315,1.772,0.134,162.818
1994,"1,351.450",7.879,0.596,723.776,1.775,0.134,163.017
1995,737.870,16.242,55.270,762.330,3.212,0.270,61.336
1996,"2,825.470",8.077,0.627,800.851,3.211,0.260,60.874
1997,48.740,8.259,0.673,837.888,3.217,0.267,60.136
1998,"1,313.140",6.246,0.440,583.848,3.217,0.268,60.126
1999,613.580,7.402,0.546,683.521,3.220,0.272,60.009
2001,112.410,5.624,0.371,196.726,"No Data","No Data","No Data"
2002,199.100,7.026,0.514,699.816,3.222,0.273,59.966
2004,743.620,6.411,0.372,431.283,2.062,0.440,101.742
2058,"1,283.650",209.374,17.830,"1,174.374",3.252,0.267,58.400
3888,857.200,48.474,4.163,997.300,1.947,0.093,23.854
3889,28.420,0.309,0.009,68.751,9.576,0.721,92.563
3890,9.980,1.396,0.067,180.217,9.592,0.725,92.733
3891,4.950,0.073,0.001,40.926,1.859,0.068,89.171
3892,11.420,1.616,0.052,214.462,1.744,0.081,157.336
3893,54.860,8.113,0.502,898.578,1.758,0.082,158.842
3894,19.960,8.979,0.650,"1,360.015",1.756,0.081,155.538
3895,85.070,7.152,0.393,765.651,1.752,0.080,150.877
3896,5.400,8.171,0.555,988.922,1.769,0.080,146.778
3897,27.430,6.838,0.352,675.887,1.769,0.080,146.160
3898,4.140,8.716,0.583,"1,164.799",1.777,0.080,146.565
3899,856.030,8.419,0.379,688.037,1.776,0.080,145.179
3900,261.330,6.928,0.365,720.729,1.858,0.067,75.866
3901,1.350,5.944,0.270,197.001,"No Data","No Data","No Data"
3902,15.830,7.761,0.403,711.920,"No Data","No Data","No Data"
3903,3.780,7.224,0.359,488.565,"No Data","No Data","No Data"
3904,114.030,8.515,0.408,514.077,"No Data","No Data","No Data"
3905,100.090,8.371,0.334,155.023,1.861,0.067,66.106
3907,480.040,7.773,0.341,223.969,1.651,0.072,47.578
4068,24.550,2.465,0.069,48.529,231.193,19.593,138.622
4069,3.240,2.836,0.058,43.961,242.129,20.672,152.590
4070,504.410,"1,135.266",97.056,727.343,243.650,20.830,156.102
4071,942.270,"1,621.684",40.475,887.007,1.893,0.090,23.427
4072,13.040,3.413,0.311,650.623,1.413,0.084,136.649
4073,135.340,8.140,0.677,952.873,1.416,0.085,136.630
4075,4.320,6.162,0.432,539.182,1.499,0.111,153.942
4076,26.260,5.254,0.338,446.158,1.501,0.111,154.434
4078,24.460,6.692,0.490,624.450,1.530,0.115,159.558
4079,121.760,6.902,0.504,669.585,1.538,0.116,161.758
4080,50.000,8.077,0.648,891.947,"No Data","No Data","No Data"
4081,132.460,7.593,0.579,841.991,"No Data","No Data","No Data"
4082,12.230,5.714,0.419,542.249,1.366,0.077,128.111
4083,43.800,6.936,0.529,765.384,1.367,0.077,128.187
4086,12.770,6.453,0.440,658.533,1.345,0.074,124.659
4088,57.370,6.039,0.390,547.749,1.344,0.074,124.457
4090,66.010,5.947,0.379,572.311,1.345,0.074,125.049
4091,1.170,6.673,0.486,775.497,1.347,0.074,125.409
4093,2.160,5.128,0.280,614.560,"No Data","No Data","No Data"
4094,73.380,5.386,0.335,600.280,"No Data","No Data","No Data"
4096,108.630,6.117,0.416,675.916,1.335,0.072,124.675
4097,4.140,5.463,0.329,520.812,1.336,0.071,122.002
4098,106.920,5.919,0.370,674.478,1.336,0.071,122.102
4099,1.980,5.972,0.419,705.311,1.343,0.070,117.678
4100,137.320,5.762,0.347,615.688,1.343,0.070,117.565
4105,17.990,5.849,0.389,471.778,1.346,0.078,127.507
4107,10.430,6.824,0.503,657.104,1.347,0.078,128.493
4108,79.320,6.554,0.447,692.253,1.345,0.078,128.352
4109,34.620,6.633,0.451,739.531,1.336,0.076,125.363
4110,9.710,5.591,0.340,518.671,1.331,0.075,122.964
4111,3.150,7.817,0.620,841.422,1.332,0.076,123.261
4112,172.840,5.944,0.375,595.863,1.331,0.075,122.958
4113,1.980,5.428,0.293,479.035,1.347,0.074,122.213
4114,39.570,5.900,0.388,624.219,1.348,0.074,122.438
4121,"3,949.200",447.024,22.044,991.400,1.764,0.087,23.028
4122,229.770,3.032,0.203,313.523,1.763,0.077,186.988
4123,488.220,9.062,0.683,"1,715.876",1.785,0.079,190.461
4124,"1,236.060",8.630,0.553,"1,584.337",1.772,0.078,170.937
4172,"3,825.910",759.580,33.039,811.046,1.613,0.082,22.085
4176,"3,504.780",8.417,0.616,692.122,1.512,0.095,111.549
13147,629.410,7.102,2.232,"1,049.468",1.567,0.399,241.473
13151,150.090,7.152,0.429,"1,384.108","No Data","No Data","No Data"
13220,"3,021.890",8.125,0.561,645.402,1.570,0.095,117.217
13221,813.500,6.549,0.346,442.994,1.302,0.067,94.068
13222,4.320,6.042,0.289,465.683,"No Data","No Data","No Data"
13223,206.390,6.372,0.311,596.824,"No Data","No Data","No Data"
13224,"1,447.500",6.739,0.359,444.797,1.335,0.070,85.443
kellyi commented 6 years ago

Here's the set of resources currently created on HydroShare: https://beta.hydroshare.org/resource/655b37d31c144760949ec47166c5513a/

aufdenkampe commented 6 years ago

@rajadain & @ajrobbins, thanks for setting this up!

I would like to clarify that we DO NOT want "the HS-exported CSVs to exactly match the ones downloaded from the UI, in schema and filename."

Rather, there are elements of each that we want to retain in a single harmonized output. I will explain and mock those up in a moment.

aufdenkampe commented 6 years ago

Thank you @kellyi for posting examples of all the CSVs from the UI Analyze output.

Here are my thoughts:

Filenames:

I'll comment on columns next.

aufdenkampe commented 6 years ago

Columns: Let's combine BOTH sets of tables into a single table

Part of the idea here is to make this as easy as possible for Azavea developers. The other part of the idea here is to make it as easy as possible for others to use and reuse these output CSVs for a variety of different things.

I realize there is some redundancy with this approach, but I think the benefits outweigh those drawbacks. I am, however, open to feedback.

aufdenkampe commented 6 years ago

@ajrobbins just sent an email suggesting you all had concerns with this approach, and offering to talk. I would be very happy to discuss, and develop a consensus approach. I actually went back and forth a bunch on my recommendations, so I'm very interested in hearing your perspectives and concerns.

Ideally, we would change the field names/tags and output order in the API, because they really should have units associated with them and it might help readability. However, I understand that that would have large consequences to the rest of your code, and I wanted to make this a relatively low-effort task for you all. ​ Another concern for me is the issue of rounding. Rounded numbers are the right thing to present in the tables on the UI, but not what is needed for export, as anyone exporting will likely be reusing the numbers for calculations, and the rounded values will introduce noticeable and annoying errors in subsequent calculations. We want to avoid that.

I'm expecting we'll find a good solution that meets all of these needs. Looking forward to the call.

ajrobbins commented 6 years ago

After talking with @aufdenkampe, we are going to revisit this issue later in the contract. For now, we will implement only the file name changes for the csv's downloaded from the Analyze step in the user interface (NOT the HydroShare-exported csv's):

To paraphrase Anthony, we'll continue to append the filename with a date and token, as we do now. This is necessary to distinguish outputs. However, we will change the words in the filename to mirror those used in the matching csv's for HydroShare export (which match the API). Filename conversions for csvs downloaded from Analyze are:

Streams:

streams2018-02-20T19_55_22.470Z.csv becomes analyze_streams_2018-02-20T19_55_22.470Z.csv

Land:

nlcd_land_cover_2018-02-20T19_58_37.199Z.csv becomes analyze_land_2018-02-20T19_58_37.199Z.csv

Soil:

nlcd_soils_2018-02-20T19_59_50.340Z.csv becomes analyze_soil_2018-02-20T19_59_50.340Z.csv

Terrain:

terrain2018-02-20T20_00_23.442Z.csv becomes analyze_terrain_2018-02-20T20_00_23.442Z.csv

Climate:

climate2018-02-20T20_01_09.243Z.csv becomes analyze_climate_2018-02-20T20_01_09.243Z.csv

Pt Sources:

pointsource_2018-02-20T20_01_45.395Z.csv becomes analyze_pointsource_2018-02-20T20_01_45.395Z.csv

Animals:

animal_estimate_2018-02-20T20_02_28.153Z.csv becomes analyze_animals_2018-02-20T20_02_28.153Z.csv

Water Qual:

catchment_water_quality_2018-02-20T20_03_05.181Z.csv becomes analyze_catchment_water_quality_2018-02-20T20_03_05.181Z.csv

Note for the future: Ultimately, the creation of the UI and HydroShare csvs should likely be refactored out into a single function and used for both implementations. It's important for users to have field names that include units, as we do with the UI-generated csvs, but also to have full significant digits for use in other applications, as we do with the HS-destined csvs. However, we want to remain in sync with the API field names so as not to confuse users. We need to:

aufdenkampe commented 6 years ago

Thanks @ajrobbins, that is an excellent summary. Can you create a new issue that outlines out ultimate goal of refactoring the CSVs, so we don't loose those ideas when this issue is closed?

One additional thought: when we do refactor, we'll want to modify API field names to not only indicate units, but also unambiguously describe the variable/property (similar to the UI) while following a defined pattern. I can help with those semantics when the time comes.

rajadain commented 6 years ago

Created #2673 for the follow up refactor.