Changes in input files (list type)

[nheeren]

Just for documentation purposes (in case another set of files need to be corrected too). Close whenever you see fit.

Work in progress -- still updating the issue with more changes. Will remove this message once done

I made the following changes to the list-type input files in order to make them work with the parser & uploader scripts. Ideally they were errors 😆:

• remove first column in Cover sheet
  • 3_LT_AluCycle_LIU_2013.xlsx
  • 3_LT_IAI_GARC_2011.xlsx
• Column header in Values_Master:
  • 2_IUS_Buildings_Europe_2008_Enerdata.xlsx: "regin" to "region"
• Changed values
  • 3_LT_NonResBuildings_Minneapolis_2000.xlsx: Cover!D25: "engineering_material" to "material"
  • 3_MC_Vehicles_Hawkins_2012.xlsx: Cover!D26: "engineering_material" to "material"
  • 3_MC_Vehicles_Modaresi_2014.xlsx:** Cover!D26: "engineering_material" to "material"
  • 3_LT_AluCycle_LIU_2012.xlsx: cell G15 "custom " to "custom"
• Changed cell format
  • 3_LT_NonResBuildings_Minneapolis_2000.xlsx: Cover!G12:G14 from timestamp to integer
• Rename sheet
  • 3_SHA_FabricationYield_CollectionRate_Alu_Liu_2012.xlsx "Data" to "Values_Master"
  • 4_PY_FabricationYield_Alu_Liu_2012.xlsx "Data" to "Values_Master"
  • 4_PY_FabricationYield_Cullen2012.xlsx "Data" to "Values_Master"
  • 4_PY_FabricationYield_Dahlstroem2004.xlsx "Data" to "Values_Master"
  • 4_PY_FabricationYield_Hatayama2010.xlsx "Data" to "Values_Master"
  • 4_PY_WorldSteel_EoL_RR_SteelScrap.xlsx "Data" to "Values_Master"
• Fixed lower-case issues (see #11):
  • 1_UPI_USLCI_Aluminum_cold_rolling_at_plant.xlsx: Hydrogen Chloride, Carbon-14, Iodine-129
• Corrected values to comply with existing (non-custom) classification:
  • 2_IUS_Buildings_Europe_2008_Enerdata.xlsx: "residential buildings" to "residential building" and "Use phase" to "use phase"
  • 2_IUS_Wood_buildings_Pingoud_2000.xlsx: "Carbon" to "carbon"
  • 3_LT_ResBuildings_USA_1997_2009.xlsx: "USA" to "United States"
  • 3_MC_Buildings_Gustavsson_2006.xlsx: "multi-family house, wooden frame" to "multi-family houses, wooden frame". "multi-family house, concrete frame" could not be unambiguously fixed.
  • 3_MC_Buildings_Gustavsson_2011.xlsx: Almost all values for engineering_material to lowercase and "multi-family house, wooden frame" to "multi-family houses, wooden frame"
  • 3_MC_Buildings_Kleemann_2014.xlsx: "multi-family house, concrete frame" could not be unambiguously fixed.
  • 3_MC_Buildings_Monahan_2010.xlsx: "brick" to "bricks" and "single family house" to "single-family house"
  • 3_MC_Buildings_Sinha_2016.xlsx: Removed whitespaces after concrete, plywood, particle board, plasterboard, rock wool
  • 3_MC_ConstructionImpact_Liu_2006.xlsx: "residential buildings, urban" to "residential building, urban"
  • 3_MC_MetalDemand_DEETMAN_2018.xlsx Contains 'none' in value column, which is not allowed
  • 3_MC_Steel_DRCSCC_2005.xlsx: "residential buildings, urban" to "residential building, urban"
  • 3_MC_SteelDemand_HU_2010.xlsx "residential buildings, rural" to "residential building, rural"
  • 3_MC_Vehicles_Hawkins_2012.xlsx: Too many value errors (magnesium vs. manganese, zinc vs zinc roof, etc)
  • 4_PE_EnergyIntensity_AluminiumCycle_Liu_2012.xlsx: replaced all by lowercase; renamed column "unit denomitator"
  • '4_PY_WorldSteel_EoL_RR_SteelScrap.xlsx' lowercase "Waste management"
  • 5_CAP_PowerGenCapacity_Germany_2018.xlsx: rename column "stats_array_string" to "stats_array string". the column "value" contains empty values.
  • 6_PCS_Buildings_Europe_2008_Enerdata.xlsx: replaced "residential buildings" with "residential building" and "Use phase" with "use phase"
  • 6_PCS_Buildings_Indonesia_1985.xlsx: time classification is wrong (classification_id 9). apect_4 is not well defined as custom
  • 6_PCS_Buildings_Indonesia_1985.xlsx: time classification is wrong (classification_id 9)
  • 6_PCS_Buildings_USA_MOURA_2015.xlsx: time classification is wrong (classification_id 9)

The points in bold are breaking issues!

[stefanpauliuk]

engineering_material was created by Graedel/Fishman to specify that we are talking about 'typical' engineering materials, like steel, al, plastics, opposed to the general notion of 'materials' as 'goods or substances'. Since both 'engineering_material' and 'material' aspects link to the same dimension (5: material), this change does not matter much. But 'engineering_material' should be kept if the materials clearly are engineering materials.

"Data" to "Values_Master": I planned to use cell H11 to indicate the sheet where the data are. That is why data should be in on the sheet with the name as in H11. But I was not consistent with that rule, many datasets still are on "Values_Master", which is an ODYM template legacy.

[nheeren]

But 'engineering_material' should be kept if the materials clearly are engineering materials.

In that case it should be either indicated as custom or added to the classifications I suppose.

[nheeren]

Added a couple of new changes. Some breaking issues in bold.

[stefanpauliuk]

3_MC_Buildings_Gustavsson_2006

• changed to correct building type in template.

4_PY_WorldSteel_EoL_RR_SteelScrap

• changed classification for aspect 5 from 5 to 4, please update dataset table entry!

'3_LT_AluCycle_LIU_2012.xlsx', '3_LT_AluCycle_LIU_2013.xlsx', '3_LT_IAI_GARC_2011.xlsx', '3_LT_MetalDemand_DEETMAN_2018.xlsx', '3_LT_SteelCycle_PAULIUK_2013.xlsx', '3_MC_SteelDemand_HU_2010.xlsx'

• Changed stats_array_string to new format

3_MC_Buildings_Kleemann_2014.xlsx,

• fixed classification items in data template

6_PCS_Buildings_Indonesia_1985.xlsx, 6_PCS_Buildings_Indonesia_BPS_2008_2015.xlsx, '6_PCS_Buildings_USA_MOURA_2015.xlsx'

• changed time classification from 9 to 3, please update dataset table entry!

Issue that remain "apect_4 is not well defined as custom" -> This I did not understand!

3_MC_MetalDemand_DEETMAN_2018.xlsx, 5_CAP_PowerGenCapacity_Germany_2018.xlsx,

• needs discussion: in the main database, I dropped the NOT NULL constraint for data.value to allow us to distinguish between '0' and 'no data'. Does that make sense?

'3_MC_Vehicles_Hawkins_2012.xlsx',

• Need to add missing classification items to db. will do so during weekend.

[nheeren]

Thanks!

changed time classification from 9 to 3, please update dataset table entry!

You mean manually change aspect2_classification from 9 to 3? I assume that is just a quick fix for now? It is curious my script worked. Probably there should be another safety feature implemented...

needs discussion: in the main database, I dropped the NOT NULL constraint for data.value to allow us to distinguish between '0' and 'no data'. Does that make sense?

Good question. I think it makes sense for now. In our material intensity data project i was extremely explicit about missing values: https://github.com/nheeren/material_intensity_db/blob/master/codebook.md#special-values (Stefan has access - for everybody else: will be made public after the paper review). In short: I differentiate between "no observation" and "no information". I guess I would suggest to think about such a codebook here as well and define exactly what a NULL implies.

Need to add missing classification items to db. will do so during weekend.

Thanks. Please notify me. This is the last file I cannot process right now.

[stefanpauliuk]

You mean manually change aspect2_classification from 9 to 3? I assume that is just a quick fix for now?

Actually, 3 was the right classification, that is why it worked. Unlikely that two classifications will contain the same items, don't think that checks are necessary at this point.

OK, let's allow for NULL value, and specify in the comment field whether it's no observation or no information.

[nheeren]

it seems like we solved all list-data related file issues