Conventions on the use of constants

[mmyrte]

TL;DR: climada.util.constants could (mostly) be migrated to climada/conf/defaults.conf; this thread should resolve whether we should do this.

We got to talking about the proper place for storing constants and configuration aspects at today's coding convention meeting; I offered to summarise and start a general discussion here:

Distinct use cases

At least those that I know of, and that were discussed.

Reader support, e.g. through constant dictionary mappings: https://github.com/CLIMADA-project/climada_python/blob/b9d0c49bef93e224c2191eb448d97d85ba55fbe3/climada/hazard/base.py#L69-L72

Unit conversion, e.g. through floats: https://github.com/CLIMADA-project/climada_python/blob/91cfc25370ff84631fe69e85f9a4a6b74c7d0fa9/climada/util/constants.py#L174-L175

Index-based mappings, e.g. ISIMIP_NATID_TO_ISO https://github.com/CLIMADA-project/climada_python/blob/91cfc25370ff84631fe69e85f9a4a6b74c7d0fa9/climada/util/constants.py#L148-L150

Resource locaters, local or remote, e.g. relating to IBTRACS https://github.com/CLIMADA-project/climada_python/blob/91cfc25370ff84631fe69e85f9a4a6b74c7d0fa9/climada/hazard/tc_tracks.py#L70-L74

Options

I may have missed something in the meeting, and there are of course other options. Please mention them.

• @Evelyn-M Move constant definitions that do not clearly belong with a class (e.g. those making assumptions about file formats) to a separate location for better overview.
• Let things be as they are.
• @mmyrte Start by moving URLs to the configuration file. May also include moving the config file to the (in my experience slightly more orthodox) root of the repository.

What should we do?

[Evelyn-M]

Thanks, very nice overview @mmyrte!

In accordance with your classifications, I'd think

• URLs: moving them away from inside specific modules is a good idea. I would leave the config.py file as is, though, and treat them as candidates for constants.py. Even if they are used only in one module. T
• file paths: for the sake of uniformity, same handling as URLS (?). Might seem like an overkill, would apply to situations like this one:

  from climada.util.constants import SYSTEM_DIR
  TOT_RADIATIVE_FORCE = os.path.join(SYSTEM_DIR, 'rcp_db.xls')
  """© RCP Database (Version 2.0.5) http://www.iiasa.ac.at/web-apps/tnt/RcpDb.
  generated: 2018-07-04 10:47:59.""" 

• Index-based mappings (to keep with your terminology of the list example ISIMIP_NATID_TO_ISO and hard-coded dictionaries, which are long, crowd both the module & would crowd the constants-file, and are used only by one module: this is really the case where I'm wondering to move them to a yaml (or any other file format which is convenient for storing such data, yes, one file for each one), give them a clear name and move them to a /resouces sub-folder in the hazard, exposure, measures, engine folders. This is debatable, since we don't have such a folder structure right now. Alternatively, could also be utils/resources/all those files (wouldn't recommend, though). An example which I'm not referring to in this suggestion (because it's "short enough"):

  BM_YEARS = [2016, 2012] # latest first
  # Years with GPW population data available:
  GPW_YEARS = [2020, 2015, 2010, 2005, 2000]

General things to keep in mind:

• for the sake of backwards-compatibility & pragmatism, I'd leave both the config & the constants file where they are (in util)
• moving "simple" constants like URLs & file paths to the constant.py should be relatively straight-forward (not breaking any code), but still needs to be done, manually, by someone.
• for better readability, the constants.py file might be structured into sections, e.g. URLS, FILEPATHS, NUMERIC CONSTANTS, ...
• Are there any other uses of "constants" in climada, which are not yet mentioned in your overview?

[emanuel-schmid]

More general things to keep in mind:

• a configuration file is an interface, something a user may want or need to adapt, while constants can or must stay unchanged.
• at some point in the (hopefully near) future CLIMADA-python is going to be a package installable through Pip or Conda or both. By then the path to the config file has to be adjustable, because in general it is not granted that a user can modify a file within the conda or pip package directories, for in general they may not have the required permissions.
• The same applies for any path that is subject to change during or preceding the usage of CLIMADA-python modules. These should be put into the config.yaml file rather than into the constants.py file.

Less general things:

• pathes of files written during tests or demos should be defined in the config file

[mmyrte]

Thanks for the feedback! What do you think of the following proposal?

Constants and Configuration

When developing for CLIMADA, please distinguish between constants, i.e. parameters that are not supposed to ever change, and configuration parameters, where we provide the user with a sensible set of defaults, but which should be able to be overridden. A few examples of reasoning:

• Numerical constants to decode files (e.g. -999 = NaN) should be kept in the module that implements the reader because…

  • if the file format changes, it shouldn't be up to the user to fix it through a configuration, but up to us, since we would probably want to fix it for us a well.
  • there will not be a second function relying on the constant, since this indicates a need for abstraction anyways.

• The location of a test file should be declared in its corresponding unit test because...

  • the file's primary purpose is to ensure the numerical reproducibility accross commits/environments and should not ever be changed.
  • it is logically tied to the unit test. Integration tests should import the definition from the unit test, since they refer to several such test files.

• The location of an online database should be declared in the settings since…

  • a user may want to point it to a different version of the data.

• The location of a demo file should be declared in the settings since…

  • the software doesn't rely on these data, hence making them subject to change.

• The location of a template required by a writer function should be declared in the settings since...

  • a user may wish to adapt the template for their own use. (Caveat: No numerical index logic allowed.)

Regarding the location of the config, the package itself should contain a defaults.conf, but to allow for it to be overridden first by $HOME/.climada.conf and then by $PWD/climada.conf or $PWD/.climada.conf. I think environmental variables are not transparent enough for a majority of the userbase.

I would also aim to abolish the constants.py, since constants make sense through their context. The danger of abusing it as a config file seems too great to me.

Are there any other uses of "constants" in climada, which are not yet mentioned in your overview?

Not that I know of, but please check out the overview below if you see any other usecase. I've excluded TESTS and LOGGER.

Show All The Constants

``` Searching 300 files for "^\h*(?!TESTS|LOGGER)([\u\d\_]+)\h=\h" (regex, case sensitive) ./climada/engine/calibration_opt.py: 66 or event. 67 """ 68: IFS = ImpactFuncSet() 69 IFS.append(impact_func) 70 impacts = Impact() 71 impacts.calc(exposure, IFS, hazard) 72 if yearly_impact: # impact per year 73: IYS = impacts.calc_impact_year_set(all_years=True) 74 # Loop over whole year range: 75 if df_out.empty | df_out.index.shape[0] == 1: ./climada/engine/cost_benefit.py: 34 LOGGER = logging.getLogger(__name__) 35 36: DEF_PRESENT_YEAR = 2016 37 """ Default present reference year """ 38 39: DEF_FUTURE_YEAR = 2030 40 """ Default future reference year """ 41 42: NO_MEASURE = 'no measure' 43 """ Name of risk metrics when no measure is applied """ 44 ./climada/engine/impact_data.py: 37 LOGGER = logging.getLogger(__name__) 38 39: PERIL_SUBTYPE_MATCH_DICT = dict(TC='Tropical cyclone', 40 T1='Storm', 41 TS='Coastal flood', .. 53 FB='Land fire (Brush, Bush, Pastur') 54 55: PERIL_TYPE_MATCH_DICT = dict(DR='Drought', 56 TC='Storm', 57 EQ='Earthquake', .. 206 for track in range(0, len(names)): 207 # select track 208: TC = inten[track,] 209 # select only indices that are not zero 210 hits = TC.nonzero()[1] ./climada/engine/test/test_cost_benefit.py: 32 from climada.util.constants import ENT_DEMO_FUTURE, ENT_DEMO_TODAY 33 34: HAZ_DATA_DIR = os.path.join(os.path.dirname(__file__), '../../hazard/test/data') 35: HAZ_TEST_MAT = os.path.join(HAZ_DATA_DIR, 'atl_prob_no_name.mat') 36: ENT_TEST_MAT = os.path.join(os.path.dirname(__file__), 37 '../../entity/exposures/test/data/demo_today.mat') 38 ./climada/engine/test/test_impact.py: 31 from climada.util.constants import ENT_DEMO_TODAY, DEF_CRS 32 33: HAZ_DIR = os.path.join(os.path.dirname(__file__), os.pardir, os.pardir, 'hazard/test/data/') 34: HAZ_TEST_MAT = os.path.join(HAZ_DIR, 'atl_prob_no_name.mat') 35 36: DATA_FOLDER = os.path.join(os.path.dirname(__file__) , 'data') 37 38 class TestFreqCurve(unittest.TestCase): ./climada/engine/test/test_impact_data.py: 25 import climada.engine.impact_data as im_d 26 27: DATA_FOLDER = os.path.join(os.path.dirname(__file__) , 'data') 28: EMDAT_TEST_CSV = os.path.join(DATA_FOLDER, 'emdat_testdata_BGD_USA_1970-2017.csv') 29: EMDAT_TEST_CSV_FAKE = os.path.join(DATA_FOLDER, 'emdat_testdata_fake_2007-2011.csv') 30 31 class TestEmdatImport(unittest.TestCase): ./climada/entity/disc_rates/base.py: 37 LOGGER = logging.getLogger(__name__) 38 39: DEF_VAR_MAT = {'sup_field_name': 'entity', 40 'field_name': 'discount', 41 'var_name': {'year' : 'year', .. 45 """ MATLAB variable names """ 46 47: DEF_VAR_EXCEL = {'sheet_name': 'discount', 48 'col_name': {'year' : 'year', 49 'disc' : 'discount_rate' ./climada/entity/disc_rates/test/test_base.py: 26 from climada.util.constants import ENT_TEMPLATE_XLS, ENT_DEMO_TODAY 27 28: CURR_DIR = os.path.dirname(__file__) 29: ENT_TEST_MAT = os.path.join(CURR_DIR, 30 '../../exposures/test/data/demo_today.mat') 31 ./climada/entity/exposures/base.py: 41 LOGGER = logging.getLogger(__name__) 42 43: INDICATOR_IF = 'if_' 44 """ Name of the column containing the impact functions id of specified hazard""" 45 46: INDICATOR_CENTR = 'centr_' 47 """ Name of the column containing the centroids id of specified hazard """ 48 49: DEF_REF_YEAR = 2018 50 """ Default reference year """ 51 52: DEF_VALUE_UNIT = 'USD' 53 """ Default reference year """ 54 55: DEF_VAR_MAT = {'sup_field_name': 'entity', 56 'field_name': 'assets', 57 'var_name': {'lat' : 'lat', ./climada/entity/exposures/black_marble.py: 42 LOGGER = logging.getLogger(__name__) 43 44: DEF_RES_NOAA_KM = 1 45 """ Default approximate resolution for NOAA NGDC nightlights in km.""" 46 47: DEF_RES_NASA_KM = 0.5 48 """ Default approximate resolution for NASA's nightlights in km.""" 49 50: DEF_HAZ_TYPE = 'TC' 51 """ Default hazard type used in impact functions id. """ 52 53: DEF_POLY_VAL = [0, 0, 1] 54 """ Default polynomial transformation used. """ 55 ./climada/entity/exposures/gdp_asset.py: 35 LOGGER = logging.getLogger(__name__) 36 37: DEF_HAZ_TYPE = 'RF' 38 39: CONVERTER = os.path.join(SYSTEM_DIR, 'GDP2Asset_converter_2.5arcmin.nc') 40 41 ./climada/entity/exposures/gpw_import.py: 19 LOGGER = logging.getLogger(__name__) 20 21: FILENAME_GPW = 'gpw_v4_population_count_rev%02i_%04i_30_sec.tif' 22: FOLDER_GPW = os.path.join(SYSTEM_DIR, \ 23 'gpw-v4-population-count-rev%02i_%04i_30_sec_tif') 24: GPW_VERSIONS = [11, 10, 12, 13] 25 # FILENAME_GPW1 = '_30_sec.tif' 26: YEARS_AVAILABLE = np.array([2000, 2005, 2010, 2015, 2020]) 27: BUFFER_VAL = -340282306073709652508363335590014353408 28 # Hard coded value which is used for NANs in original GPW data 29 ./climada/entity/exposures/litpop.py: 47 """Define LitPop class.""" 48 # Black Marble nightlight tile names, %i represents the Black Marble reference year 49: BM_FILENAMES = ['BlackMarble_%i_A1_geo_gray.tif', \ 50 'BlackMarble_%i_A2_geo_gray.tif', \ 51 'BlackMarble_%i_B1_geo_gray.tif', \ .. 58 # https://earthobservatory.nasa.gov/features/NightLights/page3.php 59 # Update if new years get available! 60: BM_YEARS = [2016, 2012] # latest first 61 # Years with GPW population data available: 62: GPW_YEARS = [2020, 2015, 2010, 2005, 2000] 63 64: NASA_RESOLUTION_DEG = (15*UnitRegistry().arc_second).to(UnitRegistry().deg). \ 65 magnitude 66 67: WORLD_BANK_INC_GRP = \ 68 "http://databank.worldbank.org/data/download/site-content/OGHIST.xls" 69 """ Income group historical data from World bank.""" 70 71: DEF_RES_NASA_KM = 0.5 72 """ Default approximate resolution for NASA's nightlights in km.""" 73 74: DEF_RES_GPW_KM = 1 75 """ Default approximate resolution for the GPW dataset in km.""" 76 77: DEF_RES_NASA_ARCSEC = 15 78 """ Default approximate resolution for NASA's nightlights in arcsec.""" 79 80: DEF_RES_GPW_ARCSEC = 30 81 """ Default approximate resolution for the GPW dataset in arcsec.""" 82 83: DEF_HAZ_TYPE = '' 84 """ Default hazard type used in impact functions id, i.e. TC """ 85 ./climada/entity/exposures/nightlight.py: 42 LOGGER = logging.getLogger(__name__) 43 44: NOAA_SITE = "https://ngdc.noaa.gov/eog/data/web_data/v4composites/" 45 """ NOAA's URL used to retrieve nightlight satellite images. """ 46 47: NOAA_RESOLUTION_DEG = (30*UnitRegistry().arc_second).to(UnitRegistry().deg). \ 48 magnitude 49 """ NOAA nightlights coordinates resolution in degrees. """ 50 51: NASA_RESOLUTION_DEG = (15*UnitRegistry().arc_second).to(UnitRegistry().deg). \ 52 magnitude 53 """ NASA nightlights coordinates resolution in degrees. """ 54 55: NOAA_BORDER = (-180, -65, 180, 75) 56 """ NOAA nightlights border (min_lon, min_lat, max_lon, max_lat) """ 57 58: NASA_SITE = 'https://www.nasa.gov/specials/blackmarble/*/tiles/georeferrenced/' 59 """NASA nightlight web url.""" 60 61: BM_FILENAMES = ['BlackMarble_*_A1_geo_gray.tif', 62 'BlackMarble_*_A2_geo_gray.tif', 63 'BlackMarble_*_B1_geo_gray.tif', ./climada/entity/exposures/spam_agrar.py: 31 LOGGER = logging.getLogger(__name__) 32 33: DEF_HAZ_TYPE = 'DR' 34 """ Default hazard type used in impact functions id.""" 35 36: FILENAME_SPAM = 'spam2005V3r2_global' 37 """ """ 38 39: FILENAME_CELL5M = 'cell5m_allockey_xy.csv' 40 """ """ 41 42: FILENAME_PERMALINKS = 'spam2005V3r2_download_permalinks.csv' 43 """ """ 44 45: BUFFER_VAL = -340282306073709652508363335590014353408 46 """ Hard coded value which is used for NANs in original data """ 47 ./climada/entity/exposures/test/test_base.py: 35 from climada.util.coordinates import equal_crs 36 37: DATA_DIR = os.path.join(os.path.dirname(__file__), 'data') 38 39 def good_exposures(): ./climada/entity/exposures/test/test_black_marble.py: 30 from climada.entity.exposures.nightlight import NOAA_BORDER, NOAA_RESOLUTION_DEG 31 32: SHP_FN = shapereader.natural_earth(resolution='10m', \ 33 category='cultural', name='admin_0_countries') 34: SHP_FILE = shapereader.Reader(SHP_FN) 35 36: ADM1_FILE = shapereader.natural_earth(resolution='10m', 37 category='cultural', 38 name='admin_1_states_provinces') 39: ADM1_FILE = shapereader.Reader(ADM1_FILE) 40 41 class TestCountryIso(unittest.TestCase): ./climada/entity/exposures/test/test_mat.py: 25 from climada.entity.exposures.base import Exposures, DEF_VAR_MAT 26 27: ENT_TEST_MAT = CURR_DIR = os.path.join(os.path.dirname(__file__), 'data/demo_today.mat') 28 29 class TestReader(unittest.TestCase): ./climada/entity/exposures/test/test_nighlight.py: 25 from climada.util.constants import SYSTEM_DIR 26 27: BM_FILENAMES = nightlight.BM_FILENAMES 28 29 class TestNightLight(unittest.TestCase): ./climada/entity/impact_funcs/base.py: 72 def plot(self, axis=None, **kwargs): 73 """Plot the impact functions MDD, MDR and PAA in one graph, where 74: MDR = PAA * MDD. 75 76 Parameters: ./climada/entity/impact_funcs/flood.py: 29 LOGGER = logging.getLogger(__name__) 30 31: DEF_VAR_EXCEL = {'sheet_name': 'damagefunctions', 32 'col_name': {'func_id': 'DamageFunID', 33 'inten': 'Intensity', ./climada/entity/impact_funcs/impact_func_set.py: 37 LOGGER = logging.getLogger(__name__) 38 39: DEF_VAR_EXCEL = {'sheet_name': 'impact_functions', 40 'col_name': {'func_id' : 'impact_fun_id', 41 'inten' : 'intensity', .. 49 """ Excel and csv variable names """ 50 51: DEF_VAR_MAT = {'sup_field_name': 'entity', 52 'field_name': 'damagefunctions', 53 'var_name': {'fun_id' : 'DamageFunID', ./climada/entity/impact_funcs/test/test_imp_fun_set.py: 26 from climada.util.constants import ENT_TEMPLATE_XLS, ENT_DEMO_TODAY 27 28: CURR_DIR = os.path.dirname(__file__) 29: ENT_TEST_MAT = os.path.join(CURR_DIR, '../../exposures/test/data/demo_today.mat') 30 31 class TestConstructor(unittest.TestCase): ./climada/entity/measures/base.py: 32 LOGGER = logging.getLogger(__name__) 33 34: IF_ID_FACT = 1000 35 """ Factor internally used as id for impact functions when region selected.""" 36 37: NULL_STR = 'nil' 38 """ String considered as no path in measures exposures_set and hazard_set or 39 no string in imp_fun_map""" ./climada/entity/measures/measure_set.py: 36 LOGGER = logging.getLogger(__name__) 37 38: DEF_VAR_MAT = {'sup_field_name': 'entity', 39 'field_name': 'measures', 40 'var_name': {'name' : 'name', .. 59 """ MATLAB variable names """ 60 61: DEF_VAR_EXCEL = {'sheet_name': 'measures', 62 'col_name': {'name' : 'name', 63 'color' : 'color', ./climada/entity/measures/test/test_base.py: 33 from climada.util.constants import EXP_DEMO_H5, HAZ_DEMO_H5 34 35: DATA_DIR = os.path.join(os.path.dirname(__file__), os.pardir, os.pardir, \ 36 os.pardir, 'hazard', 'test', 'data') 37: HAZ_TEST_MAT = os.path.join(DATA_DIR, 'atl_prob_no_name.mat') 38 39: DATA_DIR = os.path.join(os.path.dirname(__file__), 'data') 40: ENT_TEST_MAT = os.path.join(os.path.dirname(__file__), 41 '../../exposures/test/data/demo_today.mat') 42 ./climada/entity/measures/test/test_meas_set.py: 27 from climada.util.constants import ENT_TEMPLATE_XLS, ENT_DEMO_TODAY 28 29: DATA_DIR = os.path.join(os.path.dirname(__file__), 'data') 30: ENT_TEST_MAT = os.path.join(os.path.dirname(__file__), 31 '../../exposures/test/data/demo_today.mat') 32 ./climada/entity/test/test_entity.py: 30 from climada.util.constants import ENT_TEMPLATE_XLS 31 32: ENT_TEST_MAT = os.path.join(os.path.dirname(__file__), 33 '../exposures/test/data/demo_today.mat') 34 ./climada/hazard/ag_drought.py: 37 from climada.util import dates_times as dt 38 39: DFL_CROP = '' 40: INT_DEF = 'Yearly Yield' 41 42 LOGGER = logging.getLogger(__name__) 43 44: HAZ_TYPE = 'AD' 45 """ Hazard type acronym for Agricultural Drought """ 46 ./climada/hazard/base.py: 48 LOGGER = logging.getLogger(__name__) 49 50: DEF_VAR_EXCEL = {'sheet_name': {'inten' : 'hazard_intensity', 51 'freq' : 'hazard_frequency' 52 }, .. 67 """ Excel variable names """ 68 69: DEF_VAR_MAT = {'field_name': 'hazard', 70 'var_name': {'per_id' : 'peril_ID', 71 'even_id' : 'event_ID', ./climada/hazard/drought.py: 47 48 49: DFL_THRESHOLD = -1 50: DFL_INTENSITY_DEF = 1 51 52 53: SPEI_FILE_URL = r'http://digital.csic.es/bitstream/10261/153475/8' 54: SPEI_FILE_DIR = os.path.join(DATA_DIR, 'system') 55: SPEI_FILE_NAME = r'spei06.nc' 56 57 .. 59 LOGGER = logging.getLogger(__name__) 60 61: LATMIN = 44.5 62: LATMAX = 50 63: LONMIN = 5 64: LONMAX = 12 65 66 67: HAZ_TYPE = 'DR' 68 """ Hazard type acronym Drought """ 69 .. 250 251 if self.intensity_definition == 2: 252: HAZ_TYPE = 'DR_sumthr' 253 self.tag.haz_type = 'DR_sumthr' 254 elif self.intensity_definition == 3: 255: HAZ_TYPE = 'DR_sum' 256 self.tag.haz_type = 'DR_sum' 257 ./climada/hazard/flood.py: 44 LOGGER = logging.getLogger(__name__) 45 46: HAZ_TYPE = 'RF' 47 """ Hazard type acronym RiverFlood""" 48 49: RF_MODEL = ['ORCHIDEE', 50 'H08', 51 'LPJmL', .. 59 'VEGAS' 60 ] 61: CL_MODEL = ['gfdl-esm2m', 62 'hadgem2-es', 63 'ipsl-cm5a-lr', .. 68 'watch' 69 ] 70: SCENARIO = ['', 71 'historical', 72 'rcp26', .. 74 ] 75 76: PROT_STD = 'flopros' 77 78 ./climada/hazard/landslide.py: 41 LOGGER = logging.getLogger(__name__) 42 43: LS_FILE_DIR = os.path.join(DATA_DIR, 'system') 44 45: HAZ_TYPE = 'LS' 46 47 ./climada/hazard/storm_europe.py: 41 LOGGER = logging.getLogger(__name__) 42 43: HAZ_TYPE = 'WS' 44 """ Hazard type acronym for Winter Storm """ 45 46: N_PROB_EVENTS = 5 * 6 47 """ Number of events per historic event in probabilistic dataset """ 48 ./climada/hazard/tc_clim_change.py: 26 from climada.util.constants import SYSTEM_DIR 27 28: TOT_RADIATIVE_FORCE = os.path.join(SYSTEM_DIR, 'rcp_db.xls') 29 """ © RCP Database (Version 2.0.5) http://www.iiasa.ac.at/web-apps/tnt/RcpDb. 30 generated: 2018-07-04 10:47:59. """ ./climada/hazard/tc_tracks.py: 52 LOGGER = logging.getLogger(__name__) 53 54: SAFFIR_SIM_CAT = [34, 64, 83, 96, 113, 137, 1000] 55 """ Saffir-Simpson Hurricane Wind Scale in kn based on NOAA""" 56 57: CAT_NAMES = {1: 'Tropical Depression', 2: 'Tropical Storm', 58 3: 'Hurrican Cat. 1', 4: 'Hurrican Cat. 2', 59 5: 'Hurrican Cat. 3', 6: 'Hurrican Cat. 4', 7: 'Hurrican Cat. 5'} 60 """ Saffir-Simpson category names. """ 61 62: CAT_COLORS = cm_mp.rainbow(np.linspace(0, 1, len(SAFFIR_SIM_CAT))) 63 """ Color scale to plot the Saffir-Simpson scale.""" 64 65: IBTRACS_URL = 'ftp://eclipse.ncdc.noaa.gov/pub/ibtracs//v04r00/provisional/netcdf/' 66 """ FTP of IBTrACS netcdf file containing all tracks v4.0 """ 67 68: IBTRACS_FILE = 'IBTrACS.ALL.v04r00.nc' 69 """ IBTrACS v4.0 file all """ 70 71: DEF_ENV_PRESSURE = 1010 72 """ Default environmental pressure """ 73 ./climada/hazard/trop_cyclone.py: 46 LOGGER = logging.getLogger(__name__) 47 48: HAZ_TYPE = 'TC' 49 """ Hazard type acronym for Tropical Cyclone """ 50 51: INLAND_MAX_DIST_KM = 1000 52 """ Maximum inland distance of the centroids in km """ 53 54: CENTR_NODE_MAX_DIST_KM = 300 55 """ Maximum distance between centroid and TC track node in km """ 56 57: MODEL_VANG = {'H08': 0 58 } 59 """ Enumerate different symmetric wind field calculation.""" ./climada/hazard/centroids/centr.py: 44 __all__ = ['Centroids'] 45 46: DEF_VAR_MAT = {'field_names': ['centroids', 'hazard'], 47 'var_name': {'lat' : 'lat', 48 'lon' : 'lon', .. 56 """ MATLAB variable names """ 57 58: DEF_VAR_EXCEL = {'sheet_name': 'centroids', 59 'col_name': {'region_id' : 'region_id', 60 'lat' : 'latitude', ./climada/hazard/centroids/test/test_vec_ras.py: 34 from climada.util.coordinates import NE_EPSG, equal_crs 35 36: DATA_DIR = os.path.join(os.path.dirname(__file__), 'data') 37 38: VEC_LON = np.array([-59.6250000000000,-59.6250000000000,-59.6250000000000,-59.5416666666667, 39 -59.5416666666667,-59.4583333333333,-60.2083333333333,-60.2083333333333, 40 -60.2083333333333,-60.2083333333333,-60.2083333333333,-60.2083333333333, .. 111 -58.1250000000000,-58.1250000000000,-58.1250000000000,-58.1250000000000]) 112 113: VEC_LAT = np.array([13.125,13.20833333,13.29166667,13.125,13.20833333,13.125,12.625,12.70833333, 114 12.79166667,12.875,12.95833333,13.04166667,13.125,13.20833333,13.29166667, 115 13.375,13.45833333,13.54166667,13.625,13.70833333,13.79166667,12.54166667, ./climada/hazard/test/test_base.py: 32 from climada.util.coordinates import equal_crs 33 34: DATA_DIR = os.path.join(os.path.dirname(__file__), 'data') 35: HAZ_TEST_MAT = os.path.join(DATA_DIR, 'atl_prob_no_name.mat') 36 37 def dummy_hazard(): ./climada/hazard/test/test_landslide.py: 30 import math 31 from climada.util.constants import DATA_DIR 32: LS_FILE_DIR = os.path.join(DATA_DIR, 'system') 33 34: DATA_DIR_TEST = os.path.join(os.path.dirname(__file__), 'data') 35 36 #class TestTiffFcts(unittest.TestCase): ./climada/hazard/test/test_storm_europe.py: 31 from climada.util.constants import WS_DEMO_NC 32 33: DATA_DIR = os.path.join(os.path.dirname(__file__), 'data') 34 35 class TestReader(unittest.TestCase): ./climada/hazard/test/test_tc_tracks.py: 31 from climada.util.coordinates import coord_on_land, dist_to_coast 32 33: DATA_DIR = os.path.join(os.path.dirname(__file__), 'data') 34: TEST_TRACK = os.path.join(DATA_DIR, "trac_brb_test.csv") 35: TEST_TRACK_SHORT = os.path.join(DATA_DIR, "trac_short_test.csv") 36: TEST_RAW_TRACK = os.path.join(DATA_DIR, 'Storm.2016075S11087.ibtracs_all.v03r10.csv') 37 38 class TestIBTracs(unittest.TestCase): ./climada/hazard/test/test_trop_cyclone.py: 32 from climada.hazard.centroids.centr import Centroids 33 34: DATA_DIR = os.path.join(os.path.dirname(__file__), 'data') 35: HAZ_TEST_MAT = os.path.join(DATA_DIR, 'atl_prob_no_name.mat') 36: TEST_TRACK = os.path.join(DATA_DIR, "trac_brb_test.csv") 37: TEST_TRACK_SHORT = os.path.join(DATA_DIR, "trac_short_test.csv") 38 39: CENTR_DIR = os.path.join(os.path.dirname(__file__), 'data/') 40: CENTR_TEST_BRB = Centroids() 41 CENTR_TEST_BRB.read_mat(os.path.join(CENTR_DIR, 'centr_brb_test.mat')) 42 ./climada/test/test_calibration.py: 29 from climada.util.constants import ENT_DEMO_TODAY 30 31: HAZ_DIR = os.path.join(os.path.dirname(__file__), os.pardir, 'hazard/test/data/') 32: HAZ_TEST_MAT = os.path.join(HAZ_DIR, 'atl_prob_no_name.mat') 33 34: DATA_FOLDER = os.path.join(os.path.dirname(__file__) , 'data') 35 36 .. 72 impact = Impact() 73 impact.calc(ent.exposures, ent.impact_funcs, hazard) 74: IYS = impact.calc_impact_year_set(all_years=True) 75 76 # do the tests ./climada/test/test_centr.py: 25 from climada.util.constants import GLB_CENTROIDS_MAT, HAZ_TEMPLATE_XLS 26 27: HAZ_DIR = os.path.join(os.path.dirname(__file__), os.pardir, 'hazard/test/data/') 28: HAZ_TEST_MAT = os.path.join(HAZ_DIR, 'atl_prob_no_name.mat') 29 30 class TestCentroidsReader(unittest.TestCase): ./climada/test/test_hazard.py: 28 from climada.util.constants import HAZ_DEMO_FL 29 30: DATA_DIR = os.path.join(os.path.dirname(__file__), 'data') 31 32 class TestCentroids(unittest.TestCase): ./climada/test/test_landslide_integr.py: 34 from climada.util.constants import DATA_DIR 35 36: LS_FILE_DIR = os.path.join(DATA_DIR, 'system') 37 38: DATA_DIR_TEST = os.path.join(os.path.dirname(__file__), 'data') 39 40 class TestTiffFcts(unittest.TestCase): ./climada/test/test_osm.py: 25 import random 26 import geopandas 27: DATA_DIR = os.path.join(os.path.dirname(__file__), 'data') 28 29 class TestOpenStreetMapModule(unittest.TestCase): ./climada/test/test_OSM_unit.py: 25 import pandas as pd 26 27: DATA_DIR = os.path.join(os.path.dirname(__file__), 'data') 28 # just for now: 29 class TestOSMFunctions(unittest.TestCase): ./climada/test/test_tc.py: 29 from climada.hazard.centroids.centr import Centroids 30 31: DATA_DIR = os.path.join(os.path.dirname(__file__), os.pardir, 'hazard/test/data') 32: TEST_TRACK = os.path.join(DATA_DIR, "trac_brb_test.csv") 33 34: CENTR_TEST_BRB = Centroids() 35 CENTR_TEST_BRB.read_mat(os.path.join(DATA_DIR, 'centr_brb_test.mat')) 36 ./climada/test/test_tc_tracks.py: 28 from climada.util.constants import SYSTEM_DIR 29 30: DATA_DIR = os.path.join(os.path.dirname(__file__), 'data') 31 32 class TestDownload(unittest.TestCase): ./climada/util/config.py: 35 from climada.util.constants import SOURCE_DIR 36 37: WORKING_DIR = os.getcwd() 38: WINDOWS_END = WORKING_DIR[0:3] 39: UNIX_END = '/' 40 41 def remove_handlers(logger): .. 49 LOGGER.propagate = False 50 remove_handlers(LOGGER) 51: FORMATTER = logging.Formatter( 52 "%(asctime)s - %(name)s - %(levelname)s - %(message)s") 53: CONSOLE = logging.StreamHandler(stream=sys.stdout) 54 CONSOLE.setFormatter(FORMATTER) 55 LOGGER.addHandler(CONSOLE) .. 76 CONFIG['local_data'][key] = abspath 77 78: CONFIG_DIR = os.path.abspath(os.path.join(SOURCE_DIR, 'conf')) 79 80: DEFAULT_PATH = os.path.abspath(os.path.join(CONFIG_DIR, 'defaults.conf')) 81 if not os.path.isfile(DEFAULT_PATH): 82: DEFAULT_PATH = resource_filename(Requirement.parse('climada'), \ 83 'defaults.conf') 84 with open(DEFAULT_PATH) as def_conf: 85 LOGGER.debug('Loading default config file: %s', DEFAULT_PATH) 86: CONFIG = json.load(def_conf) 87 88 check_conf() ./climada/util/constants.py: 41 from fiona.crs import from_epsg 42 43: SOURCE_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), 44 os.pardir)) 45 """ climada directory """ 46 47: DATA_DIR = os.path.abspath(os.path.join(SOURCE_DIR, os.pardir, 'data')) 48 """ Folder containing the data """ 49 50: SYSTEM_DIR = os.path.abspath(os.path.join(DATA_DIR, 'system')) 51 """ Folder containing the data used internally """ 52 53 54: GLB_CENTROIDS_NC = os.path.join(SYSTEM_DIR, 'NatID_grid_0150as.nc') 55 """ Global centroids nc.""" 56 57: GLB_CENTROIDS_MAT = os.path.join(SYSTEM_DIR, 'GLB_NatID_grid_0360as_adv_2.mat') 58 """ Global centroids.""" 59 60: ENT_TEMPLATE_XLS = os.path.join(SYSTEM_DIR, 'entity_template.xlsx') 61 """ Entity template in xls format.""" 62 63: HAZ_TEMPLATE_XLS = os.path.join(SYSTEM_DIR, 'hazard_template.xlsx') 64 """ Hazard template in xls format.""" 65: NAT_REG_ID = os.path.join(SYSTEM_DIR, 'NatRegIDs.csv') 66 """ Look-up table ISO3 codes""" 67 68 69: HAZ_DEMO_FL = os.path.join(DATA_DIR, 'demo', 'SC22000_VE__M1.grd.gz') 70 """ Raster file of flood over Venezuela. Model from GAR2015""" 71 72 73: HAZ_DEMO_FLDDPH = os.path.join(DATA_DIR, 'demo', 'flddph_WaterGAP2_miroc5_historical_flopros_gev_picontrol_2000_0.1.nc') 74 """ NetCDF4 Flood depth from isimip simulations""" 75 76 77: HAZ_DEMO_FLDFRC = os.path.join(DATA_DIR, 'demo', 'fldfrc_WaterGAP2_miroc5_historical_flopros_gev_picontrol_2000_0.1.nc') 78 """ NetCDF4 Flood fraction from isimip simulations""" 79 80: HAZ_DEMO_MAT = os.path.join(DATA_DIR, 'demo', 'atl_prob.mat') 81 """ Hazard demo from climada in MATLAB: hurricanes from 1851 to 2011 over Florida with 100 centroids.""" 82 83: HAZ_DEMO_H5 = os.path.join(DATA_DIR, 'demo', 'tc_fl_1975_2011.h5') 84 """ Hazard demo in h5 format: ibtracs from 1975 to 2011 over Florida with 2500 centroids.""" 85 86: DEMO_GDP2ASSET = os.path.join(DATA_DIR, 'demo', 'gdp2asset_CHE_exposure.nc') 87 """Exposure demo file for GDP2Asset""" 88 89: WS_DEMO_NC = [os.path.join(DATA_DIR, 'demo', 'fp_lothar_crop-test.nc'), 90 os.path.join(DATA_DIR, 'demo', 'fp_xynthia_crop-test.nc')] 91 """ Winter storm in Europe files. These test files have been generated using .. 94 95 96: ENT_DEMO_TODAY = os.path.join(DATA_DIR, 'demo', 'demo_today.xlsx') 97 """ Entity demo present in xslx format.""" 98 99: ENT_DEMO_FUTURE = os.path.join(DATA_DIR, 'demo', 'demo_future_TEST.xlsx') 100 """ Entity demo future in xslx format.""" 101 102: EXP_DEMO_H5 = os.path.join(DATA_DIR, 'demo', 'exp_demo_today.h5') 103 """ Exposures over Florida """ 104 105 106: TC_ANDREW_FL = os.path.join(DATA_DIR, 'demo', 107 'ibtracs_global_intp-None_1992230N11325.csv') 108 """ Tropical cyclone Andrew in Florida """ 109 110 111: ONE_LAT_KM = 111.12 112 """ Mean one latitude (in degrees) to km """ 113 114: EARTH_RADIUS_KM = 6371 115 """ Earth radius in km """ 116 117: DEF_EPSG = 4326 118 """ Default EPSG code """ 119 120: DEF_CRS = from_epsg(DEF_EPSG) 121 """ Default coordinate reference system WGS 84 """ 122 ./climada/util/coordinates.py: 48 LOGGER = logging.getLogger(__name__) 49 50: NE_EPSG = 4326 51 """ Natural Earth CRS EPSG """ 52 53: NE_CRS = from_epsg(NE_EPSG) 54 """ Natural Earth CRS """ 55 56: TMP_ELEVATION_FILE = os.path.join(SYSTEM_DIR, 'tmp_elevation.tif') 57 """ Path of elevation file written in set_elevation """ 58 59: DEM_NODATA = -9999 60 """ Value to use for no data values in DEM, i.e see points """ 61 62: MAX_DEM_TILES_DOWN = 300 63 """ Maximum DEM tiles to dowload """ 64 ./climada/util/finance.py: 43 LOGGER = logging.getLogger(__name__) 44 45: WORLD_BANK_WEALTH_ACC = \ 46 "https://databank.worldbank.org/data/download/Wealth-Accounts_CSV.zip" 47 """ Wealth historical data (1995, 2000, 2005, 2010, 2014) from World Bank (ZIP). .. 49 Includes variable Produced Capital (NW.PCA.TO)""" 50 51: FILE_WORLD_BANK_WEALTH_ACC = "Wealth-AccountsData.csv" 52 53: WORLD_BANK_INC_GRP = \ 54 "http://databank.worldbank.org/data/download/site-content/OGHIST.xls" 55 """ Income group historical data from World bank.""" 56 57: INCOME_GRP_WB_TABLE = {'L' : 1, # low income 58 'LM': 2, # lower middle income 59 'UM': 3, # upper middle income .. 63 """ Meaning of values of world banks' historical table on income groups. """ 64 65: INCOME_GRP_NE_TABLE = {5: 1, # Low income 66 4: 2, # Lower middle income 67 3: 3, # Upper middle income .. 71 """ Meaning of values of natural earth's income groups.""" 72 73: FILE_GWP_WEALTH2GDP_FACTORS = 'WEALTH2GDP_factors_CRI_2016.csv' 74 """ File with wealth-to-GDP factors from the 75 Credit Suisse's Global Wealth Report 2017 (household wealth)""" ./climada/util/interpolation.py: 34 LOGGER = logging.getLogger(__name__) 35 36: DIST_DEF = ['approx', 'haversine'] 37 """ Distances """ 38 39: METHOD = ['NN'] 40 """ Interpolation methods """ 41 42: THRESHOLD = 100 43 """ Distance threshold in km. Nearest neighbors with greater distances are 44 not considered. """ ./climada/util/plot.py: 45 LOGGER = logging.getLogger(__name__) 46 47: RESOLUTION = 250 48 """ Number of pixels in one direction in rendered image """ 49 50: BUFFER = 1.0 51 """ Degrees to add in the border """ 52 53: MAX_BINS = 2000 54 """ Maximum number of bins in geo_bin_from_array """ 55 ./climada/util/test/test_finance.py: 26 nat_earth_adm0, world_bank, wealth2gdp, world_bank_wealth_account, _gdp_twn 27 28: SHP_FN = shapereader.natural_earth(resolution='10m', \ 29 category='cultural', name='admin_0_countries') 30: SHP_FILE = shapereader.Reader(SHP_FN) 31 32 class TestNetpresValue(unittest.TestCase): ./climada/util/test/test_hdf5.py: 27 import climada.util.hdf5_handler as hdf5 28 29: DATA_DIR = os.path.join(os.path.dirname(__file__), 'data') 30: HAZ_TEST_MAT = os.path.join(DATA_DIR, 'atl_prob_short_name.mat') 31 32 class TestFunc(unittest.TestCase): ./climada/util/test/test_save.py: 27 from climada.util.config import CONFIG 28 29: DATA_DIR = os.path.join(os.path.dirname(__file__), 'data') 30 31: IN_CONFIG = copy.copy(CONFIG['local_data']['save_dir']) 32 33 class TestSave(unittest.TestCase): 209 matches across 61 files ```

Index-based mappings […] are long […] and are used only by one module: […] move them to a /resouces sub-folder in the hazard, exposure, measures, engine folders.

This is a problem indeed; in my experience, this is basically used to clean up messy data. I agree that they have no place in the constants file and are better suited to some sort of config (and therefore one single folder like climada/resources). The index logic is very vulnerable to changes in the files anyways, so you'd want a proper key-based mapping.

In the case of the natural earth dataset, I would even propose to fork their current version (they don't iterate much) and insert/correct the metadata we need directly into the dataset, then pull the data from our own repo instead of their official mirror.

[Evelyn-M]

Discussion that happened via e-mail; added for the sake of completeness:

@mmyrte : As I suggested on the github issue on constants, local config files should follow (at least on *NIX) the convention of having a system-wide config in /usr/local/etc/climada.conf, superseded by first ~/.climada.conf and then $pwd/climada.conf - these could then be used to indicate a local "scratch directory" for climada. I personally am more in favour of having ~/climada then .climada, seeing as people will probably also want to access these files from e.g. GIS software instead of being invisible to the average user.

@emanuel-schmid : These are my objections: The system wide configuration in /usr/local/etc/climada.conf seems a bit odd to me. Firstly, that’s not really a canonical path. As far as I can remember, I have never seen a single file there. And secondly, what’s the point in a system wide configuration when it’s not even possible to install climada outside of a virtual environment? I think we can skip it and just have these: first place to look at is pwd, then home, at last default immutable config in the package folder. Having different names for config files as you suggest, climada.conf in pwd .climada.conf elsewhere is actually a pattern that is quite commonly used. But, frankly, I always found it annoying. Why not have one name and stick to it regardless of the location? The leading dot of the suggested data directory .climada makes it arguably somewhat harder for the less savvy user to access their files, agreed. But – it has also its advantages: it is a sign that the path is not arbitrarily chosen but embedded in some context and that one better don’t move the thing around carelessly. So, please convince me, I’m happy to hear counter arguments.

[mmyrte]

To add my reply from that mail thread:

Point taken that it's probably only going to be installed in a virtual environment. […] No global config outside the package then. (I guess /usr/local/etc is primarily used for BSD systems in that case...) The rationale for naming it .climada.conf in the homedir would be to allow technically savvy users to have a baseline config. Since a per-project configuration in a project folder should also be accessible to people who don't know about dotfiles, I think allowing both $pwd/.climada.conf and $pwd/climada.conf might be sensible. The rationale for having a visible global data directory to me is that there are going to be significant amounts of geodata in there; again, assuming that most of the students from the course don't know about dotfiles and dotfolders, they might be confused about what's eating up potentially gigabytes of storage. I've only ever seen dotfolders used for small configs and templates and such. If the folder were indeed moved, the programme in general is robust enough that necessary files are simply downloaded again. […]

Gabriela implemented the config reader so that a climada.conf in the working directory gets read; fallback is climada/conf/defaults.conf. Maybe we can just leave it at that. I imagine that if a configuration is used, it'll be on a project basis. Maybe I'm wrong in this, but I imagine that not many people have more than a handful of concurrent climada projects on their machine.

[emanuel-schmid]

There is a new feature branch 'config_conventions' introducing a change in the configuration policy of CLIMADA, particularly addressing external resources and file locations.

[emanuel-schmid]

Principles

• the default user must not be affected
• a super user may get involved by editing the config file(s)
• developers will have to distinguish between constants and configs
• test and ordinary use of the library may have different configs
• all "external resources" must go to config
  • external urls. (Although they are i.g. inert it's possible that a particular service stops and another host takes over.)
  • data directories.
  • infrastructure settings. e.g,, memory boundaries, number of workers for parallel processing.
  • logging settings.
• all "internal resources" (anything else) don't have to go to config -> must not go to config.

[mmyrte]

@emanuel-schmid I think we may close this issue, no? If there needs to be more discussion about specificly migrating constants to settings, that may be better placed in a new issue.