search

OSeMOSYS / otoole

OSeMOSYS Tools for Energy
https://otoole.readthedocs.io
MIT License
23 stars 17 forks source link

[Bug]: `ReadStrategy._check_index_dtypes` not converting dtypes correctly when reading from CSV #167

Closed trevorb1 closed 1 year ago

trevorb1 commented 1 year ago

The Issue

When reading in CSV data, the indices are not being converted to the correct datatype. This in turn triggers the following pandas error, where 2014.0 will be replaced with whatever the incorrect value is. 

ValueError: invalid literal for int() with base 10: '2014.0'

This is only an issue for reading CSV data, I can not replicate it for MathProg or Excel formats.

Expected Behavior

If I have a dataframe with the incorrect data types, I would expect otoole to correct them for me via the ReadStrategy._check_index_dtypes(..) function.

For example, if my input datafile was

REGION YEAR VALUE
A 2020.0 10
A 2021.0 15
A 2022.0 20

I would expect this to go to this

REGION YEAR VALUE
A 2020 10
A 2021 15
A 2022 20

If the defenition of YEAR in the config file is: 

YEAR:
    dtype: int
    type: set

Steps To Reproduce

  1. Start with a valid set of CSV datafiles
  2. Change the year in any of the parameters to be a float (ie. change 2020 to 2020.0)
  3. Run the command otoole convert csv datafile <data> <data.txt> <config.yaml>

Log output

(otoole) trevorb1@DESKTOP-M23U3I0:~/repositories/otoole/trevor$ otoole -v convert csv datafile data data.txt config.yaml 
INFO:otoole.cli:Reading config from config.yaml
INFO:otoole.cli:Validating config from config.yaml
INFO:otoole.read_strategies:Looking for AccumulatedAnnualDemand
INFO:otoole.read_strategies:AccumulatedAnnualDemand is already in narrow form with headers ['REGION', 'FUEL', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for AccumulatedAnnualDemand
INFO:otoole.read_strategies:Looking for AnnualEmissionLimit
INFO:otoole.read_strategies:AnnualEmissionLimit is already in narrow form with headers ['REGION', 'EMISSION', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for AnnualEmissionLimit
INFO:otoole.read_strategies:Looking for AnnualExogenousEmission
INFO:otoole.read_strategies:AnnualExogenousEmission is already in narrow form with headers ['REGION', 'EMISSION', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for AnnualExogenousEmission
INFO:otoole.read_strategies:Looking for AvailabilityFactor
INFO:otoole.read_strategies:AvailabilityFactor is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'YEAR', 'VALUE']
INFO:otoole.read_strategies:Looking for CapacityFactor
INFO:otoole.read_strategies:CapacityFactor is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'TIMESLICE', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for CapacityFactor
INFO:otoole.read_strategies:Looking for CapacityOfOneTechnologyUnit
INFO:otoole.read_strategies:CapacityOfOneTechnologyUnit is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'YEAR', 'VALUE']
INFO:otoole.read_strategies:Looking for CapacityToActivityUnit
INFO:otoole.read_strategies:CapacityToActivityUnit is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'VALUE']
INFO:root:Checking datatypes for CapacityToActivityUnit
INFO:otoole.read_strategies:Looking for CapitalCost
INFO:otoole.read_strategies:CapitalCost is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for CapitalCost
INFO:otoole.read_strategies:Looking for CapitalCostStorage
INFO:otoole.read_strategies:CapitalCostStorage is already in narrow form with headers ['REGION', 'STORAGE', 'YEAR', 'VALUE']
INFO:otoole.read_strategies:Looking for Conversionld
INFO:otoole.read_strategies:Conversionld is already in narrow form with headers ['TIMESLICE', 'DAYTYPE', 'VALUE']
INFO:root:Checking datatypes for Conversionld
INFO:otoole.read_strategies:Looking for Conversionlh
INFO:otoole.read_strategies:Conversionlh is already in narrow form with headers ['TIMESLICE', 'DAILYTIMEBRACKET', 'VALUE']
INFO:root:Checking datatypes for Conversionlh
INFO:otoole.read_strategies:Looking for Conversionls
INFO:otoole.read_strategies:Conversionls is already in narrow form with headers ['TIMESLICE', 'SEASON', 'VALUE']
INFO:root:Checking datatypes for Conversionls
INFO:otoole.read_strategies:Looking for DAILYTIMEBRACKET
INFO:otoole.read_strategies:Checking set DAILYTIMEBRACKET
INFO:root:dtype does not match int for set DAILYTIMEBRACKET
INFO:otoole.read_strategies:Looking for DaysInDayType
INFO:otoole.read_strategies:DaysInDayType is already in narrow form with headers ['SEASON', 'DAYTYPE', 'YEAR', 'VALUE']
INFO:otoole.read_strategies:Looking for DaySplit
INFO:otoole.read_strategies:DaySplit is already in narrow form with headers ['DAILYTIMEBRACKET', 'YEAR', 'VALUE']
INFO:otoole.read_strategies:Looking for DAYTYPE
INFO:otoole.read_strategies:Checking set DAYTYPE
INFO:root:dtype does not match int for set DAYTYPE
INFO:otoole.read_strategies:Looking for DepreciationMethod
INFO:otoole.read_strategies:DepreciationMethod is already in narrow form with headers ['REGION', 'VALUE']
INFO:otoole.read_strategies:Looking for DiscountRate
INFO:otoole.read_strategies:DiscountRate is already in narrow form with headers ['REGION', 'VALUE']
INFO:otoole.read_strategies:Looking for DiscountRateStorage
INFO:otoole.read_strategies:DiscountRateStorage is already in narrow form with headers ['REGION', 'STORAGE', 'VALUE']
INFO:otoole.read_strategies:Looking for EMISSION
INFO:otoole.read_strategies:Checking set EMISSION
INFO:root:dtype does not match str for set EMISSION
INFO:otoole.read_strategies:Looking for EmissionActivityRatio
INFO:otoole.read_strategies:EmissionActivityRatio is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'EMISSION', 'MODE_OF_OPERATION', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for EmissionActivityRatio
INFO:otoole.read_strategies:Looking for EmissionsPenalty
INFO:otoole.read_strategies:EmissionsPenalty is already in narrow form with headers ['REGION', 'EMISSION', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for EmissionsPenalty
INFO:otoole.read_strategies:Looking for FixedCost
INFO:otoole.read_strategies:FixedCost is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for FixedCost
INFO:otoole.read_strategies:Looking for FUEL
INFO:otoole.read_strategies:Checking set FUEL
INFO:root:dtype does not match str for set FUEL
INFO:otoole.read_strategies:Looking for InputActivityRatio
INFO:otoole.read_strategies:InputActivityRatio is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'FUEL', 'MODE_OF_OPERATION', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for InputActivityRatio
INFO:otoole.read_strategies:Looking for MinStorageCharge
INFO:otoole.read_strategies:MinStorageCharge is already in narrow form with headers ['REGION', 'STORAGE', 'YEAR', 'VALUE']
INFO:otoole.read_strategies:Looking for MODE_OF_OPERATION
INFO:otoole.read_strategies:Checking set MODE_OF_OPERATION
INFO:root:dtype does not match int for set MODE_OF_OPERATION
INFO:otoole.read_strategies:Looking for ModelPeriodEmissionLimit
INFO:otoole.read_strategies:ModelPeriodEmissionLimit is already in narrow form with headers ['REGION', 'EMISSION', 'VALUE']
INFO:otoole.read_strategies:Looking for ModelPeriodExogenousEmission
INFO:otoole.read_strategies:ModelPeriodExogenousEmission is already in narrow form with headers ['REGION', 'EMISSION', 'VALUE']
INFO:otoole.read_strategies:Looking for OperationalLife
INFO:otoole.read_strategies:OperationalLife is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'VALUE']
INFO:root:Checking datatypes for OperationalLife
INFO:otoole.read_strategies:Looking for OperationalLifeStorage
INFO:otoole.read_strategies:OperationalLifeStorage is already in narrow form with headers ['REGION', 'STORAGE', 'VALUE']
INFO:otoole.read_strategies:Looking for OutputActivityRatio
INFO:otoole.read_strategies:OutputActivityRatio is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'FUEL', 'MODE_OF_OPERATION', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for OutputActivityRatio
INFO:otoole.read_strategies:Looking for REGION
INFO:otoole.read_strategies:Checking set REGION
INFO:root:dtype does not match str for set REGION
INFO:otoole.read_strategies:Looking for REMinProductionTarget
INFO:otoole.read_strategies:REMinProductionTarget is already in narrow form with headers ['REGION', 'YEAR', 'VALUE']
INFO:otoole.read_strategies:Looking for ReserveMargin
INFO:otoole.read_strategies:ReserveMargin is already in narrow form with headers ['REGION', 'YEAR', 'VALUE']
INFO:otoole.read_strategies:Looking for ReserveMarginTagFuel
INFO:otoole.read_strategies:ReserveMarginTagFuel is already in narrow form with headers ['REGION', 'FUEL', 'YEAR', 'VALUE']
INFO:otoole.read_strategies:Looking for ReserveMarginTagTechnology
INFO:otoole.read_strategies:ReserveMarginTagTechnology is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'YEAR', 'VALUE']
INFO:otoole.read_strategies:Looking for ResidualCapacity
INFO:otoole.read_strategies:ResidualCapacity is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for ResidualCapacity
INFO:otoole.read_strategies:Looking for ResidualStorageCapacity
INFO:otoole.read_strategies:ResidualStorageCapacity is already in narrow form with headers ['REGION', 'STORAGE', 'YEAR', 'VALUE']
INFO:otoole.read_strategies:Looking for RETagFuel
INFO:otoole.read_strategies:RETagFuel is already in narrow form with headers ['REGION', 'FUEL', 'YEAR', 'VALUE']
INFO:otoole.read_strategies:Looking for RETagTechnology
INFO:otoole.read_strategies:RETagTechnology is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for RETagTechnology
INFO:otoole.read_strategies:Looking for SEASON
INFO:otoole.read_strategies:Checking set SEASON
INFO:root:dtype does not match int for set SEASON
INFO:otoole.read_strategies:Looking for SpecifiedAnnualDemand
INFO:otoole.read_strategies:SpecifiedAnnualDemand is already in narrow form with headers ['REGION', 'FUEL', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for SpecifiedAnnualDemand
INFO:otoole.read_strategies:Looking for SpecifiedDemandProfile
INFO:otoole.read_strategies:SpecifiedDemandProfile is already in narrow form with headers ['REGION', 'FUEL', 'TIMESLICE', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for SpecifiedDemandProfile
INFO:otoole.read_strategies:Looking for STORAGE
INFO:otoole.read_strategies:Checking set STORAGE
INFO:root:dtype does not match str for set STORAGE
INFO:otoole.read_strategies:Looking for StorageLevelStart
INFO:otoole.read_strategies:StorageLevelStart is already in narrow form with headers ['REGION', 'STORAGE', 'VALUE']
INFO:otoole.read_strategies:Looking for StorageMaxChargeRate
INFO:otoole.read_strategies:StorageMaxChargeRate is already in narrow form with headers ['REGION', 'STORAGE', 'VALUE']
INFO:otoole.read_strategies:Looking for StorageMaxDischargeRate
INFO:otoole.read_strategies:StorageMaxDischargeRate is already in narrow form with headers ['REGION', 'STORAGE', 'VALUE']
INFO:otoole.read_strategies:Looking for TECHNOLOGY
INFO:otoole.read_strategies:Checking set TECHNOLOGY
INFO:root:dtype does not match str for set TECHNOLOGY
INFO:otoole.read_strategies:Looking for TechnologyFromStorage
INFO:otoole.read_strategies:TechnologyFromStorage is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'STORAGE', 'MODE_OF_OPERATION', 'VALUE']
INFO:root:Checking datatypes for TechnologyFromStorage
INFO:otoole.read_strategies:Looking for TechnologyToStorage
INFO:otoole.read_strategies:TechnologyToStorage is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'STORAGE', 'MODE_OF_OPERATION', 'VALUE']
INFO:root:Checking datatypes for TechnologyToStorage
INFO:otoole.read_strategies:Looking for TIMESLICE
INFO:otoole.read_strategies:Checking set TIMESLICE
INFO:root:dtype does not match str for set TIMESLICE
INFO:otoole.read_strategies:Looking for TotalAnnualMaxCapacity
INFO:otoole.read_strategies:TotalAnnualMaxCapacity is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for TotalAnnualMaxCapacity
INFO:otoole.read_strategies:Looking for TotalAnnualMaxCapacityInvestment
INFO:otoole.read_strategies:TotalAnnualMaxCapacityInvestment is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for TotalAnnualMaxCapacityInvestment
INFO:otoole.read_strategies:Looking for TotalAnnualMinCapacity
INFO:otoole.read_strategies:TotalAnnualMinCapacity is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'YEAR', 'VALUE']
INFO:otoole.read_strategies:Looking for TotalAnnualMinCapacityInvestment
INFO:otoole.read_strategies:TotalAnnualMinCapacityInvestment is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for TotalAnnualMinCapacityInvestment
INFO:otoole.read_strategies:Looking for TotalTechnologyAnnualActivityLowerLimit
INFO:otoole.read_strategies:TotalTechnologyAnnualActivityLowerLimit is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for TotalTechnologyAnnualActivityLowerLimit
INFO:otoole.read_strategies:Looking for TotalTechnologyAnnualActivityUpperLimit
INFO:otoole.read_strategies:TotalTechnologyAnnualActivityUpperLimit is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for TotalTechnologyAnnualActivityUpperLimit
INFO:otoole.read_strategies:Looking for TotalTechnologyModelPeriodActivityLowerLimit
INFO:otoole.read_strategies:TotalTechnologyModelPeriodActivityLowerLimit is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'VALUE']
INFO:otoole.read_strategies:Looking for TotalTechnologyModelPeriodActivityUpperLimit
INFO:otoole.read_strategies:TotalTechnologyModelPeriodActivityUpperLimit is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'VALUE']
INFO:otoole.read_strategies:Looking for TradeRoute
INFO:otoole.read_strategies:TradeRoute is already in narrow form with headers ['REGION', 'FUEL', 'YEAR', 'VALUE']
INFO:otoole.read_strategies:Looking for VariableCost
INFO:otoole.read_strategies:VariableCost is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'MODE_OF_OPERATION', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for VariableCost
INFO:otoole.read_strategies:Looking for YEAR
INFO:otoole.read_strategies:Checking set YEAR
INFO:root:dtype does not match int for set YEAR
INFO:otoole.read_strategies:Looking for YearSplit
INFO:otoole.read_strategies:YearSplit is already in narrow form with headers ['TIMESLICE', 'YEAR', 'VALUE']
INFO:root:Checking datatypes for YearSplit
INFO:otoole.read_strategies:Looking for AnnualEmissions
INFO:otoole.read_strategies:Looking for AccumulatedNewCapacity
INFO:otoole.read_strategies:Looking for AnnualFixedOperatingCost
INFO:otoole.read_strategies:Looking for AnnualTechnologyEmission
INFO:otoole.read_strategies:Looking for AnnualTechnologyEmissionByMode
INFO:otoole.read_strategies:Looking for AnnualVariableOperatingCost
INFO:otoole.read_strategies:Looking for CapitalInvestment
INFO:otoole.read_strategies:Looking for Demand
INFO:otoole.read_strategies:Looking for DiscountedSalvageValue
INFO:otoole.read_strategies:Looking for DiscountedTechnologyEmissionsPenalty
INFO:otoole.read_strategies:Looking for NewCapacity
INFO:otoole.read_strategies:Looking for NewStorageCapacity
INFO:otoole.read_strategies:Looking for NumberOfNewTechnologyUnits
INFO:otoole.read_strategies:Looking for ProductionByTechnology
INFO:otoole.read_strategies:Looking for ProductionByTechnologyAnnual
INFO:otoole.read_strategies:Looking for RateOfActivity
INFO:otoole.read_strategies:Looking for RateOfProductionByTechnology
INFO:otoole.read_strategies:Looking for RateOfProductionByTechnologyByMode
INFO:otoole.read_strategies:Looking for RateOfUseByTechnology
INFO:otoole.read_strategies:Looking for RateOfUseByTechnologyByMode
INFO:otoole.read_strategies:Looking for SalvageValue
INFO:otoole.read_strategies:Looking for SalvageValueStorage
INFO:otoole.read_strategies:Looking for StorageLevelDayTypeFinish
INFO:otoole.read_strategies:Looking for StorageLevelDayTypeStart
INFO:otoole.read_strategies:Looking for StorageLevelSeasonStart
INFO:otoole.read_strategies:Looking for StorageLevelYearStart
INFO:otoole.read_strategies:Looking for StorageLevelYearFinish
INFO:otoole.read_strategies:Looking for TotalAnnualTechnologyActivityByMode
INFO:otoole.read_strategies:Looking for TotalCapacityAnnual
INFO:otoole.read_strategies:Looking for TotalDiscountedCost
INFO:otoole.read_strategies:Looking for TotalTechnologyAnnualActivity
INFO:otoole.read_strategies:Looking for TotalTechnologyModelPeriodActivity
INFO:otoole.read_strategies:Looking for Trade
INFO:otoole.read_strategies:Looking for UseByTechnology
Traceback (most recent call last):
  File "/home/trevorb1/.local/bin/otoole", line 8, in <module>
    sys.exit(main())
  File "/home/trevorb1/repositories/otoole/src/otoole/cli.py", line 470, in main
    args.func(args)
  File "/home/trevorb1/repositories/otoole/src/otoole/cli.py", line 220, in conversion_matrix
    input_data, _ = read_strategy.read(args.from_path)
  File "/home/trevorb1/repositories/otoole/src/otoole/read_strategies.py", line 214, in read
    input_data = self._check_index(input_data)
  File "/home/trevorb1/repositories/otoole/src/otoole/input.py", line 392, in _check_index
    df = self._check_index_dtypes(name=name, config=details, df=df)
  File "/home/trevorb1/repositories/otoole/src/otoole/input.py", line 497, in _check_index_dtypes
    df.dropna(axis=0, how="all")
  File "/home/trevorb1/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 6226, in astype
    res_col = col.astype(dtype=cdt, copy=copy, errors=errors)
  File "/home/trevorb1/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 6240, in astype
    new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
  File "/home/trevorb1/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 448, in astype
    return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
  File "/home/trevorb1/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 352, in apply
    applied = getattr(b, f)(**kwargs)
  File "/home/trevorb1/.local/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 526, in astype
    new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
  File "/home/trevorb1/.local/lib/python3.8/site-packages/pandas/core/dtypes/astype.py", line 299, in astype_array_safe
    new_values = astype_array(values, dtype, copy=copy)
  File "/home/trevorb1/.local/lib/python3.8/site-packages/pandas/core/dtypes/astype.py", line 230, in astype_array
    values = astype_nansafe(values, dtype, copy=copy)
  File "/home/trevorb1/.local/lib/python3.8/site-packages/pandas/core/dtypes/astype.py", line 170, in astype_nansafe
    return arr.astype(dtype, copy=True)
ValueError: invalid literal for int() with base 10: '2014.0'
(otoole) trevorb1@DESKTOP-M23U3I0:~/repositories/otoole/trevor$

Operating System

Linux

What version of otoole are you running?

Fails on all of 1.0.0, 1.0.1, 1.0.2

Possible Solution

There are some tests written that use the following data

https://github.com/OSeMOSYS/otoole/blob/330627931f7315f6772a4c150654d9ec630aedb3/tests/test_input.py#L335-L351

https://github.com/OSeMOSYS/otoole/blob/330627931f7315f6772a4c150654d9ec630aedb3/tests/test_input.py#L362-L366

https://github.com/OSeMOSYS/otoole/blob/330627931f7315f6772a4c150654d9ec630aedb3/tests/test_input.py#L375-L378

With this following tests passing:

https://github.com/OSeMOSYS/otoole/blob/330627931f7315f6772a4c150654d9ec630aedb3/tests/test_input.py#L416-L425

I guess we are not explicitly checking that the conversion from floats to ints work okay. Rather it is checking the conversion of strings to ints. But in either case, I would expect it to identify the incorrect type. However, running otoole in debug mode (ie -vvv), we can see that it is totally skipping over identifying datatypes. The warning of DEBUG:otoole.input:Unable to set index on ... is raised for all parameters. Edit: this warning is because the df is already a multi-index, so its fine that it is being logged as is

DEBUG:otoole.input:Identified CapitalCost as a parameter
DEBUG:otoole.input:Unable to set index on CapitalCost
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'TECHNOLOGY': 'str', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input:                                 VALUE
REGION     TECHNOLOGY YEAR            
SIMPLICITY BACKSTOP1  2014.0  999999.0
           BACKSTOP2  2014    999999.0
           ETHPLANT   2014        25.0
           GRID_EXP   2014      4000.0
           HYD1       2014      4500.0
Traceback (most recent call last):
  File "/home/trevorb1/.local/bin/otoole", line 8, in <module>
    sys.exit(main())
  File "/home/trevorb1/repositories/otoole/src/otoole/cli.py", line 470, in main
    args.func(args)
  File "/home/trevorb1/repositories/otoole/src/otoole/cli.py", line 220, in conversion_matrix
    input_data, _ = read_strategy.read(args.from_path)
  File "/home/trevorb1/repositories/otoole/src/otoole/read_strategies.py", line 214, in read
    input_data = self._check_index(input_data)
  File "/home/trevorb1/repositories/otoole/src/otoole/input.py", line 392, in _check_index
    df = self._check_index_dtypes(name=name, config=details, df=df)
  File "/home/trevorb1/repositories/otoole/src/otoole/input.py", line 497, in _check_index_dtypes
    df.dropna(axis=0, how="all")
  File "/home/trevorb1/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 6226, in astype
    res_col = col.astype(dtype=cdt, copy=copy, errors=errors)
  File "/home/trevorb1/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 6240, in astype
    new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
  File "/home/trevorb1/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 448, in astype
    return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
  File "/home/trevorb1/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 352, in apply
    applied = getattr(b, f)(**kwargs)
  File "/home/trevorb1/.local/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 526, in astype
    new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
  File "/home/trevorb1/.local/lib/python3.8/site-packages/pandas/core/dtypes/astype.py", line 299, in astype_array_safe
    new_values = astype_array(values, dtype, copy=copy)
  File "/home/trevorb1/.local/lib/python3.8/site-packages/pandas/core/dtypes/astype.py", line 230, in astype_array
    values = astype_nansafe(values, dtype, copy=copy)
  File "/home/trevorb1/.local/lib/python3.8/site-packages/pandas/core/dtypes/astype.py", line 170, in astype_nansafe
    return arr.astype(dtype, copy=True)
ValueError: invalid literal for int() with base 10: '2014.0'

Anything else?

Here is the function that is failing

https://github.com/OSeMOSYS/otoole/blob/330627931f7315f6772a4c150654d9ec630aedb3/src/otoole/input.py#L464-L506

trevorb1 commented 1 year ago

Okay, ya. After doing some tests, if the value is a string it can be converted okay to whatever. So it seems like floats -> ints is the cause of the error.