[python] Run times-excel-reader on TIMES-NZ model

[siddharth-krishna]

This issue tracks progress towards running https://github.com/etsap-TIMES/times-excel-reader/ successfully on TIMES-NZ. (Opening the issue in this repo and not the other one as the latter is public.)

For me the first error is in process_flexible_import_table:

https://github.com/etsap-TIMES/times-excel-reader/blob/f6167631a36bb8d30f90ab1ffb235572342e19ab/times_reader/transforms.py#L586-L597

IndexError: index 0 is out of bounds for axis 0 with size 0

The error occurs when process = 'MINWODSUPOSWOD', and this is the offending excel table:

That process is not in veda_process_sets:

> [t for t in veda_process_sets['techname'].unique() if t.startswith('MINWOD')]
['MINWODWST00', 'MINWODWST01', 'MINWODSUPCUR00', 'MINWODWST02']

@olejandro , thoughts on how to fix this?

[olejandro]

Thanks @siddharth-krishna for looking into this! Looks like the process in question is not of the expected type. COST should normally be used with IRE processes, but MINWODSUPOSWOD is PRE.

[olejandro]

Opened #kanors-emr/Veda2.0-Installation/issues/16 to see whether to align this with Veda.

[olejandro]

@willcattoneeca got confirmation in the discussion of the issue above, that Veda 2 doesn't have the same behaviour in this respect. For TIMES-NZ model, this means that either:

• one should change the set of affected processes
• or change the attribute used to ACT_COST (in a separate table, otherwise IRE processes would be affected)

[willcattoneeca]

Thanks Olex, the timing is pretty good as there is a plan to migrate our configuration files to Veda 2 imminently.

@gg-eeca and @BakytzhanSuleimenov, could you please confirm that Olex's specification for what needs to change makes sense to you?

[BakytzhanSuleimenov]

Thanks @olejandro and @willcattoneeca, yes, sure, this seems like one of the small things that need to be corrected to migrate to Veda 2. I will take care of such things.

[BakytzhanSuleimenov]

Deat @willcattoneeca I have changed the type of the process to "MIN" (mining) from "PRE" (generic process) so with this, we don't need to change "COST" attribute as it works perfectly for such type of process.

[willcattoneeca]

Hi all, I have merged the changes from @BakytzhanSuleimenov, together with the suggestions from @olejandro , to main. Running the times-excel-reader on the files again, I see an error in the same place again, The table in question this time is And this time the issue seems to be that the "techname" column (in the dataframe df) contains all None values. An extremely naive attempt to fix this by adding a techname column to the table in the Excel sheet, with values copied from Pset_PN, as per this screenshot, doesn't help: Does anyone have any advice or thoughts?

[olejandro]

Thanks @willcattoneeca! Actually there should be only one column of that kind, i.e. either techname or pset_pn. The tool will raise an error in the future when both are present. I'll investigate this error.

[olejandro]

The tool should run on the model once #https://github.com/etsap-times/times-excel-reader/issues/123 is merged.

[willcattoneeca]

Can confirm that the tool ran to completion and produced three .dd files when run with --dd argument!

[olejandro]

Great it runs to completion now. Maybe we can use the *.dd files intended for MIRO development to work on the coverage of the tool?

[siddharth-krishna]

Good news, with the changes in https://github.com/etsap-TIMES/times-excel-reader/pull/132 I can run the tool and compare against the DD files:

72.0% of ground truth rows present in output (67385/93604), 9060 additional rows 

Here is the full output of the tool: nz-out.txt