search

transportenergy / database

Tools for accessing and maintaining the iTEM model & historical databases
https://transportenergy.rtfd.io
GNU General Public License v3.0
24 stars 8 forks source link

Mode & Vehicle Type values missing when running historical.process() #69

Closed francescolovat closed 3 years ago

francescolovat commented 3 years ago

Hi iTEM colleagues,

I'm raising this issue since recent usages of process.historical(0) (T000) and process.historical(4) (T004) led to NaNs in the resulting dataframe columns Mode & Vehicle Type (for the former) and Vehicle Type in the case of the latter.

Values seem to be present in the .csv files, so this might come from the processing scripts, unless I'm missing something.

pic-1 pic-2

soniayeh commented 3 years ago

It seems like a code issue as you suggested. To verify that, I am attaching the final spreadsheet that Humberto generated from his local codes, which does not have the same issue that you raised.
fig

The spreadsheet I showed here does not have "nan." They are shown as ALL. But I am not sure how to fix the online code, though.

khaeru commented 3 years ago

@francescolovat good catch.

The best way to address this will be a consistency check, that gets run on the cleaning script for each of the upstream data sources.

khaeru commented 3 years ago

Thanks @francescolovat for opening #72. This will be closed by #71, which adds the consistency checks I mentioned:

The best way to address this will be a consistency check, that gets run on the cleaning script for each of the upstream data sources.

…along with other improvements, namely avoiding the use of easily-confused column names (either via the older ColumnName enumeration or the newer column_name() function) and simply using everywhere the consistent IDs of the SDMX dimensions introduced by #62. This has the advantage of associating the column labels to the SDMX concepts with the same IDs, and those can have long & verbose names and descriptions; longer than would be practical to stick in a column label.

francescolovat commented 3 years ago

Thank you, @khaeru!