Set FBA columns and column dtypes

USEPA / flowsa

Library that attributes resource use, waste, emissions, and loss to economic sectors

MIT License

24 stars 19 forks source link

Set FBA columns and column dtypes #362

Closed catherinebirney closed 1 year ago

catherinebirney commented 1 year ago

FBA generation only returns columns identified as "required: True" in schema.py, can also include optional columns, such as "Suppressed". Previously, all columns in schema.py where added to FBA.
All empty cells are now represented by np.nan (or 0) rather than a mix of np.nan and None

bl-young commented 1 year ago

were you finding issues in some cases? I'm not sure the best way to test this - looks reasonable to me.

Suppressed still shows up when needed?

catherinebirney commented 1 year ago

Suppressed does show up when we want it - so Suppressed is included in the EIA_MECS_Energy FBAs. But now a column of np.nan is not added for no reason.

There wasn't a real issue with the dtypes, but the FBAs we were generating had a mix of None and np.nan columns, so I changed it for consistency.

bl-young commented 1 year ago

ok, i still get confused since there is also the new flowby_config.yaml so we will have to consolidate at somepoint

catherinebirney commented 1 year ago

Yes, we do need to consolidate, but right now FBAs are still generated with our original method so that is the code I tweaked

bl-young commented 1 year ago

@catherinebirney this seems to have caused some issues with collapse_fbs_sectors() because the sectors are showing up as 'nan' strings instead of np.nan (i believe)

bl-young commented 1 year ago

fixed with https://github.com/USEPA/flowsa/commit/1c4a1e2c54f2dddb21fbe2b00069fad14d70bf4c