Hand FIM improvements - Githubissues

NOAA-OWP / hydrovis

Other

12 stars 9 forks source link

Hand FIM improvements #944

Closed groutr closed 3 weeks ago

groutr commented 3 weeks ago

The biggest change is selective column loading for csv files. This could potentially significantly reduce memory usage.

The rest of the changes are relatively minor.

np.where(condition) is an inefficient way of masking an array. It's about 2x slower and allocates an extra array.
sorting values is not an efficient way to find the max. ...and other minor fixes that simplify the code.

RobHanna-NOAA commented 3 weeks ago

We might want to double check which fields from the that table show up in the UI. There may be UI columns that the field want to see. Update here on what you find as I can take it back to FIM-Dev and find out why they are submitting those columns to HV.

groutr commented 3 weeks ago

I'm not sure I understand what you mean. The code is not discarding any columns that were not already being discarded.

RobHanna-NOAA commented 3 weeks ago

doh.. on nothing. I am a dork. I was mis-reading the FIM-dev input dataset. :)