search

HydrologicEngineeringCenter / HEC-FDA

Hydrologic Engineering Center - Flood Damage Analysis
MIT License
11 stars 2 forks source link

Saving histograms is taking a very long time #687

Closed codymccoy closed 1 year ago

codymccoy commented 1 year ago

I'm guessing there must have been some new code added because this didn't used to take so long. I think the issue is the size of the bin counts. In the muncie study there are 420,000 bins that are being written out. image

At my current rate, this is going to take like 20 minutes to save.

codymccoy commented 1 year ago

I wonder if part of the slow down is that these are being added as attributes and not as their own xelement? I will test that and see if it saves faster.

codymccoy commented 1 year ago

image This change made the saving basically instantaneous. The only issue is that the sqlite viewer bogs down and freezes when i try to verify the results. I am going to try to write all the values out as a single string and see if that helps.

rnugent3 commented 1 year ago

Thanks for looking into this, @codymccoy. Any changes you make to Histogram.cs - would you mind also changing to ThreadsafeInlineHistogram.cs?

codymccoy commented 1 year ago

I think this version is the best. I see no reason to write out the iteration value "i". It seems to be fast to write AND pretty fast to open the xml in sqlite reader. I convert the list of bincounts into a single comma separated value string. image

the updated reading logic: image

I will remove the dead code and add this logic to the threadsafe histogram.

codymccoy commented 1 year ago

This is fixed in branch save_checkbox_state.