search

SEED-platform / seed

Standard Energy Efficiency Data (SEED) Platform™ is a web-based application that helps organizations easily manage data on the energy performance of large groups of buildings.
Other
110 stars 55 forks source link

Program is significantly slower importing files -- optimization needed #2516

Closed RDmitchell closed 1 year ago

RDmitchell commented 3 years ago

Describe the bug Several cities using SEED for benchmarking compliance has noticed that the program is taking significantly longer to import files, to the point that they think the program has crashed (see #2515 )

Expected Behavior Import, map and match files in a reasonable amount of time.

Actual Behavior see Describe the Bug

Steps to Reproduce Import a file that has 2000+ records into a cycle that has 2000+ existing records, and at least 1 other cycle.

Instance Information Instance: LBNL production server SHA: e8d5cde48

Additional Context Most cities using that latest SEED on the LBNL instance are reporting this

adrian-lara commented 3 years ago

The biggest pain point seems to be the start_system_matching_and_geocoding endpoint. I'd suspect that initial first steps would be to confirm that, then identify the slowest queries within the underlying task.

RDmitchell commented 3 years ago

is geocoding a significant time sink?

Since we have the option now to turn geocoding off in Settings, would turning if off speed things up? I'm not sure all of our current users are interested in geocoding (although I could be wrong).

github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity within 60 days. It will be closed if no further activity occurs. Thank you for your contributions.

isalanglois commented 2 years ago

@RDmitchell Is this still relevant?

RDmitchell commented 2 years ago

@isalanglois -- I suspect it is since no work was done on it. You could put it on the new board, in the Test column, assign it to me, and if it is still a problem we can set it to To Do. How does that sound?

isalanglois commented 2 years ago

@RDmitchell That's great, thanks for the quick follow-up!

RDmitchell commented 1 year ago

I imported 4500 records (and 45 fields, so not as large as an ESPM file) into a new organization. From the initial import to the records being added to the Property table took approximately 7 minutes, which seems pretty reasonable.

The bulk of the time, probably more than half of it, was step 6/6, matching / merging

The geocoding time was trivial in this particular case

I am going to close this issue and we can always make another one if we find a file that takes a very long time to import