Slow Imports

[BryanHurst]

Importing legacy xml is very slow.

When it completes, the page doesn't update so you have to watch the cmd window to know when it is done.

[missyschoenbaum]

Here are the expected steps

[BryanHurst]

Test was on Texas_nodepop.xml and TX_NAADSM_pop_yearling.xml

The entire import took exactly 20 minutes.

Reading the population took 8 minutes.
Reading parameters was also fairly quick.
Both of these seem to be CPU bound on a single core.

When we got to constructing the new scenario, this was maxed out on the HDD.
This happens in the "readParameters" function in xml2sqlite.py after the reading of the parameters file.
This whole function is not optimized. The multitude of xml seeks are killing the performance. This could be fixed to initially reading the xml file into a custom in-memory data structure. However, this improvement would take several weeks of work.

[BryanHurst]

Once the import was done, I was properly taken to the Scenario Description page

[missyschoenbaum]

@BryanHurst Does that mean our different environments are causing the differences?

[BryanHurst]

@missyschoenbaum Does the import take longer than 20 minutes for you?

[missyschoenbaum]

@BryanHurst Historically, 20 minutes was expected. On this build it has been more like an hour. I can run again for more exact time.

[BryanHurst]

Since I just saw 20 minutes, we'll say that is the goal.

Go ahead and try running the import again and see if it takes significantly longer than expected.

I'll run in production on my end and see if there is a noted difference.

[missyschoenbaum]

OK, have it going.

[missyschoenbaum]

Is there some portion of the import that would be dependent on internet quality?

[BryanHurst]

Just the initial startup when it checks for updates. Nothing else should rely on internet quality.
Note that I'm also on a really bad internet connection today.

[BryanHurst]

When I ran in production, the import took 21 minutes, so no significant difference.

[missyschoenbaum]

At 38 minutes, mine is at building airborne spread. Load pop took 8 minutes.

[BryanHurst]

I've confirmed that the memory error is fairly consistent on this scenario. It happens on the first run of just 20 iterations.

It is the same GSlice array allocation issue within the CEngine as we have seen before.

This is something that Neil would have to look into.

[missyschoenbaum]

I am getting this today (new). First time I used TX_NAADSM_para and TX_NAADSM_pop_yearling (that I modified the xml). It failed. I tried with the other pop file that I hadn't tinkered with, in case I messed something up. Getting this after about 40 minutes. Going to reboot, which I did earlier today before starting.

[BryanHurst]

@missyschoenbaum what version does the program say you are running currently?

Hopefully it is 3.5.10.19

[missyschoenbaum]

@BryanHurst I'm about 40 minutes into another import attempt. Will check when it wraps up.

start 11:55, still running 2:12 stopped it at 3:30, started again at 3:15 after reboot

[missyschoenbaum]

Still running 5pm.

[missyschoenbaum]

@BryanHurst yes I am running 3.5.10.19. Import did finished. I quit watching it at 5 and it was not finished.

[missyschoenbaum]

I'm trying HPAI MN now. Pop loaded quickly, within 3 minutes. Has been running 1hr 15 min at this point. UPDATE started 8:57 finished 10:39, no screen refresh after 10 minutes. Apply button is live and I can push it again. Manually hit save and got an updated screen.

[ConradSelig]

OK as per Bryan's request, I'm also running the imports through my build as well.

Just replacing the population, it took just short of 27 minutes. Using the "import legacy scenario" button, and importing both the population and parameters, the entire import took just under 11 minutes.

Interesting food for thought there. Hope this helps.

[missyschoenbaum]

@ConradSelig that's good to hear.

[BryanHurst]

@missyschoenbaum can you confirm in the ADSM settings window that your workspace folder is on your computer's internal HDD?

If so, my next question would be if you have any way to confirm if your HDD is starting to fail? Since the import is I/O bound by the disk, if it is taking several hours to import, it could be pointing to HDD failure.

If we can rule out hardware issues, then we can move forward trying to track down the differences on the two systems.

[missyschoenbaum]

@BryanHurst I'm running both app and workspace on the desktop.

It is not beyond possibility that my HDD is starting to fail. However, it was also slow for Sarah, but she is running a student hand-me-down. It was slow on my home PC, which is slow on other things. Let me see if I can find a better PC, I have a couple of options.

[missyschoenbaum]

Took longer than expected to get PC option. Import worked within 10 minutes on new PC. Sarah also tried and hers loaded in 30 minutes on older PC.

However, it would not complete 50 iterations without throwing error

[BryanHurst]

I'm having the same issue with not being able to complete any number of iterations over 20.

If @ndh2 is available to look at it in the next two days, that would be great. I'll follow up and see what we can get worked out.

[missyschoenbaum]

@BryanHurst I tried Beta just to check, and I am having it fail as well. I made a load test issue. Is there a way to step back through Beta and figure out when it last worked?

[BryanHurst]

There is. I'll start stepping through previous builds and see what I find.

[BryanHurst]

It is confirmed that legacy imports are slow and the import process can use some major refactoring. Moving this to the backlog.