GLM import times out on huge (multi-MB) glms

[dpinney]

TODO

• Switch glm import to background instead of realtime.

STEPS TO REPRODUCE

1. Import the IEEE 8500 test feeder in production. (See @dpinney for the glm of that feeder.)
2. If connection is sufficiently slow, the conversion times out.

Issue is that we do GLM import in a web server process because no one has huge .glm files. Quick fix is to do the import in a separate process like the Cyme and Windmil imports.

[dpinney]

Mannan assigned.

[mannanj]

Getting problems with pre-existing code. Is the file "IEEE_8500_with_schedules.glm" the same file that generated the above issue? If so, I get an unexpected issue when generating the list tree structure where its attempting to pop an empty list (see attached screenshot).

I applied a fix on lines 298 and 299 of feeder.py:

if len(guidStack) > 1: guidStack.pop()

It at least let me proceed with testing the GLM, but I am sure the fix isn't quite robust since it is oversimplifying a part of the GLM parsing function (which I know very little about & has a lot going on).

[mannanj]

With that being said the GLM import now works in the background, it's waiting for it to be checked if it works in production.

The physical view of the feeder doesn't look quite right though. Not sure if this was how it is supposed to look, but this seems like more of an issue on the feeder.py side, than from import times.

[dpinney]

fix on lines 298 and 299 of feeder.py

This sounds like a pretty safe change. Let's keep it for now. feeder._tests() still passing? The parsing functions in feeder.py are very complicated, which is not the best, but making them clearer will be very hard.

checked if it works in production

I'll push today and try it out.

[dpinney]

The physical view of the feeder doesn't look quite right though

Oh, yeah, the glm has no geodata so it will look weird. That's expected.

[dpinney]

When I import a number of GLMs with the new code I see zero data showing up in gridEdit.

[dpinney]

The problem was here: 33f1fbe8a95a8321204b44687db660ea4b17f635 I fixed it and added a test here: 58c0212acdd82e04a71b7916184b5b9ada42e2b6

[dpinney]

Actually, the problem was that defensive code if len(guidStack) > 1: guidStack.pop() broke all glm imports. I added a better version in 40ab7bac9df5ffb60ff218cc75245b6d351f3e22.

[mannanj]

Good catch, is this working now?

[dpinney]

With 8500? No.

[mannanj]

We will not support the 8500 glm due to it's unofficial nature.