search

UCHIC / iUTAHData

The iUTAH Modeling and Data Federation website - data.iutahepscor.org.
BSD 3-Clause "New" or "Revised" License
1 stars 1 forks source link

Re-write Data Parser work flow #89

Closed jcaraballo17 closed 6 years ago

jcaraballo17 commented 7 years ago

The Data Parser runs for more than 30 minutes, making a large quantity of unnecessary queries to the database.

fryarludwig commented 7 years ago

I became very familiar with this data parser, and can help you optimize it. I added a cache to the iUtahUtilities script, and that could be further improved by making smaller (and more) queries to the DB. Let me know if you want help on this.

jcaraballo17 commented 7 years ago

Got 6 minutes for all the data! @fryarludwig you can look at it and see if you can do more optimizations or clean things up a little bit. I'll test it in the sandbox a couple of times to make sure it doesn't break anything.

AmberSJones commented 7 years ago

I don't know how to test this... so if this is complete and functioning, you can go ahead and close it. Let me know if there IS something that I can do to test it.