@lbross I finished up the loadpourpoint management command and was able to load all the pourpoints from the geojson files I previously dumped. I did so by using this command (run from a directory containing pourpoint files): ls -1 . | xargs -L 1 -P 8 snodas loadpourpoint.
It was kinda slow; I didn't time it exactly, but it was probably around 10 minutes in total to load all 1385 pourpoints. It has a fair bit of overhead, in that it spawns a new python process and database connection for every pourpoint, which is rather inefficient. Also, Atlas is rather CPU constrained, so that didn't help.
I think that's okay though. I don't think we will often be loading many pourpoints at once. For testing you probably only need a handful, and a lot will simply slow down SNODAS raster imports.
Also be aware that it spit out a ton of logging warnings, as multiple processes can't all open the log file for writing at the same time (the whole logging thing might need some attention).
Anyway, just wanted to let you know this is working now. Feel free to use it as reference if you want to do your own implementation, or if you want we can merge this and you can work on other things. Or, for the time being, you could simply run it off this branch to get some pourpoints into your database for development.
Note that I also created a repo for the pourpoint files: https://github.com/PSU-CSAR/BAGIS-pourpoints. I started with it as private in case we have concerns about making that data public, or we decide this whole idea of mine to use geojson like this is not a good one. I'm hoping we have time to discuss at our next meeting. Either way, those files are there for reference if you need them.
@lbross I finished up the
loadpourpointmanagement command and was able to load all the pourpoints from the geojson files I previously dumped. I did so by using this command (run from a directory containing pourpoint files):ls -1 . | xargs -L 1 -P 8 snodas loadpourpoint.It was kinda slow; I didn't time it exactly, but it was probably around 10 minutes in total to load all 1385 pourpoints. It has a fair bit of overhead, in that it spawns a new python process and database connection for every pourpoint, which is rather inefficient. Also, Atlas is rather CPU constrained, so that didn't help.
I think that's okay though. I don't think we will often be loading many pourpoints at once. For testing you probably only need a handful, and a lot will simply slow down SNODAS raster imports.
Also be aware that it spit out a ton of logging warnings, as multiple processes can't all open the log file for writing at the same time (the whole logging thing might need some attention).
Anyway, just wanted to let you know this is working now. Feel free to use it as reference if you want to do your own implementation, or if you want we can merge this and you can work on other things. Or, for the time being, you could simply run it off this branch to get some pourpoints into your database for development.
Note that I also created a repo for the pourpoint files: https://github.com/PSU-CSAR/BAGIS-pourpoints. I started with it as private in case we have concerns about making that data public, or we decide this whole idea of mine to use geojson like this is not a good one. I'm hoping we have time to discuss at our next meeting. Either way, those files are there for reference if you need them.