search

djlampert / PyHSPF

Python extensions to the Hydrological Simulation Program in Fortran (HSPF), including classes for gathering input data, building input files, performing simulations, postprocessing results, calibrating hydrology process parameters, and forecasting climate and land use change effects on water resources
Other
83 stars 51 forks source link

Overflow issue in wdm #13

Closed scalet98 closed 4 years ago

scalet98 commented 4 years ago

Hello ! I am Jiheon Lee who is a Ph.D. candidate. in South Korea.

Briefly, I found OverflowError in reading WDM. Even though you consider only 1 year for test folder files, longer duration is the case mostly in the scope of HSPF modeling.

Only way to tackle this issue is shortening the data length? Or do you have any other suggestions to resolve this problem? An attached file is a picture of error screen.

Thank you for reading the issue.

106334398_297296768079372_8319878530441935662_n

djlampert commented 4 years ago

It looks like this has something to do with the value of (end - start) / len(CLOU), which is a timedelta object. You could work around this perhaps by just manually providing the timedelta (1 hour, 1 day, etc.). I don't see this as being a pyhspf problem. The error results when a timedelta is too large in Python.

https://docs.python.org/2.4/lib/datetime-timedelta.html

It seems maybe the value above is over 999999999 days = 2.7 million years, which would be a very large time step size. I can't see why you would need a 2.7 million year simulation. Without seeing more of the code, I cannot really diagnose this any further. You will have to figure out why the delta is so large.

scalet98 commented 4 years ago

Hello ! Lampert, I applied my WDM dataset with your test01 code. It sounds strange that I simulated 2.7 million years ...

Actually, it is not daily but hourly dataset. Sometimes we use the hourly weather component dataset for more detailed modeling. And I am considering the climate change scenario, so all dataset is from 1999 to 2099. I did know what the code said. But I am questioning whether the code support hourly dataset or not. And I though your program mainly focused on the daily dataset for short period. That's why this problem is coming up.

Thanks for your comments.

Jiheon.

It looks like this has something to do with the value of (end - start) / len(CLOU), which is a timedelta object. You could work around this perhaps by just manually providing the timedelta (1 hour, 1 day, etc.). I don't see this as being a pyhspf problem. The error results when a timedelta is too large in Python.

https://docs.python.org/2.4/lib/datetime-timedelta.html

It seems maybe the value above is over 999999999 days = 2.7 million years, which would be a very large time step size. I can't see why you would need a 2.7 million year simulation. Without seeing more of the code, I cannot really diagnose this any further. You will have to figure out why the delta is so large.

djlampert commented 4 years ago

HSPF supports time steps as small as 1 minute. A 100-year hourly time series should not cause problems. But you must understand that the issue you have appears to be related to your code and usage of timedelta objects in Python, not the PyHSPF software. You state that you used one of the PyHSPF examples. That may be true, but it seems that you modified the example in such a way that you broke some rule in Python. The error message you are getting indicates that you tried to create a "time difference" of 2.7 million+ years. You will have to troubleshoot to understand why this happened.

scalet98 commented 4 years ago

Firstly, I do not want to judge your code. If you felt uncomfortable I am sorry for that.

However, I am a HSPF user and modeler to simulate the water quality in Korea as well. So, it sounds very unpleasant with the sentence, "I can't see why you would need a 2.7 million year simulation."

And as you said, the code from your Test01"(end - start) / len(CLOU)" is problem.. Calculating unit is different from daily to hourly. And every single code is not a problem but I think you need to understand the user's behavior when you are designing the codes. You might consider every wdm files as a daily input format.. But it is actually not the case as I have attached bleow. Your test code works well with short period with my WDM file as well. But it is not work with longer period as I announced earlier due to the code. ( So it was not your pyhspf problem but test code problem.) So '999999999 days = 2.7 million years error' comes from your code, not by my WDM dataset.

And I solved the issue, so you don't need to mind it. As a learner, I just wanted to apply my case with your pyhspf. Basically, this wdm file used to work well with HSPF fortran version.
So that is why I was writing this memo here in the first place as you requested in the first page(readme.md) of your GitHub. "Please report any issues encountered with the compilation, installation or running the example files."

But thanks anyway.

106640213_578769442836190_6924984849271962400_n

HSPF supports time steps as small as 1 minute. A 100-year hourly time series should not cause problems. But you must understand that the issue you have appears to be related to your code and usage of timedelta objects in Python, not the PyHSPF software. You state that you used one of the PyHSPF examples. That may be true, but it seems that you modified the example in such a way that you broke some rule in Python. The error message you are getting indicates that you tried to create a "time difference" of 2.7 million+ years. You will have to troubleshoot to understand why this happened.

djlampert commented 4 years ago

"Basically, this wdm file used to work well with HSPF fortran version. So that is why I am curious about."

But the issue that is causing your code to crash has nothing to do with HSPF! It appears to me (and I have very little to go on here other than what you just posted) that you are not very familiar with dates and times in Python. Here is a tutorial: https://www.dataquest.io/blog/python-datetime-tutorial/. I suggest you practice going through this. If you understand dates, times, and timedeltas, you can perhaps find the issue.

This page is here to raise issues with the software. The error you posted does not appear to be an issue with the software. If you want to ask questions, it is probably better to just send me an email and provide more details, including your code. I would be happy to try to help you resolve your issue. By posting here, all 24 followers get an email message and it gives the appearance that the software needs to be updated to correct some kind of error.

scalet98 commented 4 years ago

Firstly, I do not want to judge your code. If you felt uncomfortable I am sorry for that.

However, I am a HSPF user and modeler to simulate the water quality in Korea as well. So, it sounds very unpleasant with your sentence, "I can't see why you would need a 2.7 million year simulation."

After I browsing it, as you said, the code from your Test01"(end - start) / len(CLOU)" is problem.. Calculating unit is different from daily to hourly. And every single code is not a problem but I think you need to understand the user's behavior when you are designing the codes. You might consider every wdm files as a daily input format.. But it is actually not the case as I have attached bleow. Your test code works well with short daily or hourly period with my WDM file as well. But it is not work with longer hourly period as I announced earlier due to the code. ( So it was not your pyhspf problem but your test code problem.) So '999999999 days = 2.7 million years error' comes from your code, not by my WDM dataset.

And I solved the issue, so you don't need to mind it. As a learner, I just wanted to apply my case with your pyhspf. Basically, this wdm file used to work well with HSPF fortran version.
So that is why I was writing this memo here in the first place as you requested in the first page(readme.md) of your GitHub. "Please report any issues encountered with the compilation, installation or running the example files."

But thanks anyway.

Jiheon

106640213_578769442836190_6924984849271962400_n

HSPF supports time steps as small as 1 minute. A 100-year hourly time series should not cause problems. But you must understand that the issue you have appears to be related to your code and usage of timedelta objects in Python, not the PyHSPF software. You state that you used one of the PyHSPF examples. That may be true, but it seems that you modified the example in such a way that you broke some rule in Python. The error message you are getting indicates that you tried to create a "time difference" of 2.7 million+ years. You will have to troubleshoot to understand why this happened.

"Basically, this wdm file used to work well with HSPF fortran version. So that is why I am curious about." But the issue that is causing your code to crash has nothing to do with HSPF! It appears to me (and I have very little to go on here other than what you just posted) that you are not very familiar with dates and times in Python. Here is a tutorial: https://www.dataquest.io/blog/python-datetime-tutorial/. I suggest you practice going through this. If you understand dates, times, and timedeltas, you can perhaps find the issue. This page is here to raise issues with the software. The error you posted does not appear to be an issue with the software. If you want to ask questions, it is probably better to just send me an email and provide more details, including your code. I would be happy to try to help you resolve your issue. By posting here, all 24 followers get an email message and it gives the appearance that the software needs to be updated to correct some kind of error.