zhaokg / Rbeast

Bayesian Change-Point Detection and Time Series Decomposition
208 stars 36 forks source link

calculations are incomplete in beast123 when using large array as input #22

Closed MaJiapei closed 6 months ago

MaJiapei commented 6 months ago

I found that beast123 can't calculate compeletly when the input array is big enough(number of row or column is bigger than 200)

`import numpy as np import Rbeast as rb import xarray as xr import matplotlib.pyplot as plt

ndvi3d = np.load("sub_r_ndvi.npz")['r_nvdi'] # (722, 1378, 38) array

metadata = lambda: None metadata.start = 1986 metadata.season = 'none' metadata.whichDimIsTime = 3 metadata.period = 1.0 extra = rb.args() extra.numThreadsPerCPU = 2 o = rb.beast123(ndvi3d, metadata, [], [], extra) plt.imshow(o.trend.cp[:, :, 0]) plt.show()`

the abolve code shows a plot like

11820

however when we decrease the number of row and column ndvi3d = ndvi3d[0:150, 0:150, :] the calculation can be completely like 8140

dirt commented 6 months ago

Jiapei, thanks for the feedback and information. To better pinpoint the problem, would you please help to share a sample ndvi array for you to debug? You can send it to @.***

Kaiguang

Get Outlook for iOShttps://aka.ms/o0ukef


From: Jiapei Ma @.> Sent: Tuesday, December 12, 2023 9:09:02 PM To: zhaokg/Rbeast @.> Cc: Subscribed @.***> Subject: [zhaokg/Rbeast] calculations are incomplete in beast123 when using large array as input (Issue #22)

I found that beast123 can't calculate compeletly when the input array is big enough(number of row or column is bigger than 200) `import numpy as np import Rbeast as rb import xarray as xr import matplotlib. pyplot as plt ndvi3d = np. load("sub_r_ndvi. npz")['r_nvdi']

I found that beast123 can't calculate compeletly when the input array is big enough(number of row or column is bigger than 200)

`import numpy as np import Rbeast as rb import xarray as xr import matplotlib.pyplot as plt

ndvi3d = np.load("sub_r_ndvi.npz")['r_nvdi'] # (722, 1378, 38) array

metadata = lambda: None metadata.start = 1986 metadata.season = 'none' metadata.whichDimIsTime = 3 metadata.period = 1.0 extra = rb.args() extra.numThreadsPerCPU = 2 o = rb.beast123(ndvi3d, metadata, [], [], extra) plt.imshow(o.trend.cp[:, :, 0]) plt.show()`

the abolve code shows a plot like

11820.jpg (view on web)https://urldefense.com/v3/__https://github.com/zhaokg/Rbeast/assets/26560891/06cc7876-08e0-4ab1-b654-1f3af328010e__;!!KGKeukY!0f9Mzvv8lRMkzBlkuztO3yMuhFa8LE2dmzQFC4t2_52_-t2GVQP7yy0PiqBj2YEQv7as2Yv2UaimBIhj33V0dkE3GNVd$

however when we decrease the number of row and column ndvi3d = ndvi3d[0:150, 0:150, :] the calculation can be completely like 8140.jpg (view on web)https://urldefense.com/v3/__https://github.com/zhaokg/Rbeast/assets/26560891/627a7b2c-9e3b-4c9d-91f8-d6b134c79d5e__;!!KGKeukY!0f9Mzvv8lRMkzBlkuztO3yMuhFa8LE2dmzQFC4t2_52_-t2GVQP7yy0PiqBj2YEQv7as2Yv2UaimBIhj33V0dis0DRBQ$

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/zhaokg/Rbeast/issues/22__;!!KGKeukY!0f9Mzvv8lRMkzBlkuztO3yMuhFa8LE2dmzQFC4t2_52_-t2GVQP7yy0PiqBj2YEQv7as2Yv2UaimBIhj33V0dqFLMRAs$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AMCAGWQB4HEXXFMNMJHQICLYJEZ65AVCNFSM6AAAAABASRTG72VHI2DSMVQWIX3LMV43ASLTON2WKOZSGAZTQOJWGUZTANQ__;!!KGKeukY!0f9Mzvv8lRMkzBlkuztO3yMuhFa8LE2dmzQFC4t2_52_-t2GVQP7yy0PiqBj2YEQv7as2Yv2UaimBIhj33V0dqPr9rw-$. You are receiving this because you are subscribed to this thread.Message ID: @.***>

MaJiapei commented 6 months ago

Thanks for your reply, the data has been sent to your e-mail, because the data is relatively large, if there is any problem with decompression, please go to Google Driver https://drive.google.com/file/d/17nWL0pODMbBipglM7RnPYXCiA-l0ExG9/view

dirt commented 6 months ago

Got it. Thanks. I am at the AGU meeting this week and will take a look at the data later this week—sometime this Friday.

kai

Get Outlook for iOShttps://aka.ms/o0ukef


From: Jiapei Ma @.> Sent: Tuesday, December 12, 2023 10:50:55 PM To: zhaokg/Rbeast @.> Cc: Zhao, Kaiguang @.>; Comment @.> Subject: Re: [zhaokg/Rbeast] calculations are incomplete in beast123 when using large array as input (Issue #22)

Thanks for your reply, the data has been sent to your e-mail, because the data is relatively large, if there is any problem with decompression, please go to Google Driver https: //drive. google. com/file/d/17nWL0pODMbBipglM7RnPYXCiA-l0ExG9/view

Thanks for your reply, the data has been sent to your e-mail, because the data is relatively large, if there is any problem with decompression, please go to Google Driver https://drive.google.com/file/d/17nWL0pODMbBipglM7RnPYXCiA-l0ExG9/view

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/zhaokg/Rbeast/issues/22*issuecomment-1853358430__;Iw!!KGKeukY!2nPU0mATlKbj95stBKc0klr_JeYY29mK4nJofueuDOtYw5P-35ZfLa6ZnXxVNuFo5OsPLfYAQ-wpcYsYtKSHBcFG9sd2$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AMCAGWUSUXC5U345OH52XRTYJFF47AVCNFSM6AAAAABASRTG72VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJTGM2TQNBTGA__;!!KGKeukY!2nPU0mATlKbj95stBKc0klr_JeYY29mK4nJofueuDOtYw5P-35ZfLa6ZnXxVNuFo5OsPLfYAQ-wpcYsYtKSHBZq6LTT5$. You are receiving this because you commented.Message ID: @.***>

zhaokg commented 6 months ago

Dear Jiapei,

Sorry for the long waiting, if any. I just posted a new version to PyPi, which can be installed via via "pip install Rbeast=0.1.18". I tested it on my machine. It fixed the problem.

Here is what happened: beast123 uses multiple threads to go over all the pixels. When there is a bad pixel (e.g., beast fails to fit for some reason-- when a message like "skip3 precValue=inf..." is printed out) occurring in one thread, then all the remaining pixels handled by the thread will be filled with invalid values. I fixed the problem by resetting the thread when bad pixels occur so that the good pixels can be succefully processed.

Thanks a lot for letting me know the problem. Your feedback is extremely valuable. Please let me know if there are any more problems.

Kaiguang

Thanks for your reply, the data has been sent to your e-mail, because the data is relatively large, if there is any problem with decompression, please go to Google Driver https://drive.google.com/file/d/17nWL0pODMbBipglM7RnPYXCiA-l0ExG9/view

zhaokg commented 6 months ago

Dear Dear Jiapei, I believe the issue has been resolved. Thanks again for letting me know this important bug. Now I willl close this ticket.

Kaiguang