search

earthlab / firedpy

A Python Command Line Interface for classifying fire events from the Collection 6 MODIS Burned Area Product.
MIT License
34 stars 20 forks source link

no events created with ignition_date after 2019 #42

Closed mikoontz closed 3 years ago

mikoontz commented 3 years ago

Hi all,

I just successfully cloned this to my machine and ran it successfully. Thanks for the great documentation in the README!

One thing I notice is that the latest ignition_date is 2019-12-31. This is true in both the daily polygons and in the event level polygons as well as in the .csv tables. I notice in your latest refresh on s3 that the same thing is true: no events for 2020 or 2021. Those burned area products do get incorporated into the FIRED output though, because the daily polygons do include daily polygons with dates up through April of 2020 (but no later).

I'm not totally sure where to start hunting for bugs on this, but it seems that there's something keeping the algorithm from initiating events that start January 1, 2020 or later.

maxwellCcook commented 3 years ago

@mikoontz thanks for the heads up. This is a new issue (we had previously built through 2020) so thanks for opening it up. I noticed this yesterday after we made some updates. I am starting by checking HDF file downloads from the UMD server.

CONUS events through 2020 can be found on the s3 at "s3://earthlab-mcook/FIRED/y2000to2020/update05122021/" - the GPKG files in there are the most recent. If you want the events with the HISDAC-US Built-up Property Records let me know.

mikoontz commented 3 years ago

Hi @maxwellCcook , thanks for looking into this! The ones in the folder you reference on s3 show the same behavior, as far as I can tell (they only go through 2019). I saw they didn’t go through 2020 which is what led me to try to build the product on my own.

maxwellCcook commented 3 years ago

@mikoontz I think it is just the daily file that does not go through 2020. Regardless, I think I found the bug in the landcover workflow and I may have resolved this issue. I ran without specifying landcover and the last ignition year is 2021. By chance, did you designate a landcover parameter in your run?

mikoontz commented 3 years ago

I did specify a landcover in my run. I used this line to run everything:

firedpy -proj_dir H:\dev\megafired_california\data\raw --shapefile -landcover_type 1 -daily yes
maxwellCcook commented 3 years ago

Got it. So I think I found the issue and have updated the logic. Re-running for CONUS and will confirm when it finishes. I will also upload that product to the S3. Thank you!

mikoontz commented 3 years ago

Nice! I am also re-running. Funnily, the latest ignition date is 2020-12-01 for the event scale polygons in the s3 folder you reference, which implies no new fires after the first of December? Maybe will be fixed with the updated landcover logic. In any case, I've pulled your latest changes and thank you very much for the (super) fast response!

edit: whoops, just realizing that the changes I pulled in were from yesterday and not the ones you just implemented. So I suspect my re-run won't be a good gauge as to whether this behavior persists.

maxwellCcook commented 3 years ago

I am not sure if the latest changes are reflected in the master yet. I am waiting for the build to complete before pushing them up.

No problem! I was already working on this stuff today and it's good to catch these bugs early.

mikoontz commented 3 years ago

Just keeping up with this since I've been testing it and maybe there's some value to sharing what I've found.

I pulled in your latest changes https://github.com/earthlab/firedpy/commit/9a585f828a8e8e7510a15171ff0311457defad13 and didn't see any change in behavior (the events still stop at an ignition date of 2019-12-31).

Notably, this was still the case even without specifying a landcover option. I saw during the processing that firedpy was still trying to add landcover info to the fires, so maybe that step is still trying to complete even if no landcover option is used in the call to firedpy?

I rolled the code back (git reset v0.0.2) to the latest released version and the same thing happens when trying to specify a landcover. BUT, when I run v0.0.2 without specifying a landcover, I get events that start all the way up to 2021-03-01.

maxwellCcook commented 3 years ago

@mikoontz testing on my end indicates that this is resolved in the most recent push. I was able to build events through 2021-03-01 with landcover attributes. Are you still seeing the issue on your end?

mikoontz commented 3 years ago

all looks good to me! I think we can safely close the issue. Thanks so much for your work on this!