FVS: post-processing audit, cut trees not being recognized for one Rx package

[Cat-Ducat-USDA]

The project in question is located in the temp 30 days folder, called cwi20240808.7z

After appending the FVSOut data in BioSum, an error message popped up saying that no trees were cut for one of the prescription packages (Rx 872, run name FVSOUT_EC_P872-872-872-872-872). However, trees were definitely cut. We looked at the cut list and tree list from the FVSOut.db and found cut trees from that prescription. Additionally, there are 173,000+ cut tree records in the BioSum table shown after the data was appended. We checked that packages/runs were appropriately named, verified there were trees cut, but for whatever reason BioSum is not recognizing those cut trees in the post-processing audit. @lbross @druepdx @jsfried

[jsfried]

I worked some on summarizing what I was seeing here on Friday but didn't finish, as I hoped to dig deeper on the weekend but didn't get around to it. In case it is useful, this is what I was finding on this cwi project: investigation of cwi project FVS out load problems.docx

@lbross @druepdx @Cat-Ducat-USDA

[jsfried]

Is it probably a function of the FVS output load problems already described but when we can get into processor (after clearing the Exception message), the only available package to process reports to be 760 when in fact the only package with trees in the CutTree table is 761.

@lbross

[lbross]

Working with Cat's project, I was able to replicate the initial error, that is no trees making it into the BioSum FVS_CutTree table for RxPackage 872. After some digging around, I found that there were a bunch of tables in the /fvs/data/FVSOUT_TREE_LIST.db that shouldn't be there including all of the sequence number matrices. The only two tables that should be in this .db are the FVS_CutTree and (optionally) FVS_InForestTree. I deleted all of the extra tables and re-ran FVSIn for RxPackage 872. It correctly loaded all of the cut trees into FVS_CutTree and I was able to run the post-processing audit. It is a mystery how these extra tables got into this database. The only potential culprit I can think of is the FVS_InForest table task. I'm running that now. It's taking a long time because there are so many trees. But so far those extra tables aren't showing up. I'm stuck from working further along in the workflow until that completes. But I wanted to share what I found to see if you wanted to try it to get a step further.

[lbross]

It appears that this project is too large to successfully create the FVS_InForest table. We still use MS Access to pull information out of the cond and plot tables when manipulating the trees for FICS. I just got this error about running up against the 2GB MS Access limit when trying to create the FVS_InForest table. I'm going to drop trying to find the origin of the errant tables and pivot to seeing if Processor will work for this project.

[Cat-Ducat-USDA]

Oh okay, yes, that is the same error message I got when I first tried to do the FVS_InForest table. That explains that I suppose but not how those tables got there.

From: Lesley @.> Sent: Monday, August 12, 2024 11:52 AM To: USFS-PNW/Fia-Biosum-Manager @.> Cc: Ducat, Caitlin - FS, OR @.>; Mention @.> Subject: Re: [USFS-PNW/Fia-Biosum-Manager] FVS: post-processing audit, cut trees not being recognized for one Rx package (Issue #342)

It appears that this project is too large to successfully create the FVS_InForest table. We still use MS Access to pull information out of the cond and plot tables when manipulating the trees for FICS. I just got this error about running up against the 2GB MS Access limit when trying to create the FVS_InForest table. I'm going to drop trying to find the origin of the errant tables and pivot to seeing if Processor will work for this project. image.png (view on web)https://github.com/user-attachments/assets/7e2072a1-8bc9-4e11-9224-db2caed3ed60

- Reply to this email directly, view it on GitHubhttps://github.com/USFS-PNW/Fia-Biosum-Manager/issues/342#issuecomment-2284697753, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BJ57OYLPFV3CJBNA2IHKPCDZRD75XAVCNFSM6AAAAABMHOO3SOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBUGY4TONZVGM. You are receiving this because you were mentioned.Message ID: @.***>

This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.

[lbross]

When I look at Cat's project in Processor, it shows me all of the rxpackages regardless if they have trees in FVS_CutTree. My recollection is that it should only be rxPackages with CutTree records.

[jsfried]

That is true- Processor should only show packages with valid cut tree records-- without those, there is nothing to process (in terms of costs, volume summaries or revenues)

[lbross]

I am able to run Processor successfully for rxPackages 761 and 872 once they have trees in the FVS_CutTree table. Processor is currently showing all packages whether or not they have records in the FVS_CutTree tables. I will fix this, but am curious why Jeremy is only showing package 760. I can't recreate this display with Cat's posted project. One word of warning: If you try to run a package that doesn't have records in FVS_CutTree you won't be able to run packages with records in FVS_CutTree without closing and reopening Processor. I have already fixed this. When you click the Run button for the second package, it just doesn't do anything. I'm working on cleaning up the package listing on the Processor screen to only show packages with cut trees. We can decide later if you want another .msi with this fix. I think this should give you all you need to move forward with work on this project, but let me know if I'm missing anything. You need to:

1. Delete the extraneous tables from /fvs/data/FVSOUT_TREE_LIST.db. This should let you run FVS Out for 872 and get trees in the CutList
2. Run Processor only for 761 and 872. Again, trying to run Processor for a package without cut trees will keep it from running correctly until you close and reopen the screen

[jsfried]

Thanks very much! Is it also OK to just delete the PREPOST and FVSOUT_TREE_LIST DBs and load all (both) packages from scratch? @lbross @catducat

[jsfried]

I do wonder whether this warning (Red zone database size) is indicative of issues in this CWI project, and I am surprised that there are too many trees to make an InForest Tree table given that there are only about 1000 plots in this project-- much fewer than in the massive westside projects that Sebastian ran for PN and for WC.

Since it is a db not an accdb, there isn't really a 2GB limit on the FVS_Audits.db but why DOES it get so large? When I ran a vacuum on it in an earlier run that spewed this red line, it shrank to be tiny.l

[lbross]

I don't have an answer about the size of FVS_Audits.db. I was unfamiliar with the vacuum command until I just looked it up. It looks like SQLite manages data storage differently from Access and there are multiple reasons why vacuum reduces the database size. You can delete all of the tables out of FVS_Audits.db and see if that helps. BioSum will recreate the ones it needs.

[jsfried]

I was able to get past the exception (unhandled) by deleting the first Rxpackage in the Rxpackages table in the FVS_MASTER.MDB -- that allowed me into Processor Rule Defs and ultimately the Run dialog, where I found an odd thing given that only 2 packages have data loaded at this point (761 and 872):

Why are there tree counts for all the the packages? Perhaps that points to the cause of the iissue. And it may explain why having no trees for the first package encountered in the table (760) had such terminal consequence.

@lbross @druepdx @Cat-Ducat-USDA

[lbross]

There are 2 glitches on the 'Run Processor' screen in the current .msi for packages without cut trees that are nasty because they are conspirators.

1. All RxPackages are listed on the screen regardless if they have tree cut list records. The tree counts for these RxPackages reflect the counts for the most recent RxPackage that had tree cut list records so they are incorrect. RxPackages without tree cut list records should not be listed on this screen. This was an unintended consequence of the re-work to apply harvest costs to stands that don't have harvest activity. I'm attaching a copy of the screen for the cwi project with my fixes applied.
2. If you try to run an RxPackage that doesn't have tree cut list records, BioSum won't let you do it, but unfortunately prevents Processor from running an RxPackage with tree cut list records without closing and reopening Processor. There was an error variable in the validation routine that wasn't reset with each validation.

I can post a new v5.11.0 .msi with these fixes if you would like me to.

[Cat-Ducat-USDA]

Good morning Lesley,

It appears I am running into the same issue as Jeremy. To avoid getting the error message when trying to open the rule definitions page, I had to delete the 760 from the fvs master. Now rule definitions will open up. However, it appears that only 761 is loading still (pops up in the bottom right corner while buffering), not 872. No trees for 872 in the cut list still.

From: Lesley @.> Sent: Tuesday, August 13, 2024 11:03 AM To: USFS-PNW/Fia-Biosum-Manager @.> Cc: Ducat, Caitlin - FS, OR @.>; Mention @.> Subject: Re: [USFS-PNW/Fia-Biosum-Manager] FVS: post-processing audit, cut trees not being recognized for one Rx package (Issue #342)

There are 2 glitches on the 'Run Processor' screen in the current .msi for packages without cut trees that are nasty because they are conspirators.

1. All RxPackages are listed on the screen regardless if they have tree cut list records. The tree counts for these RxPackages reflect the counts for the most recent RxPackage that had tree cut list records so they are incorrect. RxPackages without tree cut list records should not be listed on this screen. This was an unintended consequence of the re-work to apply harvest costs to stands that don't have harvest activity. I'm attaching a copy of the screen for the cwi project with my fixes applied.
2. If you try to run an RxPackage that doesn't have tree cut list records, BioSum won't let you do it, but unfortunately prevents Processor from running an RxPackage with tree cut list records without closing and reopening Processor. There was an error variable in the validation routine that wasn't reset with each validation.

I can post a new v5.11.0 .msi with these fixes if you would like me to.

image.png (view on web)https://github.com/user-attachments/assets/fc3b3dd0-5998-409f-9421-3229515c6569

- Reply to this email directly, view it on GitHubhttps://github.com/USFS-PNW/Fia-Biosum-Manager/issues/342#issuecomment-2286823427, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BJ57OYPVORLWPZJUCEBPZX3ZRJC5HAVCNFSM6AAAAABMHOO3SOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBWHAZDGNBSG4. You are receiving this because you were mentioned.Message ID: @.**@.>>

This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.

[jsfried]

Note that Cat was running the old version of her project before taking the steps to remove the FVSOUT TREE LIST database and the PREPOST_FVSOUT.DB before she can append the packages and end up with a fully correct set of data to load into processor. That is why there are not 872 trees in her version. But, it is clear that 760 being empty is the cause of the problem. It was the first row in her FVS_MASTER.ACCDB.RxPackages table also, not the 999 shown in your screen snap @lbross

[lbross]

To get 872 into the cutlist you need to delete ALL tables out of /fvs/data/FVSOUT_TREE_LIST.db EXCEPT for FVS_CutTree and FVS_InForestTree. Although I'm not sure of the integrity of FVS_InForestTree since we're getting the 2GB error, so you may want to delete that one too. Then run FVS Out for 872 again. I will post a new v5.11.0 .msi with the fixes for the Run Processor screen this afternoon. That should clear the 2 Processor glitches up.

[jsfried]

I deleted those entire databases and they got regenerated when I did the new append, so no need to delete specific tables, I think. I am pretty sure there is still a glitch in that if the first package (in the Rxpackages table) has no FVS output loaded, Processor will fail.

[lbross]

I just posted a new .msi to GitHub that should alleviate the Processor issues. I tested Cat's uploaded project on a different computer at PSU. I

1. Deleted /fvs/data/FVSOUT_TREE_LIST.db. As Jeremy suggested, the database did regenerate with a new append
2. Ran FVS out for 761, 872, 999
3. Ran Processor for 761, 872. Packages without cut trees no longer show up on the list

[jsfried]

Happy to report that this fix also resolved the Processor unhandled exception and freeze up over the unloaded package 760!

[lbross]

Fixed with changes to Processor screen implemented with 8/13 .msi