search

open-AIMS / ADRIA.jl

ADRIA: Adaptive Dynamic Reef Intervention Algorithms. A multi-criteria decision support platform for informing reef restoration and adaptation interventions.
MIT License
18 stars 6 forks source link

Error when calculating total cover #855

Closed VHallerBull closed 1 month ago

VHallerBull commented 1 month ago

@DanTanAtAims The new setup of the CScapeResultsSet (most up to date code) is now causing an issue when I try to calculate the total_cover using the following function https://github.com/open-AIMS/ADRIA.jl/blob/c7334e5f0c2a8d3dcdf11af6f59824d42a954581/src/metrics/scenario.jl#L43

The error I receive is image

I receive this error both on the HPC and locally

ConnectedSystems commented 1 month ago

Hi Vanessa, can we have a MWE please?

VHallerBull commented 1 month ago

What is MWE?

ConnectedSystems commented 1 month ago

Minimum Working Example

Essentially a copy of the code that reproduces the problem that we can just copy/paste and test on our end

ConnectedSystems commented 1 month ago

Like how I did here, showing I couldn't reproduce your earlier issue:

https://github.com/open-AIMS/ADRIA.jl/issues/842#issue-2506531885

VHallerBull commented 1 month ago

using JLD2 rs=load_object(raw"Set2Test3_rs.jld2") stac = ADRIA.metrics.scenario_total_cover(rs)

[Uploading Set2Test3_rs.zip…]()

VHallerBull commented 1 month ago

Set2Test3_rs.zip

ConnectedSystems commented 1 month ago

FYI @VHallerBull

https://github.com/open-AIMS/ADRIA.jl/issues/842#issuecomment-2330425216

https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax

ConnectedSystems commented 1 month ago

@DanTanAtAims The recent change to generalize :sites to :locations is causing this issue. The metric Vanessa is trying to use is looking for :location but the dataset she has pre-loaded and cached in JLD2 has stored it as :sites.

Note there seems to be some C~scape specific methods that still expect :sites - not sure if this will cause an issue or not as it could be purely internal handling of data that doesn't affect the metrics.

@VHallerBull I understand why you're using JLD2 in this case but the point of the data stores is to avoid using platform/language specific file formats. Given the other issue (re very slow dataset loading), is now resolved, are you able to simply reload the data?

VHallerBull commented 1 month ago

@ConnectedSystems the dataset loads quicker, but still takes close to an hour for the larger set (~4900 files), so that on its own isn't feasible. Also, based on an idea by Pedro I am loading the dataset on the HPC where the ~Tb of C~scape data is stored, then download the ARDIA dataset in JLD2 to analyze locally. It's a lot simpler to test plotting and analysis locally than on the HPC, especially at this point where I still experience a lot of issues. Happy to do all on the HPC in the future once things run smoother

ConnectedSystems commented 1 month ago

Thanks for the context @VHallerBull

FYI, I recently benchmarked JLD2 against the built-in Serialize/Deserialize method. If you don't need the flexibility of a HDF5-backed store, dumping the binary representation to disk is much faster (as you'd expect).

https://github.com/open-AIMS/ReefGuideAPI.jl/pull/10#issue-2540610063