search

cardat / air-health-bushfire-smoke-netcdf

Software to share australian bushfire smoke data funded by CAR and ARDC. Supported by CurtinIC and ASDAF
MIT License
1 stars 0 forks source link

stop it from crashing #22

Open ivanhanigan opened 8 months ago

ivanhanigan commented 8 months ago

@cassandrayuen I pushed those minor mods we made back to working_cass branch. Please stop it from crashing

cassandrayuen commented 8 months ago

Fixed the merge statement to make it use terra - try again @ivanhanigan

cassandrayuen commented 8 months ago

The temperamental issue with whether it dies on write_mdim seems to be linked to whether or not it has been read by stars as a proxy or not. See the following (tested on my local, think it's the same on Ztower; but no errors occur when I ran it on PC254):

> r <- terra::rast("CARDAT/ResearchProjects_CAR/Bushfire_Smoke_PM_Exposures/Bushfire_specific_PM25_Aus_2001_2020_v1_3/data_derived/bushfiresmoke_v1_3_2020_compressed_20230825_6.nc", "active_fires_10000")
> r_extrap <- focal(r, 3, "max", na.policy = "only")

> stars_r <- stars::st_as_stars(r_extrap, proxy = T)
> names(stars_r) <- i
> stars::write_mdim(stars_r, filename = "fr_proxy.nc") # FAILS after converting to stars_proxy
Error in names(dim(x[[i]])) <- names(d) : 
  attempt to set an attribute on NULL
> stars_r <- stars::st_as_stars(r_extrap, proxy = F)
> names(stars_r) <- i
> stars::write_mdim(stars_r, filename = "fr_obj.nc") # FAILS after converting to stars object
Error: NA value in which_dims: logic error
> stars_r <- stars::st_as_stars(r_extrap, proxy = T)
> names(stars_r) <- i
> stars_r <- stars::st_as_stars(stars_r)
> stars::write_mdim(stars_r, filename = "fr_proxy_to_obj.nc") # SUCCEEDS after converting to stars_proxy, setting the name, THEN coercing to stars object (!?)
Warning messages:
1: In CPL_write_mdim(file, driver, dimx, cdl, wkt, xy, root_group_options,  :
  GDAL Message 1: Dimension y lacks a indexing variable
2: In CPL_write_mdim(file, driver, dimx, cdl, wkt, xy, root_group_options,  :
  GDAL Message 1: Dimension x lacks a indexing variable
3: In CPL_write_mdim(file, driver, dimx, cdl, wkt, xy, root_group_options,  :
  GDAL Error 6: SetIndexingVariable() not implemented
4: In CPL_write_mdim(file, driver, dimx, cdl, wkt, xy, root_group_options,  :
  GDAL Error 6: SetIndexingVariable() not implemented
5: In CPL_write_mdim(file, driver, dimx, cdl, wkt, xy, root_group_options,  :
  GDAL Error 6: SetIndexingVariable() not implemented
ivanhanigan commented 8 months ago

trimmed_.. variable has been fixed (wherever it was NA in smoke_p95_v1_3, I have applied to the trimmed surface) Think have solved that random fail on write_mdim, I suspect has to do with whether it converts from terra to a stars_proxy or stars object. So the randomness would have been due to the choice between stars_proxy/stars which I guess it chooses based on available memory or size of the terra object/file. Not fully understanding why (see https://github.com/cardat/air-health-bushfire-smoke-netcdf/issues/22#issuecomment-1846554677) but at least have a work-around. If it doesn't crash while my eyeballs stop looking.