search

JuliaDataCubes / EarthDataLab.jl

Julia interface for Reading from the Earth System Datacube
http://earthsystemdatacube.net
Other
33 stars 14 forks source link

saving a GDALCube from multiple vrts fails with distributed computing #217

Open felixcremer opened 4 years ago

felixcremer commented 4 years ago

When I try to use savecube for a cube generated from nested VRTs with the gdalcubes function from ESDLHelpers this works for multi threaded code, but it fails, when I am using Distributed and SlurmClusterManager with the following 

Error during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n") [1] 
runLoopError during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n")(
::Error during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n")ESDL.DAT
.Error during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n")DATConfig
{Error during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n")1
,Error during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n")1
}Error during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n"), 
::BoolError during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n"))
 at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:408
 [2] mapCube(::ESDL.Proc.var"#cop#30", ::Tuple{ESDLArray{Float32,3,ArchGDAL.RasterDataset{Float32,ArchGDAL.Dataset},Array{CubeAxis,1}}}; max_cache::Float64, indims::InDims, outdims::OutDims, inplace::Bool, ispar::Bool, debug::Bool, include_loopvars::Bool, showprog::Bool, nthreads::Dict{Int64,Int64}, loopchunksize::Dict{Any,Any}, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:278
 [3] #mapCube#34 at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:177 [inlined]
 [4] savecube(::ESDLArray{Float32,3,ArchGDAL.RasterDataset{Float32,ArchGDAL.Dataset},Array{CubeAxis,1}}, ::String; chunksize::Dict{Any,Any}, max_cache::Float64, backend::Symbol, backendargs::Base.Iterators.Pairs{Symbol,Integer,Tuple{Symbol,Symbol},NamedTuple{(:fill_value, :overwrite),Tuple{Int64,Bool}}}) at /home/crem_fe/.julia/dev/ESDL/src/Proc/CubeIO.jl:38
 [5] top-level scope at REPL[12]:1

I'd have to further check, whether this also fails on normally distributed usage.

meggart commented 3 years ago

I this still an issue or was it fixed in the meantime?