Mavisan6 commented 3 years ago

Hi! I was running an experiment when it stopped because of the following error:

ERROR: On worker 2: GDALError (CE_Failure, code 1): C:\Users\M533\Desktop\circuitscape\current map\currentmap1000m_curmap_2765_9041.asc: Write failed, disk full?

gdaljl_errorhandler at C:\Users\M533.julia\packages\GDAL\Yh4fH\src\error.jl:36 gdalcreatecopy at C:\Users\M533.julia\packages\GDAL\Yh4fH\src\gdal_h.jl:366

copy#54 at C:\Users\M533.julia\packages\ArchGDAL\6e9sT\src\dataset.jl:148 [inlined]

84 at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\out.jl:504

create#131 at C:\Users\M533.julia\packages\ArchGDAL\6e9sT\src\context.jl:216

write_raster at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\out.jl:486

write_grid#76 at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\out.jl:313

write_grid at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\out.jl:298 [inlined] write_cur_maps at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\out.jl:85 macro expansion at .\timing.jl:233 [inlined] postprocess at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\core.jl:606 f at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\core.jl:388

100 at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\core.jl:427

106 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\process_messages.jl:294

run_work_thunk at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\process_messages.jl:79 macro expansion at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\process_messages.jl:294 [inlined]

105 at .\task.jl:356

Stacktrace: [1] (::Base.var"#770#772")(::Task) at .\asyncmap.jl:178 [2] foreach(::Base.var"#770#772", ::Array{Any,1}) at .\abstractarray.jl:2009 [3] maptwice(::Function, ::Channel{Any}, ::Array{Any,1}, ::UnitRange{Int64}) at .\asyncmap.jl:178 [4] wrap_n_exec_twice(::Channel{Any}, ::Array{Any,1}, ::Distributed.var"#206#209"{Distributed.WorkerPool}, ::Function, ::UnitRange{Int64}) at .\asyncmap.jl:154 [5] async_usemap(::Distributed.var"#190#192"{Distributed.var"#190#191#193"{Distributed.WorkerPool,Circuitscape.var"#100#107"{Circuitscape.var"#f#106"{Int64,Circuitscape.RasterFlags,Dict{String,String},Array{Int64,1},Array{Int64,1},Circuitscape.Cumulative{Float64},Circuitscape.Shortcut{Float64},Circuitscape.ComponentData{Float64,Int64},Array{Circuitscape.CholmodNode,1}},UnitRange{Int64},Array{Float64,2}}}}, ::UnitRange{Int64}; ntasks::Function, batch_size::Nothing) at .\asyncmap.jl:103 [6] #asyncmap#754 at .\asyncmap.jl:81 [inlined] [7] pmap(::Function, ::Distributed.WorkerPool, ::UnitRange{Int64}; distributed::Bool, batch_size::Int64, on_error::Nothing, retry_delays::Array{Any,1}, retry_check::Nothing) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\pmap.jl:126 [8] pmap(::Function, ::Distributed.WorkerPool, ::UnitRange{Int64}) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\pmap.jl:101 [9] pmap(::Function, ::UnitRange{Int64}; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\pmap.jl:156 [10] pmap at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\pmap.jl:156 [inlined] [11] _cholmod_solver_path(::Circuitscape.GraphData{Float64,Int64}, ::Circuitscape.RasterFlags, ::Dict{String,String}, ::Bool, ::Int64) at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\core.jl:427 [12] single_ground_all_pairs(::Circuitscape.GraphData{Float64,Int64}, ::Circuitscape.RasterFlags, ::Dict{String,String}, ::Bool) at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\core.jl:59 [13] _pt_file_polygons_path(::Circuitscape.RasData{Float64,Int64}, ::Circuitscape.RasterFlags, ::Dict{String,String}) at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\raster\pairwise.jl:105 [14] raster_pairwise(::Type{T} where T, ::Type{T} where T, ::Dict{String,String}) at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\raster\pairwise.jl:27 [15] _compute(::Type{T} where T, ::Type{T} where T, ::Dict{String,String}) at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\run.jl:42 [16] macro expansion at .\timing.jl:233 [inlined] [17] compute(::Dict{String,String}) at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\run.jl:73 [18] step10(::String, ::String) at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\INIBuilder\run.jl:139 [19] step9(::String) at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\INIBuilder\run.jl:131 [20] step8() at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\INIBuilder\run.jl:120 [21] step7() at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\INIBuilder\run.jl:110 [22] step6() at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\INIBuilder\run.jl:97 [23] step5() at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\INIBuilder\run.jl:87 [24] step4() at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\INIBuilder\run.jl:76 [25] step3() at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\INIBuilder\run.jl:47 [26] step2() at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\INIBuilder\run.jl:30 [27] step1() at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\INIBuilder\run.jl:14 [28] start() at C:\Users\M533.julia\packages\Circuitscape\9x9VD\src\INIBuilder\run.jl:188

Does anyone know what I should do? Is there a way to resume the experiment?

vlandau commented 3 years ago

Unfortunately there is not a built-in way to resume a CIrcuitscape run that failed. One thing you could try is to set your output file type as .tif, as those take up less disk space, but this option is not available in the INI builder method -- you'd have to create the INI file manually and add write_as_tif = true, then run Circuitscape using compute(<path to your INI file>).

Another option would be to free up as much space as possible on your hard drive to make room for the Circuitscape output files.

Finally, in the case that you are running pairwise mode, I'm guessing you are also writing the individual pair solves. If you don't need these and just want the cumulative current map, you may consider adding write_cum_cur_map_only = true to your INI file, in which case the output files for the individual solves will not be written.

If the compute time is incredibly long, you could possibly use an include pairs file but that would require you to identify which pairs have already been solved, then create a list of pairs that haven't been solved yet, and finally you'd have to manually combine them, which could prove incredibly cumbersome.

Hope some of the above helps.