search

FluxML / FastAI.jl

Repository of best practices for deep learning in Julia, inspired by fastai
https://fluxml.ai/FastAI.jl
MIT License
588 stars 51 forks source link

Keypoint regression example: The input graph contains at least one loop #231

Closed itan1 closed 2 years ago

itan1 commented 2 years ago

Context:
FastAI v0.4.3 Flux v0.13.3 Julia 1.7.2 on mac M1

In the final fitonecycle!(learner, 5) step of the keypoint regression example, the following error occurs in topological_sort_by_dfs() in Graphs.jl: The input graph contains at least one loop.

Below are the error and MWE (reduced version of the tutorial in the documentation).

The error ```julia ERROR: The input graph contains at least one loop. Stacktrace: [1] error(s::String) @ Base ./error.jl:33 [2] topological_sort_by_dfs(::Type{Graphs.IsDirected{Graphs.SimpleGraphs.SimpleDiGraph{Int64}}}, g::Graphs.SimpleGraphs.SimpleDiGraph{Int64}) @ Graphs ~/.julia/packages/Graphs/zrMoC/src/traversals/dfs.jl:65 [3] topological_sort_by_dfs(g::Graphs.SimpleGraphs.SimpleDiGraph{Int64}) @ Graphs ~/.julia/packages/SimpleTraits/l1ZsK/src/SimpleTraits.jl:331 [4] (::FluxTraining.var"#16#17"{Learner})() @ FluxTraining ~/.julia/packages/FluxTraining/bday3/src/callbacks/execution.jl:9 [5] ignore @ ~/.julia/packages/Zygote/DkIUK/src/lib/utils.jl:25 [inlined] [6] handle(runner::FluxTraining.LinearRunner, event::FluxTraining.Events.EpochBegin, phase::TrainingPhase, learner::Learner) @ FluxTraining ~/.julia/packages/FluxTraining/bday3/src/callbacks/execution.jl:8 [7] (::FluxTraining.var"#handlefn#77"{Learner, TrainingPhase})(e::FluxTraining.Events.EpochBegin) @ FluxTraining ~/.julia/packages/FluxTraining/bday3/src/training.jl:102 [8] runepoch(epochfn::FluxTraining.var"#67#68"{Learner, TrainingPhase, DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{FastAI.TaskDataset{Tuple{MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{Vector{Int64}}, false}, LearnBase.ObsDim.Undefined}, MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadannotfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{Vector{Int64}}, false}, LearnBase.ObsDim.Undefined}}, SupervisedTask{NamedTuple{(:input, :target, :sample, :encodedsample, :x, :y, :ŷ, :pred), Tuple{Image{2}, Keypoints{2, 1}, Tuple{Image{2}, Keypoints{2, 1}}, Tuple{Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}}, Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}, Bounded{2, Keypoints{2, 1}}}}, Tuple{ProjectiveTransforms{2, NamedTuple{(:training, :validation, :inference), Tuple{DataAugmentation.BufferedThreadsafe, DataAugmentation.BufferedThreadsafe, DataAugmentation.Sequence{Tuple{DataAugmentation.CroppedProjectiveTransform{DataAugmentation.ScaleKeepAspect{2}, DataAugmentation.PadDivisible}, DataAugmentation.PinOrigin}}}}}, ImagePreprocessing{FixedPointNumbers.N0f8, 3, ColorTypes.RGB{FixedPointNumbers.N0f8}, Float32}, KeypointPreprocessing{2, Float32}}}, Training}}}}, learner::Learner, phase::TrainingPhase) @ FluxTraining ~/.julia/packages/FluxTraining/bday3/src/training.jl:104 [9] epoch! @ ~/.julia/packages/FluxTraining/bday3/src/training.jl:22 [inlined] [10] (::FastAI.var"#154#156"{Tuple{Pair{TrainingPhase, DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{FastAI.TaskDataset{Tuple{MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{Vector{Int64}}, false}, LearnBase.ObsDim.Undefined}, MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadannotfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{Vector{Int64}}, false}, LearnBase.ObsDim.Undefined}}, SupervisedTask{NamedTuple{(:input, :target, :sample, :encodedsample, :x, :y, :ŷ, :pred), Tuple{Image{2}, Keypoints{2, 1}, Tuple{Image{2}, Keypoints{2, 1}}, Tuple{Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}}, Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}, Bounded{2, Keypoints{2, 1}}}}, Tuple{ProjectiveTransforms{2, NamedTuple{(:training, :validation, :inference), Tuple{DataAugmentation.BufferedThreadsafe, DataAugmentation.BufferedThreadsafe, DataAugmentation.Sequence{Tuple{DataAugmentation.CroppedProjectiveTransform{DataAugmentation.ScaleKeepAspect{2}, DataAugmentation.PadDivisible}, DataAugmentation.PinOrigin}}}}}, ImagePreprocessing{FixedPointNumbers.N0f8, 3, ColorTypes.RGB{FixedPointNumbers.N0f8}, Float32}, KeypointPreprocessing{2, Float32}}}, Training}}}}, Pair{ValidationPhase, DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{FastAI.TaskDataset{Tuple{MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, Vector{Int64}, LearnBase.ObsDim.Undefined}, MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadannotfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, Vector{Int64}, LearnBase.ObsDim.Undefined}}, SupervisedTask{NamedTuple{(:input, :target, :sample, :encodedsample, :x, :y, :ŷ, :pred), Tuple{Image{2}, Keypoints{2, 1}, Tuple{Image{2}, Keypoints{2, 1}}, Tuple{Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}}, Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}, Bounded{2, Keypoints{2, 1}}}}, Tuple{ProjectiveTransforms{2, NamedTuple{(:training, :validation, :inference), Tuple{DataAugmentation.BufferedThreadsafe, DataAugmentation.BufferedThreadsafe, DataAugmentation.Sequence{Tuple{DataAugmentation.CroppedProjectiveTransform{DataAugmentation.ScaleKeepAspect{2}, DataAugmentation.PadDivisible}, DataAugmentation.PinOrigin}}}}}, ImagePreprocessing{FixedPointNumbers.N0f8, 3, ColorTypes.RGB{FixedPointNumbers.N0f8}, Float32}, KeypointPreprocessing{2, Float32}}}, Validation}}}}}, Learner, Int64})() @ FastAI ~/.julia/packages/FastAI/sjHxr/src/training/onecycle.jl:36 [11] withcallbacks(f::FastAI.var"#154#156"{Tuple{Pair{TrainingPhase, DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{FastAI.TaskDataset{Tuple{MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{Vector{Int64}}, false}, LearnBase.ObsDim.Undefined}, MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadannotfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{Vector{Int64}}, false}, LearnBase.ObsDim.Undefined}}, SupervisedTask{NamedTuple{(:input, :target, :sample, :encodedsample, :x, :y, :ŷ, :pred), Tuple{Image{2}, Keypoints{2, 1}, Tuple{Image{2}, Keypoints{2, 1}}, Tuple{Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}}, Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}, Bounded{2, Keypoints{2, 1}}}}, Tuple{ProjectiveTransforms{2, NamedTuple{(:training, :validation, :inference), Tuple{DataAugmentation.BufferedThreadsafe, DataAugmentation.BufferedThreadsafe, DataAugmentation.Sequence{Tuple{DataAugmentation.CroppedProjectiveTransform{DataAugmentation.ScaleKeepAspect{2}, DataAugmentation.PadDivisible}, DataAugmentation.PinOrigin}}}}}, ImagePreprocessing{FixedPointNumbers.N0f8, 3, ColorTypes.RGB{FixedPointNumbers.N0f8}, Float32}, KeypointPreprocessing{2, Float32}}}, Training}}}}, Pair{ValidationPhase, DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{FastAI.TaskDataset{Tuple{MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, Vector{Int64}, LearnBase.ObsDim.Undefined}, MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadannotfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, Vector{Int64}, LearnBase.ObsDim.Undefined}}, SupervisedTask{NamedTuple{(:input, :target, :sample, :encodedsample, :x, :y, :ŷ, :pred), Tuple{Image{2}, Keypoints{2, 1}, Tuple{Image{2}, Keypoints{2, 1}}, Tuple{Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}}, Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}, Bounded{2, Keypoints{2, 1}}}}, Tuple{ProjectiveTransforms{2, NamedTuple{(:training, :validation, :inference), Tuple{DataAugmentation.BufferedThreadsafe, DataAugmentation.BufferedThreadsafe, DataAugmentation.Sequence{Tuple{DataAugmentation.CroppedProjectiveTransform{DataAugmentation.ScaleKeepAspect{2}, DataAugmentation.PadDivisible}, DataAugmentation.PinOrigin}}}}}, ImagePreprocessing{FixedPointNumbers.N0f8, 3, ColorTypes.RGB{FixedPointNumbers.N0f8}, Float32}, KeypointPreprocessing{2, Float32}}}, Validation}}}}}, Learner, Int64}, learner::Learner, callbacks::Scheduler) @ FastAI ~/.julia/packages/FastAI/sjHxr/src/training/utils.jl:79 [12] #153 @ ~/.julia/packages/FastAI/sjHxr/src/training/onecycle.jl:33 [inlined] [13] withfields(f::FastAI.var"#153#155"{Tuple{Pair{TrainingPhase, DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{FastAI.TaskDataset{Tuple{MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{Vector{Int64}}, false}, LearnBase.ObsDim.Undefined}, MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadannotfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{Vector{Int64}}, false}, LearnBase.ObsDim.Undefined}}, SupervisedTask{NamedTuple{(:input, :target, :sample, :encodedsample, :x, :y, :ŷ, :pred), Tuple{Image{2}, Keypoints{2, 1}, Tuple{Image{2}, Keypoints{2, 1}}, Tuple{Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}}, Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}, Bounded{2, Keypoints{2, 1}}}}, Tuple{ProjectiveTransforms{2, NamedTuple{(:training, :validation, :inference), Tuple{DataAugmentation.BufferedThreadsafe, DataAugmentation.BufferedThreadsafe, DataAugmentation.Sequence{Tuple{DataAugmentation.CroppedProjectiveTransform{DataAugmentation.ScaleKeepAspect{2}, DataAugmentation.PadDivisible}, DataAugmentation.PinOrigin}}}}}, ImagePreprocessing{FixedPointNumbers.N0f8, 3, ColorTypes.RGB{FixedPointNumbers.N0f8}, Float32}, KeypointPreprocessing{2, Float32}}}, Training}}}}, Pair{ValidationPhase, DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{FastAI.TaskDataset{Tuple{MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, Vector{Int64}, LearnBase.ObsDim.Undefined}, MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadannotfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, Vector{Int64}, LearnBase.ObsDim.Undefined}}, SupervisedTask{NamedTuple{(:input, :target, :sample, :encodedsample, :x, :y, :ŷ, :pred), Tuple{Image{2}, Keypoints{2, 1}, Tuple{Image{2}, Keypoints{2, 1}}, Tuple{Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}}, Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}, Bounded{2, Keypoints{2, 1}}}}, Tuple{ProjectiveTransforms{2, NamedTuple{(:training, :validation, :inference), Tuple{DataAugmentation.BufferedThreadsafe, DataAugmentation.BufferedThreadsafe, DataAugmentation.Sequence{Tuple{DataAugmentation.CroppedProjectiveTransform{DataAugmentation.ScaleKeepAspect{2}, DataAugmentation.PadDivisible}, DataAugmentation.PinOrigin}}}}}, ImagePreprocessing{FixedPointNumbers.N0f8, 3, ColorTypes.RGB{FixedPointNumbers.N0f8}, Float32}, KeypointPreprocessing{2, Float32}}}, Validation}}}}}, Learner, Int64, Scheduler}, x::Learner; kwargs::Base.Pairs{Symbol, Flux.Optimise.ADAM, Tuple{Symbol}, NamedTuple{(:optimizer,), Tuple{Flux.Optimise.ADAM}}}) @ FastAI ~/.julia/packages/FastAI/sjHxr/src/training/utils.jl:53 [14] fitonecycle!(learner::Learner, nepochs::Int64, maxlr::Float64; phases::Tuple{Pair{TrainingPhase, DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{FastAI.TaskDataset{Tuple{MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{Vector{Int64}}, false}, LearnBase.ObsDim.Undefined}, MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadannotfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{Vector{Int64}}, false}, LearnBase.ObsDim.Undefined}}, SupervisedTask{NamedTuple{(:input, :target, :sample, :encodedsample, :x, :y, :ŷ, :pred), Tuple{Image{2}, Keypoints{2, 1}, Tuple{Image{2}, Keypoints{2, 1}}, Tuple{Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}}, Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}, Bounded{2, Keypoints{2, 1}}}}, Tuple{ProjectiveTransforms{2, NamedTuple{(:training, :validation, :inference), Tuple{DataAugmentation.BufferedThreadsafe, DataAugmentation.BufferedThreadsafe, DataAugmentation.Sequence{Tuple{DataAugmentation.CroppedProjectiveTransform{DataAugmentation.ScaleKeepAspect{2}, DataAugmentation.PadDivisible}, DataAugmentation.PinOrigin}}}}}, ImagePreprocessing{FixedPointNumbers.N0f8, 3, ColorTypes.RGB{FixedPointNumbers.N0f8}, Float32}, KeypointPreprocessing{2, Float32}}}, Training}}}}, Pair{ValidationPhase, DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{FastAI.TaskDataset{Tuple{MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, Vector{Int64}, LearnBase.ObsDim.Undefined}, MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadannotfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, Vector{Int64}, LearnBase.ObsDim.Undefined}}, SupervisedTask{NamedTuple{(:input, :target, :sample, :encodedsample, :x, :y, :ŷ, :pred), Tuple{Image{2}, Keypoints{2, 1}, Tuple{Image{2}, Keypoints{2, 1}}, Tuple{Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}}, Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}, Bounded{2, FastAI.Vision.KeypointTensor{2, Float32, 1}}, Bounded{2, Keypoints{2, 1}}}}, Tuple{ProjectiveTransforms{2, NamedTuple{(:training, :validation, :inference), Tuple{DataAugmentation.BufferedThreadsafe, DataAugmentation.BufferedThreadsafe, DataAugmentation.Sequence{Tuple{DataAugmentation.CroppedProjectiveTransform{DataAugmentation.ScaleKeepAspect{2}, DataAugmentation.PadDivisible}, DataAugmentation.PinOrigin}}}}}, ImagePreprocessing{FixedPointNumbers.N0f8, 3, ColorTypes.RGB{FixedPointNumbers.N0f8}, Float32}, KeypointPreprocessing{2, Float32}}}, Validation}}}}}, wd::Float64, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ FastAI ~/.julia/packages/FastAI/sjHxr/src/training/onecycle.jl:32 [15] fitonecycle!(learner::Learner, nepochs::Int64, maxlr::Float64) (repeats 2 times) @ FastAI ~/.julia/packages/FastAI/sjHxr/src/training/onecycle.jl:25 [16] top-level scope @ REPL[41]:1 [17] top-level scope @ ~/.julia/packages/CUDA/GGwVa/src/initialization.jl:52 ```
MWE ```julia using DelimitedFiles: readdlm using FastAI using FastAI.FilePathsBase, FastAI.StaticArrays import FastAI.DataAugmentation data = ( mapobs(i->rand(Float32, 300, 300), 1:100), mapobs(i->[50, 50], 1:100) ) traindata = datasubset(data, (1:70)) validdata = datasubset(data, (71:100)) sz = (224, 224) task = SupervisedTask( (Image{2}(), Keypoints{2}(1)), ( ImagePreprocessing(), KeypointPreprocessing(sz), ) ) model = taskmodel(task, Models.xresnet18()); traindl, validdl = FastAI.taskdataloaders(traindata, validdata, task, 16) import Flux learner = Learner( model, (traindl, validdl), Flux.ADAM(), tasklossfn(task), ToGPU()) fitonecycle!(learner, 1) ```
lorenzoh commented 2 years ago

Thanks for reporting! I'll have a look the coming days 👍

itan1 commented 2 years ago

The image segmentation example yields the same error

Error ```julia julia> fitonecycle!(learner, 1) ERROR: The input graph contains at least one loop. Stacktrace: [1] error(s::String) @ Base ./error.jl:33 [2] topological_sort_by_dfs(::Type{Graphs.IsDirected{Graphs.SimpleGraphs.SimpleDiGraph{Int64}}}, g::Graphs.SimpleGraphs.SimpleDiGraph{Int64}) @ Graphs ~/.julia/packages/Graphs/zrMoC/src/traversals/dfs.jl:65 [3] topological_sort_by_dfs(g::Graphs.SimpleGraphs.SimpleDiGraph{Int64}) @ Graphs ~/.julia/packages/SimpleTraits/l1ZsK/src/SimpleTraits.jl:331 [4] (::FluxTraining.var"#16#17"{Learner})() @ FluxTraining ~/.julia/packages/FluxTraining/bday3/src/callbacks/execution.jl:9 [5] ignore @ ~/.julia/packages/Zygote/DkIUK/src/lib/utils.jl:25 [inlined] [6] handle(runner::FluxTraining.LinearRunner, event::FluxTraining.Events.EpochBegin, phase::TrainingPhase, learner::Learner) @ FluxTraining ~/.julia/packages/FluxTraining/bday3/src/callbacks/execution.jl:8 [7] (::FluxTraining.var"#handlefn#77"{Learner, TrainingPhase})(e::FluxTraining.Events.EpochBegin) @ FluxTraining ~/.julia/packages/FluxTraining/bday3/src/training.jl:102 [8] runepoch(epochfn::FluxTraining.var"#67#68"{Learner, TrainingPhase, DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{FastAI.TaskDataset{Tuple{MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}, LearnBase.ObsDim.Undefined}, MLDataPattern.DataSubset{FastAI.Datasets.MappedData{var"#1#2", SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}, LearnBase.ObsDim.Undefined}}, SupervisedTask{NamedTuple{(:input, :target, :sample, :encodedsample, :x, :y, :ŷ, :pred), Tuple{Image{2}, Mask{2, String}, Tuple{Image{2}, Mask{2, String}}, Tuple{Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.OneHotTensor{2, String}}}, Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.OneHotTensor{2, String}}, Bounded{2, FastAI.OneHotTensor{2, String}}, Bounded{2, Mask{2, String}}}}, Tuple{ProjectiveTransforms{2, NamedTuple{(:training, :validation, :inference), Tuple{DataAugmentation.BufferedThreadsafe, DataAugmentation.BufferedThreadsafe, DataAugmentation.Sequence{Tuple{DataAugmentation.CroppedProjectiveTransform{DataAugmentation.ScaleKeepAspect{2}, DataAugmentation.PadDivisible}, DataAugmentation.PinOrigin}}}}}, ImagePreprocessing{FixedPointNumbers.N0f8, 3, ColorTypes.RGB{FixedPointNumbers.N0f8}, Float32}, OneHot{DataType}}}, Training}}}}, learner::Learner, phase::TrainingPhase) @ FluxTraining ~/.julia/packages/FluxTraining/bday3/src/training.jl:104 [9] epoch! @ ~/.julia/packages/FluxTraining/bday3/src/training.jl:22 [inlined] [10] (::FastAI.var"#154#156"{Tuple{Pair{TrainingPhase, DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{FastAI.TaskDataset{Tuple{MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}, LearnBase.ObsDim.Undefined}, MLDataPattern.DataSubset{FastAI.Datasets.MappedData{var"#1#2", SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}, LearnBase.ObsDim.Undefined}}, SupervisedTask{NamedTuple{(:input, :target, :sample, :encodedsample, :x, :y, :ŷ, :pred), Tuple{Image{2}, Mask{2, String}, Tuple{Image{2}, Mask{2, String}}, Tuple{Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.OneHotTensor{2, String}}}, Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.OneHotTensor{2, String}}, Bounded{2, FastAI.OneHotTensor{2, String}}, Bounded{2, Mask{2, String}}}}, Tuple{ProjectiveTransforms{2, NamedTuple{(:training, :validation, :inference), Tuple{DataAugmentation.BufferedThreadsafe, DataAugmentation.BufferedThreadsafe, DataAugmentation.Sequence{Tuple{DataAugmentation.CroppedProjectiveTransform{DataAugmentation.ScaleKeepAspect{2}, DataAugmentation.PadDivisible}, DataAugmentation.PinOrigin}}}}}, ImagePreprocessing{FixedPointNumbers.N0f8, 3, ColorTypes.RGB{FixedPointNumbers.N0f8}, Float32}, OneHot{DataType}}}, Training}}}}, Pair{ValidationPhase, DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{FastAI.TaskDataset{Tuple{MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}, LearnBase.ObsDim.Undefined}, MLDataPattern.DataSubset{FastAI.Datasets.MappedData{var"#1#2", SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}, LearnBase.ObsDim.Undefined}}, SupervisedTask{NamedTuple{(:input, :target, :sample, :encodedsample, :x, :y, :ŷ, :pred), Tuple{Image{2}, Mask{2, String}, Tuple{Image{2}, Mask{2, String}}, Tuple{Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.OneHotTensor{2, String}}}, Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.OneHotTensor{2, String}}, Bounded{2, FastAI.OneHotTensor{2, String}}, Bounded{2, Mask{2, String}}}}, Tuple{ProjectiveTransforms{2, NamedTuple{(:training, :validation, :inference), Tuple{DataAugmentation.BufferedThreadsafe, DataAugmentation.BufferedThreadsafe, DataAugmentation.Sequence{Tuple{DataAugmentation.CroppedProjectiveTransform{DataAugmentation.ScaleKeepAspect{2}, DataAugmentation.PadDivisible}, DataAugmentation.PinOrigin}}}}}, ImagePreprocessing{FixedPointNumbers.N0f8, 3, ColorTypes.RGB{FixedPointNumbers.N0f8}, Float32}, OneHot{DataType}}}, Validation}}}}}, Learner, Int64})() @ FastAI ~/.julia/packages/FastAI/sjHxr/src/training/onecycle.jl:36 [11] withcallbacks(f::FastAI.var"#154#156"{Tuple{Pair{TrainingPhase, DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{FastAI.TaskDataset{Tuple{MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}, LearnBase.ObsDim.Undefined}, MLDataPattern.DataSubset{FastAI.Datasets.MappedData{var"#1#2", SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}, LearnBase.ObsDim.Undefined}}, SupervisedTask{NamedTuple{(:input, :target, :sample, :encodedsample, :x, :y, :ŷ, :pred), Tuple{Image{2}, Mask{2, String}, Tuple{Image{2}, Mask{2, String}}, Tuple{Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.OneHotTensor{2, String}}}, Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.OneHotTensor{2, String}}, Bounded{2, FastAI.OneHotTensor{2, String}}, Bounded{2, Mask{2, String}}}}, Tuple{ProjectiveTransforms{2, NamedTuple{(:training, :validation, :inference), Tuple{DataAugmentation.BufferedThreadsafe, DataAugmentation.BufferedThreadsafe, DataAugmentation.Sequence{Tuple{DataAugmentation.CroppedProjectiveTransform{DataAugmentation.ScaleKeepAspect{2}, DataAugmentation.PadDivisible}, DataAugmentation.PinOrigin}}}}}, ImagePreprocessing{FixedPointNumbers.N0f8, 3, ColorTypes.RGB{FixedPointNumbers.N0f8}, Float32}, OneHot{DataType}}}, Training}}}}, Pair{ValidationPhase, DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{FastAI.TaskDataset{Tuple{MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}, LearnBase.ObsDim.Undefined}, MLDataPattern.DataSubset{FastAI.Datasets.MappedData{var"#1#2", SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}, LearnBase.ObsDim.Undefined}}, SupervisedTask{NamedTuple{(:input, :target, :sample, :encodedsample, :x, :y, :ŷ, :pred), Tuple{Image{2}, Mask{2, String}, Tuple{Image{2}, Mask{2, String}}, Tuple{Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.OneHotTensor{2, String}}}, Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.OneHotTensor{2, String}}, Bounded{2, FastAI.OneHotTensor{2, String}}, Bounded{2, Mask{2, String}}}}, Tuple{ProjectiveTransforms{2, NamedTuple{(:training, :validation, :inference), Tuple{DataAugmentation.BufferedThreadsafe, DataAugmentation.BufferedThreadsafe, DataAugmentation.Sequence{Tuple{DataAugmentation.CroppedProjectiveTransform{DataAugmentation.ScaleKeepAspect{2}, DataAugmentation.PadDivisible}, DataAugmentation.PinOrigin}}}}}, ImagePreprocessing{FixedPointNumbers.N0f8, 3, ColorTypes.RGB{FixedPointNumbers.N0f8}, Float32}, OneHot{DataType}}}, Validation}}}}}, Learner, Int64}, learner::Learner, callbacks::Scheduler) @ FastAI ~/.julia/packages/FastAI/sjHxr/src/training/utils.jl:79 [12] #153 @ ~/.julia/packages/FastAI/sjHxr/src/training/onecycle.jl:33 [inlined] [13] withfields(f::FastAI.var"#153#155"{Tuple{Pair{TrainingPhase, DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{FastAI.TaskDataset{Tuple{MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}, LearnBase.ObsDim.Undefined}, MLDataPattern.DataSubset{FastAI.Datasets.MappedData{var"#1#2", SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}, LearnBase.ObsDim.Undefined}}, SupervisedTask{NamedTuple{(:input, :target, :sample, :encodedsample, :x, :y, :ŷ, :pred), Tuple{Image{2}, Mask{2, String}, Tuple{Image{2}, Mask{2, String}}, Tuple{Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.OneHotTensor{2, String}}}, Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.OneHotTensor{2, String}}, Bounded{2, FastAI.OneHotTensor{2, String}}, Bounded{2, Mask{2, String}}}}, Tuple{ProjectiveTransforms{2, NamedTuple{(:training, :validation, :inference), Tuple{DataAugmentation.BufferedThreadsafe, DataAugmentation.BufferedThreadsafe, DataAugmentation.Sequence{Tuple{DataAugmentation.CroppedProjectiveTransform{DataAugmentation.ScaleKeepAspect{2}, DataAugmentation.PadDivisible}, DataAugmentation.PinOrigin}}}}}, ImagePreprocessing{FixedPointNumbers.N0f8, 3, ColorTypes.RGB{FixedPointNumbers.N0f8}, Float32}, OneHot{DataType}}}, Training}}}}, Pair{ValidationPhase, DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{FastAI.TaskDataset{Tuple{MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}, LearnBase.ObsDim.Undefined}, MLDataPattern.DataSubset{FastAI.Datasets.MappedData{var"#1#2", SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}, LearnBase.ObsDim.Undefined}}, SupervisedTask{NamedTuple{(:input, :target, :sample, :encodedsample, :x, :y, :ŷ, :pred), Tuple{Image{2}, Mask{2, String}, Tuple{Image{2}, Mask{2, String}}, Tuple{Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.OneHotTensor{2, String}}}, Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.OneHotTensor{2, String}}, Bounded{2, FastAI.OneHotTensor{2, String}}, Bounded{2, Mask{2, String}}}}, Tuple{ProjectiveTransforms{2, NamedTuple{(:training, :validation, :inference), Tuple{DataAugmentation.BufferedThreadsafe, DataAugmentation.BufferedThreadsafe, DataAugmentation.Sequence{Tuple{DataAugmentation.CroppedProjectiveTransform{DataAugmentation.ScaleKeepAspect{2}, DataAugmentation.PadDivisible}, DataAugmentation.PinOrigin}}}}}, ImagePreprocessing{FixedPointNumbers.N0f8, 3, ColorTypes.RGB{FixedPointNumbers.N0f8}, Float32}, OneHot{DataType}}}, Validation}}}}}, Learner, Int64, Scheduler}, x::Learner; kwargs::Base.Pairs{Symbol, Flux.Optimise.ADAM, Tuple{Symbol}, NamedTuple{(:optimizer,), Tuple{Flux.Optimise.ADAM}}}) @ FastAI ~/.julia/packages/FastAI/sjHxr/src/training/utils.jl:53 [14] fitonecycle!(learner::Learner, nepochs::Int64, maxlr::Float64; phases::Tuple{Pair{TrainingPhase, DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{FastAI.TaskDataset{Tuple{MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}, LearnBase.ObsDim.Undefined}, MLDataPattern.DataSubset{FastAI.Datasets.MappedData{var"#1#2", SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}, LearnBase.ObsDim.Undefined}}, SupervisedTask{NamedTuple{(:input, :target, :sample, :encodedsample, :x, :y, :ŷ, :pred), Tuple{Image{2}, Mask{2, String}, Tuple{Image{2}, Mask{2, String}}, Tuple{Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.OneHotTensor{2, String}}}, Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.OneHotTensor{2, String}}, Bounded{2, FastAI.OneHotTensor{2, String}}, Bounded{2, Mask{2, String}}}}, Tuple{ProjectiveTransforms{2, NamedTuple{(:training, :validation, :inference), Tuple{DataAugmentation.BufferedThreadsafe, DataAugmentation.BufferedThreadsafe, DataAugmentation.Sequence{Tuple{DataAugmentation.CroppedProjectiveTransform{DataAugmentation.ScaleKeepAspect{2}, DataAugmentation.PadDivisible}, DataAugmentation.PinOrigin}}}}}, ImagePreprocessing{FixedPointNumbers.N0f8, 3, ColorTypes.RGB{FixedPointNumbers.N0f8}, Float32}, OneHot{DataType}}}, Training}}}}, Pair{ValidationPhase, DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{FastAI.TaskDataset{Tuple{MLDataPattern.DataSubset{FastAI.Datasets.MappedData{typeof(loadfile), SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}, LearnBase.ObsDim.Undefined}, MLDataPattern.DataSubset{FastAI.Datasets.MappedData{var"#1#2", SubArray{String, 1, Vector{String}, Tuple{Vector{Int64}}, false}}, SubArray{Int64, 1, Vector{Int64}, Tuple{UnitRange{Int64}}, true}, LearnBase.ObsDim.Undefined}}, SupervisedTask{NamedTuple{(:input, :target, :sample, :encodedsample, :x, :y, :ŷ, :pred), Tuple{Image{2}, Mask{2, String}, Tuple{Image{2}, Mask{2, String}}, Tuple{Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.OneHotTensor{2, String}}}, Bounded{2, FastAI.Vision.ImageTensor{2}}, Bounded{2, FastAI.OneHotTensor{2, String}}, Bounded{2, FastAI.OneHotTensor{2, String}}, Bounded{2, Mask{2, String}}}}, Tuple{ProjectiveTransforms{2, NamedTuple{(:training, :validation, :inference), Tuple{DataAugmentation.BufferedThreadsafe, DataAugmentation.BufferedThreadsafe, DataAugmentation.Sequence{Tuple{DataAugmentation.CroppedProjectiveTransform{DataAugmentation.ScaleKeepAspect{2}, DataAugmentation.PadDivisible}, DataAugmentation.PinOrigin}}}}}, ImagePreprocessing{FixedPointNumbers.N0f8, 3, ColorTypes.RGB{FixedPointNumbers.N0f8}, Float32}, OneHot{DataType}}}, Validation}}}}}, wd::Float64, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ FastAI ~/.julia/packages/FastAI/sjHxr/src/training/onecycle.jl:32 [15] fitonecycle!(learner::Learner, nepochs::Int64, maxlr::Float64) (repeats 2 times) @ FastAI ~/.julia/packages/FastAI/sjHxr/src/training/onecycle.jl:25 [16] top-level scope @ REPL[22]:1 [17] top-level scope @ ~/.julia/packages/CUDA/GGwVa/src/initialization.jl:52 ```
MWE ```julia using FastAI, Metalhead dir = datasetpath("camvid_tiny") classes = readlines(open(joinpath(dir, "codes.txt"))) images = Datasets.loadfolderdata( joinpath(dir, "images"), filterfn=isimagefile, loadfn=loadfile) masks = Datasets.loadfolderdata( joinpath(dir, "labels"), filterfn=isimagefile, loadfn=f -> loadmask(f, classes)) data = (images, masks) sample = getobs(data, 1); task = BlockTask( (Image{2}(), Mask{2}(classes)), ( ProjectiveTransforms((128, 128)), ImagePreprocessing(), OneHot() ) ) model = taskmodel(task, Models.xresnet18()); traindl, validdl = taskdataloaders(data, task, 16) import Flux learner = Learner(model, (traindl, validdl), Flux.ADAM(), tasklossfn(task), ToGPU()) fitonecycle!(learner, 1) ```
darsnack commented 2 years ago

Possibly related to https://github.com/FluxML/FluxTraining.jl/issues/122. I'll investigate more today.

lorenzoh commented 2 years ago

Fixed in https://github.com/FluxML/FluxTraining.jl/pull/123

lorenzoh commented 2 years ago

Just tagged FluxTraining.jl 0.3.2, so should be released soon!

itan1 commented 2 years ago

Awesome, thanks a lot!