DiffEqFlux v0.4.0 will not pre-compile with Flux 0.10.0 in Julia 1.3

[cems2]

This issue is superficially similar to #81 but the resolution there was to use a newer Julia. However, this happens in the latest available stable Julia (v1.3).

Problem In julia 1.3 on MacOSX running Pkg.add("DiffEqFlux") will add DiffEqFlux v0.4.0 and Flux 0.10

Flux will precompile but DiffEqFLux will not. Differential Equations also has a problem.

platform Julia v1.3 install on mac os. using Pkg.add to install . Status ~/.julia/environments/v1.3/Project.toml [aae7a2af] DiffEqFlux v0.4.0 [0c46a032] DifferentialEquations v6.6.0 [587475ba] Flux v0.10.0 [7073ff75] IJulia v1.20.2 [91a5bcdd] Plots v0.28.4 [9f7883ad] Tracker v0.2.6

Expected outcome In Julia 1.2, which installs an older (0.9.0) Flux package these errors do not occur and one can use the packages.

Actual Outcome

Errors thrown by DiffEqFlux v0.4.0 precompile Warnings thrown by DifferentialEquations v6.6.0

Regression both warnings and errors refer to Missing Tracker dependency. Addititionally some of the messages seem to say this dependency is supposed to be supplied by the Flux Package.

However the Flux news.md says that Tracker Dependency has been removed in favor or Zygote in Flux 0.10.0 perhaps this has something to do with why it is missing?

I read an older closed Flux issue on Tracker dependency (Flux.jl#695 ) that applies only to earlier revisions of Flux and the fix there was to add Tracker package. But I did this and it does not fix the problem.

I filed a related issue on Flux.jl #975 but I have come to suspect the issue is that the problem is in DiffEqFlux expecting something that is no longer contained in Flux.jl

Workaround I'm not sure if I could revert this to Flux 0.9.0 (and I don't know how to do that ) but even if I did this is not as desirable since the functor macro is not available in that version of Flux

Code to reproduce 100% reproducible Add the packages listed above to julia 1.3 then import DiffEqFlux or DifferentialEquations

Results

julia> using  DiffEqFlux
[ Info: Precompiling DiffEqFlux [aae7a2af-3d4f-5e19-a356-7da93b79d9d0]
ERROR: LoadError: LoadError: UndefVarError: Tracker not defined
Stacktrace:
 [1] include at ./boot.jl:328 [inlined]
 [2] include_relative(::Module, ::String) at ./loading.jl:1105
 [3] include at ./Base.jl:31 [inlined]
 [4] include(::String) at /Users/cems/.julia/packages/DiffEqSensitivity/DI6VG/src/DiffEqSensitivity.jl:3
 [5] top-level scope at /Users/cems/.julia/packages/DiffEqSensitivity/DI6VG/src/DiffEqSensitivity.jl:14
 [6] include at ./boot.jl:328 [inlined]
 [7] include_relative(::Module, ::String) at ./loading.jl:1105
 [8] include(::Module, ::String) at ./Base.jl:31
 [9] top-level scope at none:2
 [10] eval at ./boot.jl:330 [inlined]
 [11] eval(::Expr) at ./client.jl:425
 [12] top-level scope at ./none:3
in expression starting at /Users/cems/.julia/packages/DiffEqSensitivity/DI6VG/src/adjoint_sensitivity.jl:1
in expression starting at /Users/cems/.julia/packages/DiffEqSensitivity/DI6VG/src/DiffEqSensitivity.jl:14
ERROR: LoadError: Failed to precompile DiffEqSensitivity [41bf760c-e81c-5289-8e54-58b1f1f8abe2] to /Users/cems/.julia/compiled/v1.3/DiffEqSensitivity/02xYn_ml8Hi.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1283
 [3] _require(::Base.PkgId) at ./loading.jl:1024
 [4] require(::Base.PkgId) at ./loading.jl:922
 [5] require(::Module, ::Symbol) at ./loading.jl:917
 [6] include at ./boot.jl:328 [inlined]
 [7] include_relative(::Module, ::String) at ./loading.jl:1105
 [8] include(::Module, ::String) at ./Base.jl:31
 [9] top-level scope at none:2
 [10] eval at ./boot.jl:330 [inlined]
 [11] eval(::Expr) at ./client.jl:425
 [12] top-level scope at ./none:3
in expression starting at /Users/cems/.julia/packages/DiffEqFlux/t2FwV/src/DiffEqFlux.jl:3
ERROR: Failed to precompile DiffEqFlux [aae7a2af-3d4f-5e19-a356-7da93b79d9d0] to /Users/cems/.julia/compiled/v1.3/DiffEqFlux/BdO4p_ml8Hi.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1283
 [3] _require(::Base.PkgId) at ./loading.jl:1024
 [4] require(::Base.PkgId) at ./loading.jl:922
 [5] require(::Module, ::Symbol) at ./loading.jl:917

A similar result from another package

julia> using DifferentialEquations
┌ Warning: Error requiring Flux from ArrayInterface:
│ UndefVarError: Tracker not defined
│ Stacktrace:
│  [1] getproperty(::Module, ::Symbol) at ./Base.jl:13
│  [2] top-level scope at /Users/cems/.julia/packages/ArrayInterface/qMMsu/src/ArrayInterface.jl:32
│  [3] eval at ./boot.jl:330 [inlined]
│  [4] eval at /Users/cems/.julia/packages/ArrayInterface/qMMsu/src/ArrayInterface.jl:1 [inlined]
│  [5] (::ArrayInterface.var"#9#18")() at /Users/cems/.julia/packages/Requires/9Jse8/src/require.jl:67
│  [6] err(::ArrayInterface.var"#9#18", ::Module, ::String) at /Users/cems/.julia/packages/Requires/9Jse8/src/require.jl:38
│  [7] #8 at /Users/cems/.julia/packages/Requires/9Jse8/src/require.jl:66 [inlined]
│  [8] withpath(::ArrayInterface.var"#8#17", ::String) at /Users/cems/.julia/packages/Requires/9Jse8/src/require.jl:28
│  [9] #7 at /Users/cems/.julia/packages/Requires/9Jse8/src/require.jl:65 [inlined]
│  [10] listenpkg(::ArrayInterface.var"#7#16", ::Base.PkgId) at /Users/cems/.julia/packages/Requires/9Jse8/src/require.jl:13
│  [11] macro expansion at /Users/cems/.julia/packages/Requires/9Jse8/src/require.jl:64 [inlined]
│  [12] __init__() at /Users/cems/.julia/packages/ArrayInterface/qMMsu/src/ArrayInterface.jl:31
│  [13] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:692
│  [14] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:776
│  [15] _tryrequire_from_serialized(::Base.PkgId, ::UInt64, ::String) at ./loading.jl:707
│  [16] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:765
│  [17] _tryrequire_from_serialized(::Base.PkgId, ::UInt64, ::String) at ./loading.jl:707
│  [18] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:765
│  [19] _require(::Base.PkgId) at ./loading.jl:1001
│  [20] require(::Base.PkgId) at ./loading.jl:922
│  [21] require(::Module, ::Symbol) at ./loading.jl:917
│  [22] eval(::Module, ::Any) at ./boot.jl:330
│  [23] eval_user_input(::Any, ::REPL.REPLBackend) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/REPL/src/REPL.jl:86
│  [24] macro expansion at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/REPL/src/REPL.jl:118 [inlined]
│  [25] (::REPL.var"#26#27"{REPL.REPLBackend})() at ./task.jl:333
└ @ Requires ~/.julia/packages/Requires/9Jse8/src/require.jl:40
┌ Warning: Error requiring Flux from DiffEqBase:
│ UndefVarError: Tracker not defined
│ Stacktrace:
│  [1] getproperty(::Module, ::Symbol) at ./Base.jl:13
│  [2] top-level scope at /Users/cems/.julia/packages/DiffEqBase/DqkH4/src/init.jl:87
│  [3] eval at ./boot.jl:330 [inlined]
│  [4] eval at /Users/cems/.julia/packages/DiffEqBase/DqkH4/src/DiffEqBase.jl:1 [inlined]
│  [5] (::DiffEqBase.var"#410#434")() at /Users/cems/.julia/packages/Requires/9Jse8/src/require.jl:67
│  [6] err(::DiffEqBase.var"#410#434", ::Module, ::String) at /Users/cems/.julia/packages/Requires/9Jse8/src/require.jl:38
│  [7] #409 at /Users/cems/.julia/packages/Requires/9Jse8/src/require.jl:66 [inlined]
│  [8] withpath(::DiffEqBase.var"#409#433", ::String) at /Users/cems/.julia/packages/Requires/9Jse8/src/require.jl:28
│  [9] #408 at /Users/cems/.julia/packages/Requires/9Jse8/src/require.jl:65 [inlined]
│  [10] listenpkg(::DiffEqBase.var"#408#432", ::Base.PkgId) at /Users/cems/.julia/packages/Requires/9Jse8/src/require.jl:13
│  [11] macro expansion at /Users/cems/.julia/packages/Requires/9Jse8/src/require.jl:64 [inlined]
│  [12] __init__() at /Users/cems/.julia/packages/DiffEqBase/DqkH4/src/init.jl:85
│  [13] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:692
│  [14] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:776
│  [15] _tryrequire_from_serialized(::Base.PkgId, ::UInt64, ::String) at ./loading.jl:707
│  [16] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:765
│  [17] _require(::Base.PkgId) at ./loading.jl:1001
│  [18] require(::Base.PkgId) at ./loading.jl:922
│  [19] require(::Module, ::Symbol) at ./loading.jl:917
│  [20] eval(::Module, ::Any) at ./boot.jl:330
│  [21] eval_user_input(::Any, ::REPL.REPLBackend) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/REPL/src/REPL.jl:86
│  [22] macro expansion at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/REPL/src/REPL.jl:118 [inlined]
│  [23] (::REPL.var"#26#27"{REPL.REPLBackend})() at ./task.jl:333
└ @ Requires ~/.julia/packages/Requires/9Jse8/src/require.jl:40

[ChrisRackauckas]

Flux 0.10 was a huge change that moved everything to Zygote, so it'll take a bit to update it. The current status is here: https://github.com/JuliaDiffEq/DiffEqFlux.jl/pull/96

[cems2]

thank you chris.

I'm experimenting this morning with trying to find a mutually compatible set of version with higher version numbers.

so far I've found only incompatible ones. for example I tried Flux 0.10.0 with DiffEqFlux 0.10.0 but package manager said these were incompatiblle so I swithced of Flux 0.9.0 and while everything did compile this time, at run time neural_ode(dudt,x,tspan,Tsit5(),saveat=t,reltol=1e-7,abstol=1e-9) gave an error message about missing some function.

So at the moment I just dowgraded to DiffEqFlux and am trying this again.

Perhaps you could tell me a combination of version numbers for DifferentialEquations DiffEqFlux Flux

that works.

One additionally caveat. I have found that unless Flux.Functor is available I cannot seem to properly add ad hoc parameters to my neuralODE equations and have them optimized. I'm unclear on why that is. Probably something undocumented that I haven't properly figured out. But I think I need that so downgrading too far loses that.

On Dec 24, 2019, at 8:46 AM, Christopher Rackauckas notifications@github.com wrote:

Flux 0.10 was a huge change that moved everything to Zygote, so it'll take a bit to update it. The current status is here: #96 https://github.com/JuliaDiffEq/DiffEqFlux.jl/pull/96 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/JuliaDiffEq/DiffEqFlux.jl/issues/99?email_source=notifications&email_token=ACRAR7TGTU35UFLCNAQJ5NLQ2IVFVA5CNFSM4J67IDMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHTLWFA#issuecomment-568769300, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRAR7SXVWNTZDZWRBKMZLDQ2IVFVANCNFSM4J67IDMA.

[cems2]

I have now exhaustively tried combinations of Flux and DiffEqFlux versions and find that nothing will work with Flux0.10.0 due to the Tracker Issue.

If I downgrade to Flux0.9.0 I lose the functor capability and have problems creating novel neuralODE layers with trainable parameters.

What is reccomended?

[ChrisRackauckas]

As mentioned, no version will work with Flux 0.10.0 right now. Flux 0.9.0 is fine for developing layers and everything, but it just uses Tracker. It will not be compatible until https://github.com/JuliaDiffEq/DiffEqFlux.jl/pull/96 is done, so if you want it, help finish the last Zygote issue! Basically, Zygote is incompatible with changing the parameters of a neural network manually right now, which is required for using an array for parameters.

[cems2]

Okay I get it now. Didn't understand. I'm new. Took me 3 days to figure out Treelike is the old functor.

On Dec 24, 2019, at 1:48 PM, Christopher Rackauckas notifications@github.com wrote:

As mentioned, no version will work with Flux 0.10.0 right now. Flux 0.9.0 is fine for developing layers and everything, but it just uses Tracker. It will not be compatible until #96 https://github.com/JuliaDiffEq/DiffEqFlux.jl/pull/96 is done, so if you want it, help finish the last Zygote issue! Basically, Zygote is incompatible with changing the parameters of a neural network manually right now, which is required for using an array for parameters.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/JuliaDiffEq/DiffEqFlux.jl/issues/99?email_source=notifications&email_token=ACRAR7UQNBP23G2AJN7UFN3Q2JYSBA5CNFSM4J67IDMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHTTNKI#issuecomment-568800937, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRAR7VYL3MYXNAT4VSDRC3Q2JYSBANCNFSM4J67IDMA.

[ChrisRackauckas]

Fixed on master. It'll be released in about a week.