search

SciML / Roadmap

A place for discussing the future of differential equations in Julia
0 stars 1 forks source link

Main directory clean up #6

Closed pwl closed 7 years ago

pwl commented 7 years ago

The current state of JuliaDiffEq is now completely overtaken by various splinters of DifferentialEquations.jl package. Splitting up the huge package like that is a move in a good direction but we should figure out how to organize all the pieces. I don't really like that these new packages all include DiffEq in their names. I presume this is a reference to DifferentialEquations.jl but some people might think that DiffEqTutorials.jl contains tutorials for all the packages from JuliaDiffEq instead of the tutorials only for DifferentialEquations.jl. This is just one example but there are many other packages with the same problem. We should either change the name or update the packages to refer to other packages. Sorry if you (@ChrisRackauckas) already had this in mind when you were refactoring the code but I haven't seen it explicitly stated anywhere.

If you plan to extend these new packages to interact with other members of DiffEq then I propose to drop the DiffEq from their names and update their README.md to at least point that's a work in progress. If they are meant to be purely parts of DifferentialEquations.jl ecosystem I would move them out of JuliaDiffEq to a different organization (but I have nothing against DifferentialEquations.jl staying in JuliaDiffEq).

mauro3 commented 7 years ago

I think the reason to have the DiffEq in the name is the "explicit" naming convention. These packages are in Metadata.jl thus having a package just named "Tutorials.jl" doesn't work.

I agree that currently it is a bit of a mess with all the packages but unless there are changes to Metadata then there is little which can be done.

pwl commented 7 years ago

Ah, now I see.

Then what about DifferentialEquationsTutorials.jl? I know it's long but this way it won't be confused with other stuff in JuliaDiffEq.

mauro3 commented 7 years ago

Yes, that might be an idea.

Also note that we can pin some repos to make them prominent, I now added DASSL.jl and ODE.jl: https://github.com/JuliaDiffEq

pwl commented 7 years ago

Pinning is a good idea, but it is only a temporary measure, not to mention that you can only pin six repositories. I don't think that JuliaDiffEq is a good place for an ecosystem that takes up 17 packages total and is this self-centered. Maybe moving the *DiffEq*.jl packages out to a different group and then moving them back one by one, with a proper review process, would be an option? As I said, we could leave DifferentialEquations.jl where it is and just move the other packages.

Another solution would be to set up a webpage with an index of packages just like www.juliadiff.org did. This way we could live with the mess in the JuliaDiffEq group.

mauro3 commented 7 years ago

No, they are here to stay, those were DifferentialEquations.jl packages, which itself is part of JuliaDiffEq and is its most popular package. Also @ChrisRackauckas is the only really active member of JuliaDiffEq!

The re-organisation in my mind makes sense, in particular if the long-term goal is to make part of them more widely usable. Chris and I were chatting over in #5 about a unified interface to ODE/DAE solvers. If that is done correctly other high-level parts of the DifferentialEquations.jl universe (such as DiffEqBenchmarks.jl) should become usable to any package implementing that interface, e.g. ODE.jl. That should give the JuliaDiffEq-organisation a bit a more a inclusive feel for none-DifferentialEquations.jl packages/packagers. But it will also take some time as Chris is likely the only person actually doing the work...

ChrisRackauckas commented 7 years ago

[Oh hey, you can pin 6 boxes? Thanks for fixing that. The tutorials should get boosted up and added there as well I think. See the part on that package]

I plan to diagram the packages like www.juliadiff.org . In fact, the only reason why I haven't made a website yet is because.... I haven't found a good looking template hahaha (not joking, check the JuliaLang Gitter). There will be a blog post too. I am waiting for the release to do the blog post it will confuse people if "the blog post isn't true yet". It is quite a bit, but here's what it looks like:

First of all, the DiffEq was appended to everything for sake of naming. You can see the naming discussions in METADATA with Tony on ParameterEstimation.jl not being appropriate because it's DiffEq related, so DiffEqParamEstim.jl was suggested by @musm, etc. A general naming scheme was developed: DiffEq* for add-on components, *DiffEq for solvers (they also just read better like that, DiffEqDevTools and OrdinaryDiffEq). This is what Tony agreed to and what we went with.

All of these do use the entire DiffEq ecosystem (they use solve on an ODEProblem, not anything specific to them). Most are not a work-in-progress, already fully functional and registered (indeed, most were parts of DifferentialEquations.jl before the modularization). This is part of why I wrapped all the other algorithms into DifferentialEquations.jl! You can use parameter estimation tools on any DifferentialEquations.jl (well, OrdinaryDiffEq.jl) algorithm, which includes the algorithms for ODE.jl's PR49, Sundials, ODEInterface, etc. Of course, there are some hiccups in there. For example, I explained earlier that PR49 and the others don't support ForwardDiff directly onto the solver, and so build_lsoptim_objective won't work for the non-native OrdinaryDiffEq.jl algorithms. You can find other hiccups in there, whereas I know that the OrdinaryDiffEq.jl algorithms are fully compatible.

Review

Maybe moving the DiffEq.jl packages out to a different group and then moving them back one by one, with a proper review process, would be an option?

If you're willing to review them, start reviewing them and leave issues. Some of these add-ons are newer, others have been there for months without anyone choosing to "review" them (well, there have been issues filed by users, and Gitter chats. That's a form of review that shouldn't be discounted). Personally, I really dislike this idea that something has to be "reviewed" by some other developer even when no other developer is around to do it or wants to take the time to do it. It's been a few days, what do you think is the solution to PyDSTool's reference counting issue? What do you think of the approach LsqFit results in DiffEqParamEstim? It's not like these things are secret: there's are issues (check PyCall, Optim, LeastSquaresOptim, etc.) and days of Gitter chats (check the JuliaLang log) devoted to them. I understand if you don't have time, most people don't/wont make the time because this isn't what they are doing or want to do.

In addition, a lot of the large changes were asked for by users, and many of the APIs were agreed on in discussions with others. These discussions have all been open and public (the only non-public discussions I will mention later), so again, you could have put in your input like many others have. And "post-publication review" is still an option: leave an issue. However, I hope it's quite clear that I am staunchly against halting development for you to review when you clearly don't want to put the time and effort into doing so (which is fine, I understand that we all have limited hours in a day!).

This is why I stay away from contributing to Base and "public packages" in the first place: people tend to be more willing to say that they will do work and "help bikeshed" than they are to review code and fix up their PRs. Instead of just pushing through that, I think we should organize to accommodate this. That transitions nicely into the main point about organizational structure...

ChrisRackauckas commented 7 years ago

This next part is significant enough I want it to be its own post:

Organizational Structure

I propose a distributed organizational structure, akin to what's truly worked in JuliaStats, following the example of how Amal.jl turned out in JuliaMath and others, and following the path that JuliaML is heading (if not explicitly, see their Roadmap). The structure is as follows:

Common Interface

A common distributed interface. Something like the DE solve interface, with some modifications (tspan in the problem object, etc.). See https://github.com/JuliaDiffEq/Roadmap/issues/5. We're working on it (indeed @mauro3 that last suggestion still isn't general enough, and the defaults are not quite sane, but I think it's pointing in a direction where I can get there. I'll have a prototype together soon enough). I say distributed interface as a phrase for the fact that it won't be "top-down imposed". With OrdinaryDiffEq.jl, I explicitly wrap each component package. With the distributed interface, there will be a well-documented way for you do "plug into the ecosystem"- you define a solve method and now your algorithm is available to all who use alg = ... or whatnot. I will then setup this distributed interface with Sundials.jl and ODE.jl, and have a wrapper over ODEInterface.jl (I'm not sure he's going to cooperate... but it's okay we can work around that and just wrap it). You can handle PR49 (it shouldn't be much work).

Package-Oriented Development, Distributed Maintenance

The reason why the interface matters is for this goal. Differential equations is a huge subject. What I want to do is encourage everyone to to contribute their work to the ecosystem. Traditionally this has meant contributing to a public package (ODE.jl or DifferentialEquations.jl). However, this heavily discourages non-developers, and the primary researchers are likely not software developers.

I want to encourage others to make/contribute packages, even if they do one thing. If your one thing is exponential Runge-Kutta methods, or a certain type of PDE, I encourage you to make a separate package and have it plug into the ecosystem through the interface. That way, anyone who uses the DiffEq ecosystem has a choice to use your algorithms! The deal here is that, if one transitions to JuliaDiffEq, one gets to keep full admin controls and "final say". The JuliaDiffEq members only are allowed to perform maintenance (i.e. update the tests to be the right releases, fix how things compile, fix depwarns) without review: the rest is all has to go through "whoever's in charge". For accepting this, you get first-class status in the DE universe. Many reasons for this any I think it's a good agreement for both sides:

A good example is DASSL.jl. I am quite unfamiliar with DAEs and don't plan on touching them very much, but the AlgebraicDiffEq.jl interface is what I hope to see as the distributed DAE interface. Then you can have your DASSL.jl, and have it plug into the common interface, and people can easily choose the algorithm if they please. You can even keep your name on it, and choose how people cite it. On the other hand, now that it's been transferred, I will help maintain it: update the tests to v0.5, get rid of depwarns, and keep it up-to-date (but not touch the core algorithm except through PRs). When it's there and plugs into the common interface, I'll add it to the DifferentialEquations.jl metapackage. Then it will be ready for testing: DiffEqBenchmarks.jl will test against the other wrapped DAE solver packages (currently just Sundials.jl) to determine a way for algorithm defaults.

I hope that this organizational structure helps pull in people who "just want to do their thing", and make the sum of their things "the thing". That said, there's one last pressing question

The Status of Public-ness

My Path

I think that this structure will help pull the distributed efforts together, while still letting me do my thing. One other thing I want to note is the commitment to playing nice. I know that in some sense as a developer in ODE, you saw me spring to life in the Julia-sphere and rampage through. I'm sorry I didn't start using Julia sooner. That said, I think the best thing to do is to be clear with any and all recommendations, and I should help other developers know how to satisfy these.

So I want to be clear that the "standard ODE solvers" are a bad showing of what I am planning on doing: I went through and re-built them from scratch to have the best possible performance I could benchmark, and the set of features necessary for an expansive ecosystem (for usage as the base for DDE solvers, usage in bifurcation plotting and parameter estimation, backends for PDE solvers, etc.). Thus I spent the summer perfecting these implementations because of how important these are. Because of this, I know the benchmarks and feature-differences vs PR49/Sundials/ODEInterface, and know that I can only recommend the OrdinaryDiffEq implementations for non-stiff ODEs (ODEInterface and Sundials have some advantages for stiff ODEs, see below). I know that in some ways it almost looks like it invalidates the work of PR49, but I think it was pretty clear that the only way to show people my whole vision and why each part was important was to, well, just do it!

That said, in order to not step on any toes, I think it will be helpful for me layout my "warpath". Here's what I have set out in-front of me for the next 6 months or so:

(of course, message me if you want to take on one of these and don't want to double up development efforts!) and lower in my priority list:

What I also need to do is clearly document "how to develop algorithms fully compatible with the DiffEq ecosystem". As mentioned in the Gitter chat, this isn't necessarily easy:

It's perfectly fine for algorithms to be compatible with only some of this. The documentation can be Plots.jl-like. Full support is recommended to be, well, recommended. However, I want to encourage all contributions because, in the full realm of DiffEqs, who knows what'll be useful to you?

mauro3 commented 7 years ago

@ChrisRackauckas: thanks for taking the time to write this up. A good plan. Thanks a lot for putting so much effort into this!

Just a few minor thoughts/remarks:

ChrisRackauckas commented 7 years ago

It would be good to have a battery of tests which can establish which level of compatibility a solver has. This is probably on your list.

Yes. I need to find out how to use testsets so that way the tests will display the compatibility, so it more open than just "hidden knowledge". When I say something is supported, I always have a barrage of tests for it. So for units, I have something like this: https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/master/test/units_tests.jl#L37 . Each of the features has something similar. Here you see that tests pass on any non-implicit algorithm: this is because NLsolve isn't units-compatible. Since all of the algorithms have this common interface, it's easy to just loop through and see what throws errors with rational timesteps, what throws errors with units, what isn't ForwardDiff-able, etc (though, after programming these things, it's pretty clear in advance what will be compatible and what won't be. You have to pretty far out of your way to make some of these things compatible, since even for most standard operations something like Unitful numbers will still throw errors).

What is the reason you split the ODE and DAE solvers into two packages? If anything, isn't the split more explicit vs implicit methods, with most implicit methods easily adaptable to handle DAEs?

They can be the same solver, but I think it's clear this should be developer-burden, not user-burden. To a user, a DAE is a problem defined by a function f(t,u,du,out) where out is the residual at each step (so this is written in-place), and has two initial conditions: u0 and du0. This is different from an ODE which is defined as f(t,u,du) with u0.

I think it's clear that we shouldn't make ODEs defined by f(t,u,du,out) since, while technically ODEs are a subset of DAEs, they are a very important subset which also will have a lot of users who won't be familiar with what a DAE is. So on the user side, ODEs != DAEs.

But on the developer side, there can be tons of interactions between solvers for different types of equations (in fact, this very idea is what DifferentialEquations as a suite was made for). So for example, DASSL.jl is essentially a BDF method. One can implement a BDF method for ODEs by:

Since closures are compiled in Julia now, I think you can just define f(t,u,du,out) using the function in the ODEProblem, and I think it should essentially ignore the initial condition on du? I am not as familiar with DAEs, but that sounds about right. Then you just expose this method through the common interface, and to users it's just a "good for stiff equations" method, where the true implementation as a DAE solver exists in documentation but doesn't need to be highlighted to the user.

This is already something happening in the ecosystem. I am building delay differential equation methods (DDEs) by building an ODEProblem (and using the interpolant and event handling of ODEs). I will be re-doing the PDE methods to build ODEs and use the ODE solvers. SDE solvers can make PDEs which solve using the ODE solvers. Etc. Building the problem types is super cheap since you're usually not copying anything: you're just setting another pointer to the same initial condition, building a new function. This cross-compatibility is all already built in (it was the leading design constraint), but it's developer-side, not user-side. I think that's the right way to handle it, unless you have a really good counterargument. Savvy users might just need a few examples in the ecosystem.

Extra stuff.

BTW, let me know if you guys plan on tackling this "native Julia BDF" thing. I'll take it off my list if you aren't. I'm not too familiar with implementing BDF methods so it would be quite a bit of work for me (and it's not clear to me how they generalize well to SDEs, which reduces my interest). It would be something I'd be interesting having multiple different implementations of too: there's so many ways to do it, it's hard to know what's good! But I really don't want to take the time to give it a shot right now: not for another year or so at least.

Instead of BDF methods, I'm doubling down on Rosenbrock methods. This is because ParameterizedFunctions can let me symbolically pre-calculate things like the Jacobian and inverse W, and because explicit methods seem to be required in order to handle some of the wackiest Julia types. I think this would be a good way to split work, and the Rosenbrock methods should come out to be a good stiff workhorse with the BDF methods taking the lead for when the function evaluations cost more. But who knows: being able to mix all of these ways of implement (using symbolic calculations on ASTs, automatic differentiation, etc.) really throws all of the old rules out the window. I think that the methods which most effectively makes use of the Julia package ecosystem will really shine and be far better than even LSODE/CVODE/Radau!

pwl commented 7 years ago

@ChrisRackauckas thanks for a great write up, this is basically what I meant by a review. Maybe I somehow missed it but I have never seen any discussion in JuliaDiffEq or any Roadmap issue about splitting up DifferentialEquations.jl, hence my concern. Even a simple issue announcing what you are planning to do would be enough and give people chance to react. You mention Gitter chat but it is not something you can follow and browse in an organized fashion, and the discussion in METADATA is another example of information being publicly accessible but not easy to find (I guess you meant https://github.com/JuliaLang/METADATA.jl/pull/6915).

I'm only saying this from my perspective. Even if I currently don't have the time to contribute It doesn't mean I don't try to follow the discussions, and some of these decisions escaped me. Admittedly I haven't been following https://github.com/JuliaDiffEq/DifferentialEquations.jl/issues/59, where the discussion/announcement seemed to have taken place but I still believe there should be at least a short note or a link in the Roadmap. I would love to see more transparency in the future, otherwise what is an organization good for? Don't get me wrong, I'm not suggesting stricter control over the packages or some such, but this is a huge change that would likely go under my radar if I didn't complain about the multitude of new "DiffEq" packages (which makes more sense to me after your explanations).

As for the organizational changes I mostly agree with your basic ideas but I will have to catch up with the details on the interface design. As you said, this would invalidate the work we done in PR49, which I personally hoped to become the new standard interface for implementing the basic solvers. PR49 was never about implementing newer or faster solvers but more about designing a flexible and simple interface, so if the plan is to wrap it up with another layer of interfaces anyway, then maybe there no point in having PR49 at all. But this is the topic for #5.

ChrisRackauckas commented 7 years ago

Admittedly I haven't been following JuliaDiffEq/DifferentialEquations.jl#59, where the discussion/announcement seemed to have taken place but I still believe there should be at least a short note or a link in the Roadmap. I would love to see more transparency in the future, otherwise what is an organization good for?

Yes, in retrospect I probably should've noted more "roadmap" discussions in this roadmap.

https://github.com/JuliaDiffEq/DifferentialEquations.jl/issues/47 https://github.com/JuliaDiffEq/DifferentialEquations.jl/issues/59

Things happen fast though: you can see that I said wasn't going to do this until Pkg3 since the Juleps just came out, and 5 days later I did it because I found the time.

PR49 was never about implementing newer or faster solvers but more about designing a flexible and simple interface, so if the plan is to wrap it up with another layer of interfaces anyway, then maybe there no point in having PR49 at all.

Let's take this over to https://github.com/JuliaDiffEq/Roadmap/issues/5#issuecomment-258611941.

ChrisRackauckas commented 7 years ago

I think this has concluded.