Get Rubi working

[rocky]

At the last SAGE Days conference in 2020, someone suggested running Rubi under Mathics.

I started with version 4.16.1.0 and hacked that to at least load. This is in the Rubi branch. (Not surprisingly some changes were needed to Mathics to get this far.)

However that code doesn't run as far as I can tell - yet.

Your challenge should you decide to work on this is to finish the job. You can change either Mathics or Rubi.

One thing I haven't found is a test suite that I can use similar to what is done in Combinatorica to guide finding Mathics bugs so they can be fixed.

[AlbertRich]

The Rubi test-suite contains 72,000+ integration problems and their solutions, as well as an integration test program.

[rocky]

Thanks for the encouragement. It really helps to know whether the authors are behind this.

As mentioned on gitter:

The problems that I understand I have worked around. For example I recently added $InputFilename apparently this doesn't handle scoping the way it is supposed to so in addition to that I changed System<tick>Private<tick>$InputFileName to System<tick>$InputFileName. And the progress bar wasn't progressing so I repaced those with Print[]s. That's the easy stuff.

Right now things load, but Int[] doesn't do anything.

It's not a question of having zillions of test cases, rather than having some test cases that go from easy to complex so that when there is a failure it is testing a small isolated part of the code.

Thanks in advance and for creating and working on Rubi. It looks really cool...

[AlbertRich]

Thanks for your kind works about Rubi.

After loading Rubi into Mathics, what does Length[DownValues[Int]] evaluate to?

[rocky]

... Loading ShowStepFormatting.m
In[2]:= Length[DownValues[Int]] 
Out[2]= 3
In[3]:= DownValues[Int]
Out[3]= {HoldPattern[Int[{Rubi`Private`u__}, Rubi`Private`x_Symbol]] :> (Int[#1, Rubi`Private`x]&) /@ {Rubi`Private`u}, HoldPattern[Int[Rubi`Private`u_, {Rubi`Private`x_Symbol, Rubi`Private`a_, Rubi`Private`b_}]] :> With[{Rubi`Private`result = Int[Rubi`Private`u, Rubi`Private`x]}, Limit[Rubi`Private`result, Rubi`Private`x -> Rubi`Private`b] - Limit[Rubi`Private`result, Rubi`Private`x -> Rubi`Private`a]], HoldPattern[Int[Rubi`Private`e_, Rubi`Private`x_, Rubi`Private`flag : Stats | Step | Steps] /; (Message[Int::oldFlag, Rubi`Private`flag] ; True)] :> Rubi`Private`flag[Int[Rubi`Private`e, Rubi`Private`x]]}

[AlbertRich]

There should be around 7000 Int rules.

[rocky]

Got it. One problem I have encountered (and I don't really know enough about WL to understand this stuff), is that I have found that there is a lot of "Protect" going on in Mathics that apparenlty doesn't exist in WL.

Get["/home/rocky/github/mathics/Mathics/mathics/packages/Rubi/IntegrationRules/1 Algebraic functions/1.1 Binomial products/1.1.1 Linear/1.1.1.1 (a+b x)^m.m"]
<ctions/1.1 Binomial products/1.1.1 Linear/1.1.1.1 (a+b x)^m.m"]
SetDelayed::write: Tag Int in Int[1 / x_, x_Symbol] is Protected.
SetDelayed::write: Tag Int in Int[x_ ^ (m_.), x_Symbol] /; FreeQ[m, x] && NeQ[m, -1] is Protected.
SetDelayed::write: Tag Int in Int[1 / (a_ + (b_.) x_), x_Symbol] /; FreeQ[{a, b}, x] is Protected.
SetDelayed::write: Tag Int in Int[((a_.) + (b_.) x_) ^ m_, x_Symbol] /; FreeQ[{a, b, m}, x] && NeQ[m, -1] is Protected.
SetDelayed::write: Tag Int in Int[((a_.) + (b_.) u_) ^ m_, x_Symbol] /; FreeQ[{a, b, m}, x] && LinearQ[u, x] && NeQ[u, x] is Protected.
Out[4]= $Failed

However when I change that file and add Unprotect[Int] I see:

Get["/home/rocky/github/mathics/Mathics/mathics/packages/Rubi/IntegrationRules/1 Algebraic functions/1.1 Binomial products/1.1.1 Linear/1.1.1.1 (a+b x)^m.m"]
<ctions/1.1 Binomial products/1.1.1 Linear/1.1.1.1 (a+b x)^m.m"]
In[7]:= Length[DownValues[Int]] 
Out[7]= 8
In[8]:= DownValues[Int]
Out[8]= {HoldPattern[Int[1 / (a_ + (b_.) x_), x_Symbol] /; FreeQ[{a, b}, x]] :> Log[RemoveContent[a + b x, x]] / b, HoldPattern[Int[1 / x_, x_Symbol]] :> Log[x], HoldPattern[Int[((a_.) + (b_.) u_) ^ m_, x_Symbol] /; FreeQ[{a, b, m}, x] && LinearQ[u, x] && NeQ[u, x]] :> 1 Subst[Int[(a + b x) ^ m, x], x, u] / Coefficient[u, x, 1], HoldPattern[Int[((a_.) + (b_.) x_) ^ m_, x_Symbol] /; FreeQ[{a, b, m}, x] && NeQ[m, -1]] :> (a + b x) ^ (m + 1) / (b (m + 1)), HoldPattern[Int[x_ ^ (m_.), x_Symbol] /; FreeQ[m, x] && NeQ[m, -1]] :> x ^ (m + 1) / (m + 1), HoldPattern[Int[{Rubi`Private`u__}, Rubi`Private`x_Symbol]] :> (Int[#1, Rubi`Private`x]&) /@ {Rubi`Private`u}, HoldPattern[Int[Rubi`Private`u_, {Rubi`Private`x_Symbol, Rubi`Private`a_, Rubi`Private`b_}]] :> With[{Rubi`Private`result = Int[Rubi`Private`u, Rubi`Private`x]}, Limit[Rubi`Private`result, Rubi`Private`x -> Rubi`Private`b] - Limit[Rubi`Private`result, Rubi`Private`x -> Rubi`Private`a]], HoldPattern[Int[Rubi`Private`e_, Rubi`Private`x_, Rubi`Private`flag : Stats | Step | Steps] /; (Message[Int::oldFlag, Rubi`Private`flag] ; True)] :> Rubi`Private`flag[Int[Rubi`Private`e, Rubi`Private`x]]}

Would I be able to see that one of those in fact worked?

Also where would I find information about what should be protected and what should not be protected? When Get[] is run and inside scopes?

I will add an issue to address the bug(s) but right now I don't know enough to intelligently describe what's wrong.

[rocky]

@AlbertRich Assuming you have a better handle on understanding from this what's up and what is wrong, perhaps you could open issues around what needs to be addressed?

Thanks.

[AlbertRich]

Unfortunately assuming I "have a better handle on understanding" on what's up is not a good assumption. I'm staying focused on enhancing Rubi's mathematical abilities.

A little background on the Rubi Project: Since 2008 I have been deriving the integration formulas and then transforming them into the pattern matching rules Rubi uses to integrate large classes of mathematical expressions. Beginning in the summer of 2017, Patrick Scheibe helped me create a website for Rubi on GitHub. Also he enhanced the display of the steps Rubi uses to integrate expressions, and wrote the code to efficiently load this enhanced version of Rubi. I am guessing this code for loading Rubi and these graphical enhancements uses Mathematica functions not fully supported by Mathics.

So, I propose you first implement Rubi in Mathics using the "classic" command-line interface I use to develop Rubi. Then once that's running smoothly on Mathics, you can consider writing a more fancy interface.

If you agree to this approach and are serious about following through on it, I will take the time required to create a command-line version of Rubi I will email you as a zip file. I look forward to your thoughts.

Aloha from Hawaii, Albert

[rocky]

Overall this approach sounds reasonable.

It is sort of similar to the approach I am taking with Combinatorica which will be added in the next mini update. With that what has been helpful is to use older versions of that, since that is the smallest consistent code base and uses the fewest fancy features, such as function compilation. With that, I have been going through the 1st edition of the Combinatorica book, section by section.

Right now I've got to page 30 or so out of a 130 page book. The next edition of the Combinatorica book is twice as large in pages and in code.

If you can create the zip file for a CLI interface of maybe some smaller set of consistent routines, along with the examples for those that would be ideal. For combinatorica right now I have about 60 or so tests. So if you can come up with say 100 tests (72K is impressive, but not helpful at this stage), I'll look at that.

If that is successful then, we can keep going. As you probably are aware: at the beginning you want to get 5 tests working. Then 50, then 500, then 5000, then 72K.

As for a timeline (and really there is none), if I am lucky, after the upcoming release we can have something ready. The last release I wanted to have some combinatorica code out, but that was too much for that release.

Also as you probably are aware, open-source code is a little like the fable of stone soup. In the beginning it really is a stone and water. But depending on the assistance of others it can be come a full-bodied source of substance that others want to participate in.

So if we can get help here of any kind, e.g. just classifying what bugs need to be fixed to make Rubi work, then the completion time will be sooner. Of course, that doesn't mean you should do this, just that any help would be desirable. So tell your friends!

Thanks.

[rocky]

Aloha from Hawaii,

What island?

Aloha from NYC (the island of Manhattan)

[AlbertRich]

Ok, I will create a zip file containing Rubi's integrations rules along with an .m file that loads a CLI version of Rubi when you issue a Run All Code (i.e. Evaluate Initialization Cells) command.

[AlbertRich]

I live on Kona (west) side of Hawaii (aka the Big Island) in The Kumulani condos. For a break from the computer, I do volunteer conservation work at the Waikoloa Dry Forest Preserve:

On the other side of the island, Madame Pele just erupted again after being quiet since the summer of 2018 as these USGS videos and photos show.

Speaking of NYC, in the winter and spring of 1997, I took a memorable and most enjoyable 6 month "sabbatical" in the City. I lived in a 3rd floor apartment on the West Side near the AMNH where I volunteered and spent many hours wandering around Central Park.

From your GitHub page, I was delighted to see, among many other things, you are a Lisp enthusiast. My first, and still favorite, programming language. I'm the author of muLISP with versions written in 8080, 8086 and 80386 (Pentium) assembly languages. It along with the Derive CAS was sold from 1979 to 1999 by my company Soft Warehouse, Inc.

Albert

[rocky]

Wow! That is very neat and sounds like a lot of fun.

I am not as big a LISP nerd as a guy I used to work with at IBM Research: Greg Chaitin. See for example: https://www.springer.com/gp/book/9781852334178#aboutBook

For a break I go out and balance bicycles in our city bike share - moving bicycles where the racks are full to racks where they need bicycles. (This condition can be found on the side of a hill such as the one in Brooklyn overlooking the Brooklyn Bridge.) I know and love the AMNH well, it is very nice for a hostel. In the park and along the river if you know where to look you might find a periguine falcon. (They love steep heights and pidgeons).

But I live not too far away from Chinatown which is always fun to hang around when there isn't a Pandemic.

If you come again to NYC I'll show you the non-tourist side - the immigrant side which is what makes this city what it is. NYC is a great place to live - but I wouldn't want to visit.

[rocky]

I should have also mentioned that I tried disabling Protect[] altogether in a test run. It helped a little but not as much as I would have hoped.

There are some problems in Get[] especially when they are nested, and probably how we treat contexts and scopes.

Someone like @mmatera or @GarkGarcia who knows WL better and has access to a real WL interpreter is better suited for guiding us here.

[AlbertRich]

To run Rubi's rational and algebraic function integrator with a command line interface:

• extract the contents of RubiCLI.zip into a directory of your choice;
• open the file RubiCLI.m in Mathematica or Mathics; and then
• issue an Evaluate Initialization Cells command that will define 3175 integration rules.

After Rubi is loaded, three examples will show how to integrate an expression with and without showing the steps. I recommend running it on Mathematica first in order to see how it is supposed to load and run.

Note:

• The integration rules in this version of Rubi are still under active development.
• This version of Rubi does not include rules for integrating elementary and special functions.
• It is provided to enable the Mathics community to add the functionality required to fully support Rubi.
• It should not be widely distributed until Mathics fully supports Rubi and is able to successfully pass Rubi's entire test-suite.

When that time comes, please coordinate with me to ensure you have Rubi's latest rules and test-suite. I look forward to following your progress.

Albert

[rocky]

I have downloaded and extracted this. Thanks — it is a lot more tractible.

issue an Evaluate Initialization Cells command that will define 3175 integration rules.

https://reference.wolfram.com/language/ref/menuitem/EvaluateInitializationCells.html seems to suggests this is a menu command in some user interface. But we were are talking CLI, remember?

Also Mathics doesn't support Notebooks and it doesn't have a NotebookDirectory[] function. That's okay though becuse I just changed that to a hard-coded and not-general path.

But this brings to mind that you would do us a big favor if you installed Mathics so you get some sense fo what we're talking about in terms of what is available.

A pip install Mathics3 is one way or there is a conda forge way or there is a docker container that can be run. See Installing and Running for the various ways.

If you don't want to install anything you can look at the PDF. Again, that should give some sense of what functions are there and what is not.

However unless you install and run from git (which is another way), note that a release in a few days will happen and that should address more of the challenges with running Rubi in particular.

When I ran Get[] (or <<) the first time, After a much much longer period of time, I saw:

In[16]:= Length[DownValues[Int]] 
Out[16]= 3137

This is still way too much to start working on. Maybe its been a while since you have started a new project, and been in this position, but 10-20 is more what you want to start out with. I am seeing 1,000's of messages like:

StringReplace::strse: String or list of strings expected at position 1 in StringReplace[StringReplace[ToString[Extract[HoldPattern[Int[1 / ((a_ + (b_.) x_ ^ 2) ^ (1 / 3) (c_ + (d_.) x_ ^ 2)), x_Symbol] /; FreeQ[{a, b, c, d}, x] && NeQ[b c - a d, 0] && EqQ[b c + 3 a d, 0] && NegQ[b / a]] :> With[{q = Rt[-b / a, 2]}, Simp[q ArcTan[Sqrt[3] / (q x)] / (2 2 ^ (2 / 3) Sqrt[3] a ^ (1 / 3) d), True] + (Simp[q ArcTan[Sqrt[3] (a ^ (1 / 3) - 2 ^ (1 / 3) (a + b x ^ 2) ^ (1 / 3)) / (a ^ (1 / 3) q x)] / (2 2 ^ (2 / 3) Sqrt[3] a ^ (1 / 3) d), True] + Simp[q ArcTanh[a ^ (1 / 3) q x / (a ^ (1 / 3) + 2 ^ (1 / 3) (a + b x ^ 2) ^ (1 / 3))] / (2 2 ^ (2 / 3) a ^ (1 / 3) d), True] - Simp[q ArcTanh[q x] / (6 2 ^ (2 / 3) a ^ (1 / 3) d), True])], {1, 1}, Defer], InputForm], {_Symbol -> , _. -> , _ -> }], {Int[ -> Integrate[, Dif[ -> D[}].
StringDrop::strse: String expected at position 1.
StringDrop::strse: String expected at position 1.
StringJoin::string: String expected.
StringJoin::string: String expected.

and seeing this closer to one time is all I need to start addressing this problem.

Making a guess at what to keep, I guessed at keeping section 1.4 Binomical Produces and 1.9 Simplification Rules and removing the other sections.

With that I now have:

In[3]:= Length[DownValues[Int]] 
Out[3]= 1749

Still an order of magnitude larger than ideal, but more tractible.

After Rubi is loaded, three examples will show how to integrate an expression with and without showing the steps.

Ok, but what are the 3 examples. And are they simple ones? Can they be made simpler?

I recommend running it on Mathematica first in order to see how it is supposed to load and run.

I don't have Mathematica of any version installed. I bought Wolram's Version 5 in book form and I have a web browser connected to the Internet though. And I do not intend on registerering for anything. Thanks for your understaning and patience here.

In[4]:= Int[x, x]
$RecursionLimit::reclim: Recursion depth of 200 exceeded.
Out[4]= $Aborted

(I hope this gives you the idea of what I mean by "simple" though.)

Ok. So let's try:

In[5]:= Int[x^2, x]
Out[5]= TimeConstrained[TimeConstrained[TimeConstrained[False TimeConstrained[1, 5., 1] ^ 2, 1.66667, False TimeConstrained[1, 5., 1] ^ 2], 5., False TimeConstrained[1, 5., 1] ^ 2] / (TimeConstrained[1, 5., 1] ^ 4 Log[TimeConstrained[1, 5., 1]]), 5., False / (TimeConstrained[1, 5., 1] ^ 2 Log[TimeConstrained[1, 5., 1]])] /; !FalseQ[False] /; !FreeQ[1, x] && PolynomialQ[x, x] && PolynomialQ[1, x]

In[6]:= ?TimeConstrained
Out[6]= Null

TimeConstrained is defined in mathics/Mathics/mathics/packages/Rubi/Rules/IntegrationUtilityFunctions.m. I did a Get[] on that although this was done initally too, and this doesn't change anything.

After the upcoming 1.1.1 release I will start investigating what is going wrong with the string functions. But whittling this program down to something small and basic would be much appreciated.

And instructions for how to debug - I see for example Debug routines.m but I have no idea how to get that to kick in right now.

Note:

The integration rules in this version of Rubi are still under active development. This version of Rubi does not include rules for integrating elementary and special functions. It is provided to enable the Mathics community to add the functionality required to fully support Rubi. It should not be widely distributed until Mathics fully supports Rubi and is able to successfully pass Rubi's entire test-suite. When that time comes, please coordinate with me to ensure you have Rubi's latest rules and test-suite.

Noted - Absolutely!

I understand where you are coming from. I am sure I'd feel the same way. What I don't think you appreciate is that this is very far from working right now. And it is not simply a matter of waving a magic wand to fix things.

Unless you or others get involved it may be a year or so before everything works.

And based on my experience so far and with Mathics, there may be performance problems as well.

I am happy to add you as a member of the project ,whenever you want, so that you can better and more directly control what goes in and how. Until then I'll act as your minion here.

---

“Everything should be made as simple as possible, but no simpler” — paraphrase of quote by Einstein

[AlbertRich]

Rocky, like I said before "I'm staying focused on enhancing Rubi's mathematical abilities."

Am struggling now to get out a major new release of Rubi that's been in the works for over 2 years now. Once that's done, I will tackle redesigning Rubi to use an if-then-else decision tree, rather then pattern matching, to select which integration rule to apply to a given integrand. Preliminary testing indicates that will improve system performance by almost 2 orders of magnitude. Also it will make it much easier to port Rubi to CA systems that don't support efficient pattern matching.

I'm not going to get involved in any new projects that will distract from that work. So sorry, but it will be up to the Mathics community to enable it to support Rubi.

Albert

[rocky]

Ok - sounds like interesting work and an interesting plan.

Perhaps whoever it was who suggested this at the SAGE math talk, will get involved. This issue can hang out for a couple of years.

At some point though I'll update the Rubi branch for this reduced code. But as you indicated, it won't get merged in though until you want it merged.

[AlbertRich]

Great! To keep our projects in sync, as soon as the new version of Rubi is thoroughly tested and ready for release (hopefully in a couple months), I will send you a RubiCLI.zip with a updated set of rules including ones for integrating elementary and special functions.

I just had an idea: You can significantly reduce the demands on Mathics when loading RubiCLI by avoiding the extensive code modifications done to enable the system to show integration steps. In the "Set control variables" section of the file RubiCLI.m change the assignment

  `$LoadShowSteps=True;`   to   `$LoadShowSteps=False;`

Please let me know if this helps.

[AlbertRich]

The RubiCLI.m file in RubiCLI.m.zip sets $LoadShowSteps to False and makes a couple other minor changes.

[rocky]

I have merged that in with what I have. All of this can be found in https://github.com/mathics/Mathics/tree/Rubi

Please especially look at https://github.com/mathics/Mathics/tree/Rubi/mathics/packages/Rubi to make sure I expressed adequately what you wrote above.

When I try Int[Sqrt[a - b*x^2]/x^2, x] I get a max recursion depth exceeded error.

When I try Int[5, x] I get

 TimeConstrained[TimeConstrained[TimeConstrained[25 x, 1.66667, 25 x], 5., 25 x] / 5, 5., 5 x]

which is similar to what i got before.

Thinking about this, why don't we wait until the next Rubi release is done, and when you start redoing this in the decision tree method? The Mathics code may be in better shape, and it may be easier to get a sense of how to start off simple.

I suspect in doing the decision tree work, you'll have a sense of how to proceed in small steps. Maybe you could just start developing that in Mathics. I also suspect that things that work in Mathics will work in WL.

[rocky]

TimeConstrained[TimeConstrained[TimeConstrained[25 x, 1.66667, 25 x], 5., 25 x] / 5, 5., 5 x]

Looking at this closer, it looks like TimeConstrained is not defined to match this. I think though that the result 5 x which appears at the end is however correct. So YAY! We can integrate this one small case!

[rocky]

Ah! TimeConstrained is a WL function that Mathics doesn't have yet.

I have created issue #1059 to track adding this.

[AlbertRich]

Thanks for including my cautionary notes in the README file for Mathics' package for Rubi.

Congratulations on achieving "First Light" for Rubi running on Mathics!

For Rubi, just define TimeConstrained to return its first argument.

What's the current value of Length[DownValues[Int]]?

[AlbertRich]

Put these in RubiCLI.m:

TimeConstrained[expr_, time_] := expr; and TimeConstrained[expr_, time_, fail_] := expr;

[rocky]

Adding the TimeConstrained[] above is a useful workaround for now.

Implementing TimeConstrained[] properly is probably not that hard to do. But I will do it after the next release.

In[5]:= Length[DownValues[Int]] 
Out[5]= 359

which is plenty for now...

However, I haven't been able to get anything other than integration of a constant or a variable: Int[y, x] works but Int[2 y, x] doesn't nor does Int[y + 3, x]

So using rules 1.1.1.1 only what should also work? How about using 1.1.1.1 up to 1.1.1.4?

Thanks!

[AlbertRich]

No, you can't just load a few rule files and expect anything to work. Are you not able to define 3175 Int rules by loading all the rule files for algebraic functions in the order specified in "Load Rubi.m"?

[AlbertRich]

I'm off to bed. Zzzzzzzzzzzz

[rocky]

Some miscellaneous information. The 1.1.1 release is about half done, so in a day or so I may be able to look at this.

@mmatera who is very capable will address the lack of a TimeConstained[] even though we have your work-around it in the Rubi branch. Actually, I suspect it isn't a workaround as there are some situations where it is probably needed. But you'd know better than I about this.

If someone else wants to work on this and they are okay with dealing with 3K rules which may take a while to load and may produce thousands of error message, I'm okay with that. But I am not going to go back to the 3K or 7K Int rules.

It is hard to believe that the code can't be be subsetted to make this work say for just integrating polynomials, which would probably be more than plenty to start out with. Again 5 mastered well, then 50 which are under control, then 500, then 5K then the rest.