search

Macaulay2 / M2

The primary source code repository for Macaulay2, a system for computing in commutative algebra, algebraic geometry and related fields.
https://macaulay2.com
345 stars 231 forks source link

get Macaulay2 into Debian #286

Closed DanGrayson closed 3 years ago

DanGrayson commented 9 years ago

Doug Torrance dtorrance@monmouthcollege.edu is working on that now, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=439888 and https://groups.google.com/forum/#!topic/macaulay2/jxtKZFRDKDA .

mahrud commented 4 years ago

By the way, I asked Mike whether I could simplify the build of mathicgb and friends, since libtool and automake are causing problems. I assume you can adapt to that easily.

Is it required, for Debian, that a package such as mathicgb be able to make shareable libraries? I'd like to not bother with that at all.

@DanGrayson consider using the CMake build for memtailor/mathic/mathicgb. I wrote it this year, so you don't have to do the same. You can do testing as well as creating shared libraries directly as well.

d-torrance commented 4 years ago

By the way, I asked Mike whether I could simplify the build of mathicgb and friends, since libtool and automake are causing problems. I assume you can adapt to that easily.

Is it required, for Debian, that a package such as mathicgb be able to make shareable libraries? I'd like to not bother with that at all.

It's strongly encouraged. Library packages are generally expected by Debian policy to ship both static and shared libraries, with a few exceptions.

However, if the Debian package of Macaulay2 were to link against the static libraries instead of shared ones, it would raise a statically-linked-binary Lintian error and likely wouldn't be accepted.

Even if you do make changes to the memtailor/mathic/mathicgb builds, I can just hold off on uploading new versions to Debian until later, since everything works fine building Macaulay2 against the current packages.

DanGrayson commented 4 years ago

@DanGrayson consider using the CMake build for memtailor/mathic/mathicgb. I wrote it this year, so you don't have to do the same. You can do testing as well as creating shared libraries directly as well.

Okay, I'll try it! What would be the syntax for the command? There is presumably a way to pass such things as compiler flags and libraries to it, and a way to specify whether to build a sharable library.

DanGrayson commented 4 years ago

By the way, I asked Mike whether I could simplify the build of mathicgb and friends, since libtool and automake are causing problems. I assume you can adapt to that easily. Is it required, for Debian, that a package such as mathicgb be able to make shareable libraries? I'd like to not bother with that at all.

It's strongly encouraged. Library packages are generally expected by Debian policy to ship both static and shared libraries, with a few exceptions.

Are there other haters of libtool out there who do it a different way?

However, if the Debian package of Macaulay2 were to link against the static libraries instead of shared ones, it would raise a statically-linked-binary Lintian error and likely wouldn't be accepted.

Wouldn't that just be if the whole of M2-binary were statically linked? They'd have no way to tell whether a single library is linked into it statically.

Even if you do make changes to the memtailor/mathic/mathicgb builds, I can just hold off on uploading new versions to Debian until later, since everything works fine building Macaulay2 against the current packages.

No, don't bother, I found another way of working around it: I use rm -f /usr/local/lib/lib*.dylib. (The "gatekeeper" doesn't permit shared libraries inside *.dmg files on the Mac.)

d-torrance commented 4 years ago

Are there other haters of libtool out there who do it a different way?

I'm not really sure. Here's a wiki page with a little more about static linking in Debian, but I'm not sure if it answers your question.

Wouldn't that just be if the whole of M2-binary were statically linked? They'd have no way to tell whether a single library is linked into it statically.

Ah, you're probably right!

dimpase commented 4 years ago

a reviewer would look at the dependencies and wonder why you link in statically a gazillion of libraries.

On Fri, 10 Jul 2020, 14:57 Doug Torrance, notifications@github.com wrote:

Are there other haters of libtool out there who do it a different way?

I'm not really sure. Here's a wiki page https://wiki.debian.org/StaticLinking with a little more about static linking in Debian, but I'm not sure if it answers your question.

Wouldn't that just be if the whole of M2-binary were statically linked? They'd have no way to tell whether a single library is linked into it statically.

Ah, you're probably right!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Macaulay2/M2/issues/286#issuecomment-656690615, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJXYHF274IGO5ZXY2RLU73R24M5JANCNFSM4BHA62DQ .

d-torrance commented 4 years ago

I think the package is ready! I've asked for sponsorship to get it uploaded. So now the process is:

DanGrayson commented 4 years ago

Great!

And then it will migrate automatically into Ubuntu, Raspbian, etc.?

dimpase commented 4 years ago

Incidentally, we are asked to look at Rasbrian (which is a 32-bit OS for Arm64, whatever this means exactly) support for Sagemath. Flint builds there. At least one C++ library, libfplll, needed something ARM-specific, linking with libatomic library.

d-torrance commented 4 years ago

And then it will migrate automatically into Ubuntu, Raspbian, etc.?

Yup! I mentioned the Ubuntu migration schedule above. I'm not 100% sure about how Raspbian's migration's schedule works (and I can't find any good docs), but it looks like it stays in sync with Debian pretty well. For example, the newest versions of memtailor, mathic, and mathicgb I packaged a couple weeks ago are already in Raspbian testing.

d-torrance commented 4 years ago

Exciting update! The Macaulay2 Debian package has finished going through the sponsorship process and has now been uploaded to the NEW queue:

https://ftp-master.debian.org/new/macaulay2_1.16+git55.94c4b7d+ds-1.html

Now we wait. Sometimes, this takes days (e.g., topcom took about 2 days to go through the NEW queue) and sometimes this takes months (e.g., mpsolve has been there since the end of May ).

DanGrayson commented 4 years ago

Great news!

mikestillman commented 4 years ago

Yes, excellent news indeed! Thanks Doug!

On Oct 5, 2020, at 9:58 AM, Daniel R. Grayson notifications@github.com wrote:

Great news!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Macaulay2/M2/issues/286#issuecomment-703650356, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAY6R5JJ2WDQZWUT2VHDSDSJHGIRANCNFSM4BHA62DQ.

d-torrance commented 3 years ago

mpsolve is finally in Debian, so once Macaulay2 finally through the NEW queue, the following upload will have a working roots function.

d-torrance commented 3 years ago

Macaulay2 is now in Debian unstable!

https://tracker.debian.org/news/1185822/accepted-macaulay2-116git5594c4b7dds-1-source-all-amd64-into-unstable-unstable/

DanGrayson commented 3 years ago

Great!

d-torrance commented 3 years ago

Good news: Macaulay2 has migrated over to Ubuntu 21.04 (Hirsute Hippo)!

Bad news: All the builds are failing.

https://launchpad.net/ubuntu/+source/macaulay2

DanGrayson commented 3 years ago

I keep having occasional errors in ThreadedGB, too. The task/thread system has some problems. Maybe you could patch out the failing example.

d-torrance commented 3 years ago

At some point, Ubuntu retried the amd64 build and it was successful [1]. So Macaulay2 should be in the release of 21.04 for amd64!

[1] https://launchpad.net/ubuntu/+source/macaulay2/1.16+git55.94c4b7d+ds-1/+build/20212619

d-torrance commented 3 years ago

Status update: I'm doing regular builds of Debian packages of the development branch that work on amd64, i386, and mips64el. (I suspect that mipsel works too, but a qemu bug is preventing me from testing it on my machine.)

Since it looks like we've figured out #1554, I'm hopeful that we can get Macaulay2 to build on the other official Debian and Ubuntu architectures.

My tentative plan is to wait until the release of 1.17, package that, and then ask for sponsorship for an upload to Debian. The process should go much more quickly now that it's already been through the NEW queue. As long as it migrates to testing by Feb. 12 (https://release.debian.org/bullseye/freeze_policy.html), it should get into the official Bullseye (Debian 11) release. Similarly, as long as it migrates over to Ubuntu by Feb. 25 (https://discourse.ubuntu.com/t/hirsute-hippo-release-schedule/18539), it should be in the official Hirsute Hippo (Ubuntu 21.04) release (1.16 already is on amd64).

DanGrayson commented 3 years ago

Great!

d-torrance commented 3 years ago

For the first time, the Macaulay2 package has built successfully on every architecture on my Ubuntu 21.04 PPA!

https://code.launchpad.net/~profzoom/+recipe/macaulay2-daily-hirsute

image

This isn't all of the official Debian architectures (it's missing armel, i386, mips64el, and mipsel), but it looks like we're in pretty good shape!

mikestillman commented 3 years ago

Excellent!

DanGrayson commented 3 years ago

Great!

d-torrance commented 3 years ago

Update: a package for the 1.16.99 beta release has been uploaded to Debian experimental! And good news -- it's building on several more architectures than 1.16 did! In particular, armel, armhf, and i386.

We still have some missing architectures, with various test failures:

It also failed to build on ricsv64 (which is an unofficial architecture) due to a bug in flint, and a slew of other unofficial architectures due to various missing dependencies.

I'll patch out these tests prior to packaging 1.17 once it's released, so hopefully we'll be in better shape then. It feels like a game of Whack-a-Mole, though. :)

d-torrance commented 3 years ago

I've just requested sponsorship for a package based on the release-1.17.1 tag for Debian unstable. Fingers crossed that everything builds!

DanGrayson commented 3 years ago

Okay!

If you're successful, does Macaulay2 end up in Ubuntu 20.10, 20.04, 19.10, etc. Or just in 21.04?

Once we're in, say to 21.04, do further releases of Macaulay2 find their way to the users of 21.04 promptly?

d-torrance commented 3 years ago

If you're successful, does Macaulay2 end up in Ubuntu 20.10, 20.04, 19.10, etc. Or just in 21.04?

Just 21.04. But I'm planning on releasing PPA packages (using the same ppa:macaulay2/macaulay2 archive that's used in the Github workflow for various build dependencies) for 18.04, 20.04, and 20.10. (19.10 has already reached its end of life.)

Once we're in, say to 21.04, do further releases of Macaulay2 find their way to the users of 21.04 promptly?

Not through the official Ubuntu archives. Ubuntu 21.04 will be released in April and likely will have 1.17.1 (it already has 1.16, at least for amd64). It will only get security updates after that. 1.18 will likely end up in 21.10 in October, and so on.

d-torrance commented 3 years ago

Macaulay2 1.17.1 is now in Debian unstable (and in Ubuntu hirsute-proposed, which eventually migrates into 21.04 "Hirsute Hippo"). The builds have been going well so far. As I write this, we're still waiting on two official Debian architectures (mips64el and mipsel), and the only build failures are for non-official architectures. Most of them have missing dependencies, but two got pretty far into the build:

The Ubuntu builds have also been going pretty well. The amd64 build, strangely, has been running for 10 hours, but I expect it should be fine, and the only failures have been riscv64 (#1833 again) and s390x (a dependency issue with Emacs).

The continuous integration test for Debian unstable on i386 failed (see #1834), though. It's marked as a regression since it passed for the 1.16.99 package (before check "Core" was a thing). But if I'm reading the docs correctly, I don't think this will prevent migration to testing since 1.16.99 isn't in testing.

So fingers crossed, I think it's likely that it will migrate to both Debian testing and Ubuntu hirsute-release in a few days!

DanGrayson commented 3 years ago

Sounds good!

d-torrance commented 3 years ago

The 1.17.1 package has migrated from "proposed" into "release" for Ubuntu 21.04, so we're definitely in on the Ubuntu side of things! Still two more days until the Debian package might migrate -- I'm still not clear on whether #1834 will block it or not. It's fixed (with an ugly hack) in my current draft, but I'll wait until 1.17.2 is released.

I've also published PPA packages based on this for earlier Ubuntu releases (18.04, 20.04, and 20.10). Would it be worth announcing this on the Google Group and/or adding it as an alternate download location in the Ubuntu section of the webpage?

mikestillman commented 3 years ago

@d-torrance That is great! Thanks!!

Yes, we should announce it on the google group, and fix the webpage for that too. @DanGrayson what do you think?

DanGrayson commented 3 years ago

That's great!

d-torrance commented 3 years ago

1834 indeed blocked the package from migrating to testing. Is there still a plan to release 1.17.2 soon? Or I could have 1.17.1 re-uploaded.

DanGrayson commented 3 years ago

1834 indeed blocked the package from migrating to testing. Is there still a plan to release 1.17.2 soon? Or I could have 1.17.1 re-uploaded.

I'm not sure what you mean by re-uploading 1.17.1, but if you want to make a debian release based on something more modern than 1.17.1, I'm happy to bump the version number to 1.17.2 for you, and to make a tag based on that. Does #1834 have to be fixed before that, though?

d-torrance commented 3 years ago

I'm not sure what you mean by re-uploading 1.17.1

I'd keep the same upstream source, but add a -2 revision to the Debian version number. The Debian packaging itself would have a few changes.

but if you want to make a debian release based on something more modern than 1.17.1, I'm happy to bump the version number to 1.17.2 for you, and to make a tag based on that.

I don't really have a preference, but whatever version I use is likely to be a part of Debian 11 and so will be around for a while (and Ubuntu 21.04, but that's not an LTS release so it's less of an issue). So if there's a 1.17.2, it would be nice if it's pretty stable. I think most of the changes since 1.17.1 have been bug fixes and not new features, so that would probably be fine.

Does #1834 have to be fixed before that, though?

I've fixed it in the Debian package by avoiding capture in the Core tests on 32-bit architectures in this patch. I did propose the change (for all architectures) upstream in #1835, but withdrew it based on Mahrud's comments.

DanGrayson commented 3 years ago

I'm under the impression that the current master branch is pretty secure, and could serve as 1.17.2, but I also have no real preference. Perhaps Mike does. Mike? @mikestillman

DanGrayson commented 3 years ago

PS: I don't see anything wrong with your patch (for 32 bit machines) being incorporated in our code -- it won't affect 99% of our users.

d-torrance commented 3 years ago

PS: I don't see anything wrong with your patch (for 32 bit machines) being incorporated in our code -- it won't affect 99% of our users.

Alrighty -- see #1866.

DanGrayson commented 3 years ago

@d-torrance : Well, maybe we should just let 1.17.1 be the Ubuntu distribution. That seems simplest for now.

By the way, our release schedule is biannual, same as Ubuntu's, but our dates are a little after theirs: May 15 and Dec 15. Maybe we should rotate our release schedule to reduce the lag time between our release and the Debian/Ubuntu deadlines. What would be best?

d-torrance commented 3 years ago

@d-torrance : Well, maybe we should just let 1.17.1 be the Ubuntu distribution. That seems simplest for now.

Sounds good. I'll still try and get it in Debian using the patch from #1866.

By the way, our release schedule is biannual, same as Ubuntu's, but our dates are a little after theirs: May 15 and Dec 15. Maybe we should rotate our release schedule to reduce the lag time between our release and the Debian/Ubuntu deadlines. What would be best?

I think the current release schedule works pretty well. Ubuntu freezes migrations from Debian about two months before each release (so February and August for the April and October releases). So that gives plenty of time to package each new version, find a sponsor, fix build failures, etc.

DanGrayson commented 3 years ago

Okay, thanks!

mikestillman commented 3 years ago

I'm under the impression that the current master branch is pretty secure, and could serve as 1.17.2, but I also have no real preference. Perhaps Mike does. Mike? @mikestillman

I think that is fine.

DanGrayson commented 3 years ago

@mikestillman -- I now prefer to just stick with 1.17.1, and what you said above is a little ambiguous. Shall we?

@d-torrance

mahrud commented 3 years ago

For the record, for the brew build it is preferable to use a versioned tarfile, and that's easiest with a tag. For instance, a week ago a test build from the main branch passed, and today it failed. There's no cost to a tag, so why not? You don't need to make binary distribution for all of them.

DanGrayson commented 3 years ago

Yes, certainly, if we bump the version number to 1.7.2, there will be a tag (release-1.7.2) for that.

d-torrance commented 3 years ago

FYI, 1.17.1 has been re-uploaded to Debian unstable. In addition to the #1866 patch, it has a few others from the development branch that hopefully should help it to build on a few more architectures.

DanGrayson commented 3 years ago

Thanks!

d-torrance commented 3 years ago

Good news: The i386 and armhf continuous integration tests are passing! So they're not blocking the migration to testing any more.

Bad news: Now the ppc64el build is failing, which is blocking migration. This has happened twice so far. I've requested a third build, so fingers crossed. I'm not sure what's going on. Both times, it's been killed building relatively small examples.

d-torrance commented 3 years ago

Third time's the charm! The ppc64el package finally built, so things are looking very good for migration to testing in a few days.