search

camlp5 / camlp5

Preprocessor / Pretty Printer for OCaml
Other
74 stars 26 forks source link

Cannot install camlp5.8.00.01 on macOS #78

Open haxiomic opened 3 years ago

haxiomic commented 3 years ago

Today trying opam install camlp5 on macOS 10.14.6 with a fresh ocaml install I get the following

<><> Gathering sources ><><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫 
Processing  3/3: [camlp5.8.00.01: dl]
[camlp5.8.00.01] downloaded from cache at https://opam.ocaml.org/cache

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫 
[ERROR] The compilation of conf-perl-string-shellquote failed at "/Users/geo/.opam/opam-init/hooks/sandbox.sh build perl -MString::ShellQuote -e 1".
[ERROR] The compilation of conf-perl-ipc-system-simple failed at "/Users/geo/.opam/opam-init/hooks/sandbox.sh build perl -MIPC::System::Simple -e 1".

#=== ERROR while compiling conf-perl-string-shellquote.1 ======================#
# context     2.0.7 | macos/x86_64 | ocaml-system.4.08.1 | https://opam.ocaml.org#a677e346
# path        ~/.opam/default/.opam-switch/build/conf-perl-string-shellquote.1
# command     ~/.opam/opam-init/hooks/sandbox.sh build perl -MString::ShellQuote -e 1
# exit-code   2
# env-file    ~/.opam/log/conf-perl-string-shellquote-4224-a00279.env
# output-file ~/.opam/log/conf-perl-string-shellquote-4224-a00279.out
### output ###
# Can't locate String/ShellQuote.pm in @INC (you may need to install the String::ShellQuote module) (@INC contains: /Library/Perl/5.18/darwin-thread-multi-2level /Library/Perl/5.18 /Network/Library/Perl/5.18/darwin-thread-multi-2level /Network/Library/Perl/5.18 /Library/Perl/Updates/5.18.4 /System/Library/Perl/5.18/darwin-thread-multi-2level /System/Library/Perl/5.18 /System/Library/Perl/Extras[...]
# BEGIN failed--compilation aborted.

#=== ERROR while compiling conf-perl-ipc-system-simple.1 ======================#
# context     2.0.7 | macos/x86_64 | ocaml-system.4.08.1 | https://opam.ocaml.org#a677e346
# path        ~/.opam/default/.opam-switch/build/conf-perl-ipc-system-simple.1
# command     ~/.opam/opam-init/hooks/sandbox.sh build perl -MIPC::System::Simple -e 1
# exit-code   2
# env-file    ~/.opam/log/conf-perl-ipc-system-simple-4224-d0f245.env
# output-file ~/.opam/log/conf-perl-ipc-system-simple-4224-d0f245.out
### output ###
# Can't locate IPC/System/Simple.pm in @INC (you may need to install the IPC::System::Simple module) (@INC contains: /Library/Perl/5.18/darwin-thread-multi-2level /Library/Perl/5.18 /Network/Library/Perl/5.18/darwin-thread-multi-2level /Network/Library/Perl/5.18 /Library/Perl/Updates/5.18.4 /System/Library/Perl/5.18/darwin-thread-multi-2level /System/Library/Perl/5.18 /System/Library/Perl/Extra[...]
# BEGIN failed--compilation aborted.

Whereas opam install "camlp5=7.14" works fine

I'm too much of an ocaml and perl newb to delve deeper. I had an attempt at installing those perl modules manually without success

Was tests on fresh install:

brew install ocaml
brew install opam
opam init
opam install camlp5
chetmurthy commented 3 years ago

Hi, you're right that these packages are required to build the new release-stream (8.xx) of Camlp5. On most UNIX-derived distros, they seem to be available, but I don't know how to get them for MacOS. Sadly, that means that it's not possible to build the new Camlp5 on MacOS.

Since I don't have a Mac, I can't figure out how to fix this. If you wanted to collaborate on doing so, I'd be happy to try to work with you. Basically, we need to figure out how to install these two Perl modules. I'm a very, very proficient Perl programmer, and these are very standard Perl packages, so the problem is MacOS package-management. Do you know much about MacOS package-management?

chetmurthy commented 3 years ago

Hi, still interested in trying to get this working on MacOS ?

haxiomic commented 3 years ago

After your message I had another shot at trying to install those modules but still no luck; for ShellQuote I tried

perl -MCPAN -e 'install -force MString::ShellQuote'

which did -something- (big newb I'm afriad) but then the opam install camlp5 failed with the same Can't locate String/ShellQuote.pm in @INC (you may need to install the String::ShellQuote module)

It looks like the two modules in question are only used for the testsuite - is it possible to separate the testsuite dependencies from the main code when running opam install camlp5?

(I'm using the perl (v5.18) install included with macOS)

chetmurthy commented 3 years ago

I started writing a long, long reply, but I think, at this point, it might be better to have a call ? B/c this is a complex issue: it touches on how OSX manages packages, and as it turns out there are at least two different ways: "brew" and "homebrew". I don't know which was used to install your perl distribution, and maybe it was neither of the above. In any case, since I don't have access to a Mac, I can't actually test anything.

This discussion thread is probably useful to read: https://discuss.ocaml.org/t/opam-depexts-on-macos-darwin-how-to-specify-perl-packages/6515/6

If you can have a con-call about this, maybe send me a direct message on the OCaml discussion forum? I'm "Chet_Murthy" (or maybe it's "Chet Murthy" without the underscore) over there.

haxiomic commented 3 years ago

I'm down for a call to work through things – I'll get in touch on discuss.ocaml.org. Does the end of next week work for you? (I'm on a project in crunch-mode this week)

chetmurthy commented 3 years ago

I am 100% indifferent to when: please choose whatever works best for you. Truthfully, you are the gating resource here, b/c you actually have and use a Mac. So, y'know, whenever you want to work on this, is all good with me.

LdBeth commented 3 years ago

(I'm using the perl (v5.18) install included with macOS)

it is usually not a good idea to use any interpreter came with macOS because they're often outdated.

In your particular case, the perl came with macOS needs sudo to have rights to write into protected directories otherwise it won't actually install anything.

However it is still desirable (maybe "mandatory") to use brew because that's how the depexts thing supposed to work.

"brew" and "homebrew"

They refer to the same thing. Maybe you mean "macport", which is the other package manager supported by ocaml-depexts? There are some other alternatives, but no one cares :P

chetmurthy commented 3 years ago

End of any week works -- we're all in a Groundhog Day movie, right?

haxiomic commented 3 years ago

Seems like it! I wasn't able to find a DM feature over at discuss.ocaml.org but happy to continue on email, I'm haxiomic@gmail.com If Friday works, I'm on GMT and can do anytime from 2pm to 9pm this Friday, but can also do any other day

pmetzger commented 3 years ago

Howdy! I maintain ocaml for MacPorts. If a version of camlp5 that supports 4.12.0 comes out, I will try to update it to work on MacPorts. No promises though.

chetmurthy commented 3 years ago

Perry, the problem seems to be not with ocaml -- it seems like once opam is installed, it's easy enough to get versions of ocaml built. The problem is with some perl modules that I use in the latest versions of camlp5 and some of the packages built on top of it.

Neither me nor haxiomc@ are sufficiently well-versed in Mac development [I'm a UNIX jock, but I don't have access to a Mac commandline environment where I can figure out what's going wrong] to be able to get these perl-dependent opam modules to work correctly.

pmetzger commented 3 years ago

Sure, but as it stands, there isn't a 4.12.0 version of camlp5 so I can't start the work, since MacPorts has 4.12.0 as its supported version. Once there's a version that supports 4.12.0, I'll try to figure out how to make it build.

pmetzger commented 3 years ago

BTW, just to be clear: 7.14 does support 4.12, but I had the impression 8.x does not. Am I correct on this?

chetmurthy commented 3 years ago

As of version 8.00, camlp5 should support ocaml 4.12.0 syntax. The alpha versions did not (b/c 4.12's alpha wasn't out yet).

The 7.xx stream is maintained only for compatibility, since the 8.00 stream entails .... big, big changes to support new ocaml syntax completely. The .7.xx stream will (hopefully) always work work with new versions of ocaml, but it won't support new ocaml syntax at all -- which is basically what the 7.xx stream did for a long time, before I took over maintenance.

The 8.xx stream is where I'm adding full support for all ocaml syntax, including PPX, etc.

pmetzger commented 3 years ago

Is it backwards compatible? Should I attempt to upgrade MacPorts to 8.00?

chetmurthy commented 3 years ago

If you are asking "does Camlp5 8.00 support previous versions of Ocaml", then the answer is "yes, any version >= 4.02.3". That is the point at which PPX was introduced, and supporting both PPX and not-PPX versions of the AST was .... well, I found it too onerous, esp. when that's already a pretty old version of ocaml. But everything new should work, and I test all such versions that are available in opam, before each release.

The test is simply to do a build of Campl5 (which is bootstrapped) so it's a little limited. For version >= 4.10.0, I also run the new testsuite, which is quite a bit more extensive and tests a lot of internal interfaces, testing both positive (accepting good input) and negative (rejecting bad input) behaviour.

pmetzger commented 3 years ago

My question was "does Camlp5 8.00 support software that used Camlp5 v7 and before", not "does it support earlier versions of OCaml."...

chetmurthy commented 3 years ago

Aha. OK. So typically it does, and when it doesn't, I'm happy to work with the owner of the software to fix any bugs. It's not possible to be fully-backward-compatible, b/c of the introduction of all this new syntax: some of it required significant reorganization to the grammars, so if somebody had extended those grammars (a common way of using camlp5), then when the grammars got reorganized, their extensions might not work, or produce crazily paradoxical results.

Typically it's not hard to fix, which is why I always offer to fix whatever the discrepancy is. I've done this for a number of projects that use camlp5, and are also hosted on opam. Do you have any projects in mind that I should have a look at?

pmetzger commented 3 years ago

So I maintain OCaml for MacPorts; there are very few packages that currently depend on camlp5, but there are a couple. They're old and probably have no active maintainer. Perhaps the pragmatic solution is to stay on 7 for now; typically MacPorts is not for people developing code but only for people trying to use prepackaged software.

chetmurthy commented 3 years ago

Can you tell me what those packages are, that depend on camlp5, in macports ?

On Sun, Mar 7, 2021 at 3:14 PM Perry E. Metzger notifications@github.com wrote:

So I maintain OCaml for MacPorts; there are very few packages that currently depend on camlp5, but there are a couple. They're old and probably have no active maintainer. Perhaps the pragmatic solution is to stay on 7 for now; typically MacPorts is not for people developing code but only for people trying to use prepackaged software.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/camlp5/camlp5/issues/78#issuecomment-792372118, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANMU66X6WEECJF5SOO27KTTCQCDZANCNFSM4W6Z2KZQ .

pmetzger commented 3 years ago

So far as I can tell, we're down to libCUDF, geneweb, and ledit, none of which appear to be very actively developed.

sagotch commented 3 years ago

So far as I can tell, we're down to libCUDF, geneweb, and ledit, none of which appear to be very actively developed.

I just want to correct this : geneweb is very actively developed https://github.com/geneweb/geneweb (I don;t know about MacPorts packages)

gares commented 3 years ago

FTR I managed to get an OSX github action install camlp5 here https://github.com/LPCIC/elpi/pull/110

sagotch commented 3 years ago

NB: This used to work on appveyor on appveyor as well, but it stopped working few days ago https://ci.appveyor.com/project/geneweb/geneweb/branch/master/job/bb43xv7vwmbcyaw6#L490

(appveyor file here: https://github.com/geneweb/geneweb/blob/master/appveyor.yml#L44)

chetmurthy commented 3 years ago

@gares : OK, so you're a MacOS hacker! Excellent! [I'm not, and have not even remote access to MacOS, so to fix anything there involves ... well, typing random text into commits and rollin the dice of the opam-ci gods]

the place such a patch (to install relevant perl modules) would go, is in opam modules

conf-perl-ipc-system-simple and conf-perl-string-shellquote .

can you have a look at those? Once those have been updated to do the right thing, then camlp5 should install fluidly.

P.S. I'm not trying to put you off. I wrote those two opam packages (at the suggestion of the OPAM maintainers) to properly modularize the dependency on these Perl modules. I just don't know how to make them work on MacOS, is all.

gares commented 3 years ago

I use linux, sorry. Google tells me that macports has packages for these perl modules, so maybe that is the way to go.

chetmurthy commented 3 years ago

Oh, that's to bad. In any case, I went thru some of these combination with the OPAM maintainers, back when we created these two packages. And basically we couldn't figure out how to set things up for MacOS. So at this point, I think it really will require having somebody with live access to MacOS. Ah, well.

gares commented 3 years ago

Are these perl modules really important for the build process? Do you really need them for a released version?

chetmurthy commented 3 years ago

They allow to write a "launcher" script that can properly quote arguments. I use that in tests, and heavily in the follow-on modules built on camlp5: pa_ppx, pappx{hashcons,unique,migrate,q_ast,ag} . And besides, on MacOS the user can just install them: it's not very hard to do manually.

Eventually, some MacOS user will ask about this, and I can ask them to work out how to make these two conf-perl packages work.

In case you were going to ask: yes, I could rewrite in OCaml. But the point is that I'm using the right tool for the right task: Perl is the right tool when doing UNIX scripting and string manipulation. In OCaml the code would be much larger, and there might be errors.

gares commented 3 years ago

I agree, I was just looking for a different solution to the problem.

haxiomic commented 2 years ago

After checking how the haxe team solves this it looks like this can be solved with the following; replacing the built-in perl install in macOS

brew install perl
brew install cpanminus

Restart shell

cpanm IPC::System::Simple
cpanm String::ShellQuote

Now this works for me :)

opam install camlp5

Testing on macOS 10.14

chetmurthy commented 2 years ago

George, yes, this should work. If only someone could figure out how to make opam do these things in its idiomatic way, on MacOS.

Blaisorblade commented 2 years ago

  1. How hard would it to port Camlp5 7.x to build on newer OCaml? I don't care for the new OCaml syntax, just for the things that work on older OCaml to keep working (I'm only upgrading OCaml because it builds a faster Coq).

  2. I'm a Mac user, I have some clue on packaging and I'd be happy to help, especially if 1 fails, since I need this to use the latest OCaml compilers; but opam's sandbox on Mac essentially only lets you write to $OPAM_SWITCH_PREFIX (see https://gist.github.com/Blaisorblade/1f564b2702ea586ffccf75d4ab078198). So I don't think anything Mac-specific is needed, only a way to install those packages inside the opam switch; you'd then need to use opam's setenv (https://opam.ocaml.org/doc/Manual.html#opamfield-setenv) to extend Perl's library search path.

In https://github.com/camlp5/camlp5/issues/78#issuecomment-912911467 you wrote:

In any case, I went thru some of these combination with the OPAM maintainers, back when we created these two packages. And basically we couldn't figure out how to set things up for MacOS. So at this point, I think it really will require having somebody with live access to MacOS.

Got a link on that discussion and what was the problem? In the logs I found https://github.com/ocaml/opam-repository/pull/17886, https://github.com/ocaml/opam-repository/pull/17987 and https://github.com/camlp5/camlp5/issues/66, but I don't see anywhere anything close to "I wrote the perl -M CPAN line in the conf package, but it fails because of X".

Blaisorblade commented 2 years ago

@gares wrote:

Google tells me that macports has packages for these perl modules, so maybe that is the way to go.

Not quite. Opam considers homebrew and macports as distinct distributions, which is a very good way to think about it. For Homebrew users, supporting Macs via Macports is as good as supporting Fedora via Ubuntu packages. But it's true the fix is a one-liner for Macports.

For instance, conf-adwaita-icon-theme has the following lines, 

depexts: [
  ["adwaita-icon-theme"] {os-family = "debian"}
  ...
  ["adwaita-icon-theme"] {os-family = "homebrew"}
  ["adwaita-icon-theme"] {os-family = "macports"}
  ["x11-themes/adwaita-icon-theme"] {os-family = "gentoo"}
  ["adwaita-icon-theme"] {os = "freebsd"}
  ["x11/gnome/adwaita-icon-theme"] {os = "openbsd"}
  ["graphics/adwaita-icon-theme"] {os = "netbsd"}
]

However, homebrew does not seem to package cpan modules, so I don't think we'll get to upstream homebrew packages to the main upstream repos. They ship bioperl, but they seem to only use cpanm --self-contained. BTW, does that install those deps "privately", and would that be an option here?

Blaisorblade commented 2 years ago

we'll get to upstream homebrew packages to the main upstream repos.

We can submit whatever we want to external homebrew formula repos ("Homebrew taps"), but we'd have to convince the opam-repository people to allow depending on such a tap.

chetmurthy commented 2 years ago

Two questions:

  1. As I understand it, Coq no longer depends on Camlp5 (they forked the essential libraries and took them into their corpus). So this is about maintaining backlevel versions of Coq, right? Are there actually users who are using these backlevel versions of Coq, who cannot use the modern version ?

I'm asking because, yes, it is straightforward to upgrade the 7.xx stream to more-modern OCaml. But if that's merely for the purpose of keeping backlevel versions of other software upgraded to more-modern OCaml, I don't quite see the point. If there are actual users of these backlevel versions, then that's a different situation.

  1. As I understand from your last comments, macports has these packages ? Can you make the conf-perl-XXX packages work under macports? Then that would seem to solve the problem, no?
Blaisorblade commented 2 years ago
  1. As I understand it, Coq no longer depends on Camlp5 (they forked the essential libraries and took them into their corpus). So this is about maintaining backlevel versions of Coq, right? Are there actually users who are using these backlevel versions of Coq, who cannot use the modern version ? I'm asking because, yes, it is straightforward to upgrade the 7.xx stream to more-modern OCaml. But if that's merely for the purpose of keeping backlevel versions of other software upgraded to more-modern OCaml, I don't quite see the point. If there are actual users of these backlevel versions, then that's a different situation.

Coq migrated as you say, but I want to use coq-elpi, a Coq library which didn't (it uses Camlp5 for its own purposes). There's a patch migrating coq-elpi to Camlp5 8, but as @gares said it's stuck on this problem.

  1. As I understand from your last comments, macports has these packages ? Can you make the conf-perl-XXX packages work under macports? Then that would seem to solve the problem, no?

This would not solve the whole problem. I use Homebrew, and Homebrew and MacPorts are incompatible, so you have two disjoint issues: support homebrew, and support macports. So this wouldn't help homebrew users, but it'd help for the macports one.

Re macports I've seen your question in https://github.com/camlp5/camlp5/issues/66#issuecomment-699510282. I don't use macports, but it seems (from https://ports.macports.org/port/p5.34-ipc-system-simple/details/ and https://ports.macports.org/port/p5.34-string-shellquote/details/) a macports fix could be the following depexts lines:

for conf-perl-ipc-system-simple

  ["p5.34-ipc-system-simple"] {os-family = "macports"}

for conf-perl-string-shellquote

  ["p5.34-string-shellquote"] {os-family = "macports"}

But it seems macports lets you install multiple perl versions in parallel and has separate packages for each, so you'd probably need to use /usr/bin/env perl5.34 in your scripts on this platform? I don't see a way to depend on "whichever perl version is the default"...

==

Could you just "vendor" these dependencies (= include a copy with your perl code), at least on mac? While ugly, it seems better than leaving this issue open for an year...

chetmurthy commented 2 years ago

Can you clone camlp5, and apply this patch ? If it works, I can release a new version. I'm reluctant to do this, b/c to my mind, a platform that doesn't have proper support for Perl5 isn't a developer's platform, but hey, I guess I'm swimmin' against the tide.

git diff diff --git a/opam b/opam index 6aba450a..e13ceee3 100644 --- a/opam +++ b/opam @@ -29,8 +29,8 @@ bug-reports: "https://github.com/camlp5/camlp5/issues" depends: [ "ocaml" {>= "4.02" & < "4.14.0"} "conf-perl"

chetmurthy commented 2 years ago

two further comments:

  1. the reason I don't "vendor" these Perl libraries, is that that means I have to track changes / bugfixes. And since there's literally nothing special I'm doing, I don't see the point of that.
  2. the patch I suggested above isn't great, either. B/c OPAM-CI will build both with & without tests, so it'll break. the Perl dependency is for a "launcher" script used in the tests, and that same launcher is used all over the build-system for pa_ppx extensions built on Camlp5.

Since you use homebrew, can I suggest that you might request the maintainers to add these two Perl modules to their main repo? They're very, very standard, and are available in many, many distros. That would solve the problem, too.

chetmurthy commented 2 years ago

If you find that that works, I'll conditionalize on "with-test and not macos" for those packages.

k4rtik commented 2 years ago

We can submit whatever we want to external homebrew formula repos ("Homebrew taps"), but we'd have to convince the opam-repository people to allow depending on such a tap.

This may be the only reasonable path forward at this point if the libraries cannot be vendored for testing. As @Blaisorblade suspects, homebrew explicitly does not package perl/python/ruby/etc. libraries.

Blaisorblade commented 2 years ago

The installation issue is fixed but https://github.com/camlp5/camlp5/pull/88#issuecomment-1017035603 mentions that:

I still need to find a way to make these perl packages available on homebrew, b/c the pa_ppx packages need them to build. As do other packages that use the "mkcamlp5" scripts. But for now, this is ..... acceptable.

Let me paste Homebrew's advice, and also remind you it won't work as-is inside opam's sandbox:

==> Caveats
By default non-brewed cpan modules are installed to the Cellar. If you wish
for your modules to persist across updates we recommend using `local::lib`.

You can set that up like this:
  PERL_MM_OPT="INSTALL_BASE=$HOME/perl5" cpan local::lib
  echo 'eval "$(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib=$HOME/perl5)"' >> ~/.profile

IIUC (also from https://metacpan.org/pod/local::lib), local::lib is a way to install extra modules in non-default location; conf-perl packages could install the perl modules inside the switch folder (that's allowed by the opam sandbox) and use setenv to extend the environment of each switch appropriately...