Need some help to package Perl6 modules for Debian..

dod38fr commented 7 years ago

Hi

I need some help to figure out how to best use zef to create Debian package from Perl6 modules.

As Debian packager, my goals are somewhat different from perl6 installer goals:

Pre-compilation must be done at installation time: we don't want to let modules installed by dpkg to be pre-compiled when user runs a perl6 program, because there's no way to clean-up pre-compiled files in user home afterwards.
Pre-compiled files must be cleaned up when a package is removed

I think this could be achieved by installing Perl6 modules a bit like emacs extensions are installed:

Debian package ships source files (i.e. more or less the files provided by a git clone)
zef is invoked to install these files in /usr/share/perl6 by post install script (package install or upgrade)
zef is invoked at removal or upgrade time to remove old pre-compiled files

Can zef support removal of all pre-compiled files ?

If I understand correctly, the pre-compiled files are valid for a single version of rakudo, hence pre-compilation must be redone when a new version of rakudo is installed.

Can you confirm that point ?

If yes, a rakudo upgrade implies to:

clean-up old precompiled files of all installed modules (including zef, which may be a bit suicidal)
trigger a recompilation of all installed modules

Dpkg has mechanisms to trigger an action on all Perl6 packages when rakudo is installed, hence the actions above may be done one by one using a cleanup and recompile step similar to a package upgrade.

I will welcome thoughts or advices before I start to implement Debian packaging for Perl6 modules ;-)

Note that packagers for other distro may have similar needs. Feel free to include them in this discussion to find common ground between Linux distribution with respect to Perl6 packaging.

All the best

ugexe commented 7 years ago

Removal of precomp files would be the responsibility of rakudo (i.e. the files should be removed internally just like all the other source/resource/script files), because its the only thing that actually knows where they are or which one is which. This is not something that is implemented yet in CompUnit::Repository::Installation.

When precomp files are created they take the current rakudo version (and other factors) to generate the final file name (some sha1). This means it is not technically necessary to delete old precomp files when you upgrade rakudo because: 1) You will just have leftover files that are no longer used 2) They will be re-precompiled on the fly when they are first used

dod38fr commented 7 years ago

I understand that running a Perl6 program can generate pre-compiled files on the fly and that's not a problem for people who are trying Perl6.

However I believe that on-the-fly precompilation may eventually have undesirables side effects:

people may not be aware that they are running perl6 programs (say the perl6 version of gcstar, shutter, frozen-bubbles...) and may wonder why the first start-up is so long and what are all those files appearing in ~/.perl6.
this may have security issues: for instance a perl6 web server will need to write pre-compiled files as www-data user. These files are writable by web server so a remote attack have the possibility to write file that are executable.( From a security point of view, we should be able to start such a Perl6 daemon with an option that forbids writing pre-compilation files.)

I guess that these 2 issues can be avoided by pre-compiling Perl6 modules with zef as installation time. I.e. run zef install . in a post-install script which will create pre-compilation files in /usr/share.

I'd rather avoid leaving pre-compilation files around. I grant you that they are harmless, but they may add up to significant space over the years, especially on ARM system. I'm pretty sure that I'll get bug report if a cleanup is not done from time to time. I guess that I could try to remove all pre-compiled files (with zef nuke ?) when upgrading rakudo and trigger a pre-compilation using the aforementioned post-install scripts.

Does that sound reasonable ?

All the best

ugexe commented 7 years ago

There is no easy solution to this problem. See: python

Precompilation will only occur by direct action from rakudo itself, not a package manager like zef. A package manager has absolutely no way of determining what other modules a specific module in a distribution depends on so it has no way of building a DAG to determine the correct order to precompile the files in/against. This is why it must happen on-the-fly, because the only way to resolve this graph is by actually compiling and running the program. When you install a module it does get precompiled, but this is essentially just doing a try { use-ok($_) } for <My::Module::Foo My::Module::Bar>, i.e. using each module in the distribution so on-the-fly precomp occurs. If you wanted to somehow control re-precompiling things this is probably the easiest way to do it, although obviously not the most efficient.

Zef has no way of cleaning up precompiled files. Ideally rakudo would provide a mechanism such that external tools could determine the precompilation file location of a given name - this would allow zef to add such a feature. But zef cannot just guess or determine the filename, because these can change based on a repo format that is used by rakudo (hence needing a mechanism in rakudo to provide this info). If you really wanted to do this manually you could probably scan the short-id folder under an installation directory, look inside all the files to determine what the sha1 in the directory name maps to name wise, then use that to somehow filter the precomp directory (I'm not sure if this is possible).

A more ideal solution would be to create a custom CompUnit::Repository that adds the features you wants. Unfortunately I'm not sure how to handle the precompilation portion of this. What I can provide is an example that does not precompile. This example also does not have an install/uninstall method - but this may be of interest to you because you probably don't want zef uninstalling modules installed by apt-get (but you could add these methods and add bits to install/uninstall that update the appropriate dpkg/apt files to reflect such a change). The repository represents a single distribution anyway (so only a single file, blah.tar.gz), so installation and uninstallation is really just moving/deleting a single file. The example is based on a .tar.gz file, but all its doing is extracting files on the fly to stdout (not to a file) that get loaded so if you can output files from a dpkg to stdout you could do something similar.

dod38fr commented 7 years ago

Hi

Sorry for the long delay, I got side tracked by new year and incoming Debian release.

I may have a workable plan to ensure installation, upgrade and removal of pre-compiled files. This should possible thanks to the information stored in Debian package manager.

BTW, many thanks for providing the --/depends option which makes this whole plan workable (at least in my mind...)

Here we go:

package build

Package will deliver:

perl6 source files (like the one stored on github)
post install script (see below)

Build will be done only to enable tests.

No build artifact will be shipped in the package.

package installation

Files will be installed by package manager in /usr/share/perl6/ as usual

Then a post install script is run:

Create a temp dir : build_dir= mktemp -d
build the files and install in temp dir without downloading dependencies: zef --install-to=inst#$build_dir --/depends --/build-depends --/test-depends install . (missing dep will lead to a build failure.)
create pre-compiled file list with cd $build_dir; find . -type f
store file list in /var/lib/perl6/modules/pkg-name.list
move files to destination in /usr/share/perl6 with rsync -a $build_dir /usr/share/perl6/

Note file list will look like:

./precomp/7272134977CED2FE3AFCD2BEF7484A7FA00299BE.1482237294.34813/B7/B70C87D6FC03E17405F0E4DFAE8F6795E30A0A39
./precomp/7272134977CED2FE3AFCD2BEF7484A7FA00299BE.1482237294.34813/B7/B70C87D6FC03E17405F0E4DFAE8F6795E30A0A39.repo-id
./precomp/.lock
./short/DF961EA9DD94E4FD7AAAB62E5362A67CC9E2D994/F99C4F79D888F73DD4A9B05C78C9416A2DEBD380
./repo.lock
./dist/F99C4F79D888F73DD4A9B05C78C9416A2DEBD380
./version
./sources/B70C87D6FC03E17405F0E4DFAE8F6795E30A0A39

:question: should the lock files be shipped or discarded ?

on module deletion

package scripts will:

remove files according to the file list saved in /var/lib/perl6/modules/pkg-name.list
remove empty directories in /usr/share/perl6

on module upgrade:

package scripts will:

remove files and dir as if the module was deleted (see above)
re-run postinstall to create pre-compiled files

on rakudo upgrade

rakudo package scripts will:

query apt to list all perl6-* packages
create build list ordered according to the dependencies listed in apt
redo the module upgrade script of all modules listed in the previous point

:question: should this be triggered when moar or nqp are upgraded as well ? :question: if yes, how to avoid running this 3 times in a row when moar, nqp and rakudo are upgraded in the same apt upgrade command ?

All in all, we should end up with a pretty clean /usr/share/perl6 directory even after several upgrade cycles of rakudo or perl6 modules.

What do you think ? Does that sound reasonable ? Did I miss something ?

All the best

ugexe commented 7 years ago

You sound like you have a good idea of what you need to do. I'm not sure how the precomp stuff will work in practice - module upgrading and re installation may be expected to re-precompile against the new distributions (reverse dependencies). If it doesn't work you could probably delete/reinstall all of the dependencies/reverse-dependencies like listed under module upgrade

I'm not sure about including the .lock files. I would think not.
I'm not sure if you need to reinstall/reprecompile when upgrading moar or nqp. I believe if either of the following change it will want to create updated precomp files: say nqp::hllize(nqp::atkey(nqp::gethllsym('perl6', '$COMPILER_CONFIG'), 'version')); say $*PERL.compiler.id
I'm not familiar enough with the debian packaging to say. I would imagine there is some sort of hooking, or some way to use ENV settings + command introspection (e.g. see what else is being installed)

Give it a shot! 👍

dod38fr commented 7 years ago

Will do. Don't hold your breath :-)

BTW, to satisfy Debian policy (and formally allow us to re-distribute your software) could you mention the copyright owners and the license under which this software is released ? (I assume Artistic-2, but assumptions don't work in legal world)

All the best

ugexe commented 7 years ago

LICENSE has been added

dod38fr commented 7 years ago

Thanks for the LICENSE file.

What is the version file ? (zef's version file contains '2').

Should this be kept or removed ?

All the best

ugexe commented 7 years ago

Version file is generated by Rakudo when installing modules. I'm not sure how you want to handle that part, but the version file contains the "repo-version" CURI can use to read the module from its installed location (it is not zef specific, it is specific to each CompUnit::Repository). This is so when you do a regular rakudo upgrade to a newer version that uses a different install format it can automatically upgrade the current repo to the new format.

ugexe commented 7 years ago

How it works is say you install zef into a really old rakudo. There will be no version file (this is essentially version 0). The locations of the files ends up changing between versions 0 and 2, so what happens is when rakudo upgrades itself it checks the version file to see what version the repo current is, and then runs an upgrade path for 0->1 and 1->2 (in this case it just moves files into slightly different locations) for all modules currently installed (so they still work). Any new modules installed will be installed using the version 2 format directly.

With this in mind you should be able to determine how you want to use the version file. If you are just doing a cp of a zef install to a new locations then you probably want to 1) check that the version file in both locations matches 2) delete the version file in zef. If you are installing zef directly to the repo it will remain installed to and ISNT an original rakudo repo (site or home) you probably want to distribute the version file.

dod38fr commented 7 years ago

Thanks for the explanation. I'm still not sure what I should do with version.

Before I tackle this, I need to better understand how precompiled files should be installed (I'm not there yet).

For instance, the Test module is compiled during rakudo build and installed in /usr/share/perl6/precomp. So I would expect that a command like perl6 -MTest -e 'say "hello\n";' would use these pre-compiled files, so no pre-compiled files would be generated in ~/.perl6.

This is not the case:

$ rm -rf ~/.perl6/precomp/
$ perl6 -MTest -e 'say "hello\n";'
hello

$ tree ~/.perl6/precomp/
/home/domi/.perl6/precomp/
└── ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235
    └── C7
        ├── C712FE6969F786C9380D643DF17E85D06868219E
        └── C712FE6969F786C9380D643DF17E85D06868219E.repo-id

2 directories, 2 files
$ find /usr/share/perl6/ -name C712FE6969F786C9380D643DF17E85D06868219E
/usr/share/perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/C7/C712FE6969F786C9380D643DF17E85D06868219E
/usr/share/perl6/sources/C712FE6969F786C9380D643DF17E85D06868219E

Why is the file C712FE6969F786C9380D643DF17E85D06868219E re-generated in my home directory even though it's already available in ~/.perl6 ?

All the best

dod38fr commented 7 years ago

For what it's worth, the extract from strace belows shows that a precompiled file is found in /usr/share/perl6/precomp, but a similar file is nonetheless written to ~/perl6/precomp:

[pid  9337] stat("/usr/share/perl6/sources/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31", {st_mode=S_IFREG|0644, st_size=30818, ...}) = 0
[pid  9337] stat("/usr/share/perl6/sources/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31", {st_mode=S_IFREG|0644, st_size=30818, ...}) = 0
[pid  9337] stat("/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31", 0x7ffc71618430) = -1 ENOENT (No such file or directory)
[pid  9337] stat("/usr/share/perl6/site/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31", 0x7ffc71618430) = -1 ENOENT (No such file or directory)
[pid  9337] stat("/usr/share/perl6/vendor/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31", 0x7ffc71618430) = -1 ENOENT (No such file or directory)
[pid  9337] stat("/usr/share/perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31", {st_mode=S_IFREG|0644, st_size=285933, ...}) = 0
[pid  9337] stat("/usr/share/perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31", {st_mode=S_IFREG|0644, st_size=285933, ...}) = 0
[pid  9338] stat("/usr/share/perl6/sources/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31", {st_mode=S_IFREG|0644, st_size=30818, ...}) = 0
[pid  9338] open("/usr/share/perl6/sources/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31", O_RDONLY|O_CLOEXEC) = 11
[pid  9338] open("/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31.bc", O_RDWR|O_CREAT|O_TRUNC, 0666) = 35
[pid  9337] stat("/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31.bc", {st_mode=S_IFREG|0644, st_size=280248, ...}) = 0
[pid  9337] stat("/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31.bc", {st_mode=S_IFREG|0644, st_size=280248, ...}) = 0
[pid  9337] open("/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31.bc", O_RDONLY|O_CLOEXEC) = 12
[pid  9337] stat("/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31.bc", {st_mode=S_IFREG|0644, st_size=280248, ...}) = 0
[pid  9337] stat("/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31.tmp", 0x7ffc71618430) = -1 ENOENT (No such file or directory)
[pid  9337] open("/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31.tmp", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 12
[pid  9337] rename("/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31.tmp", "/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31") = 0
[pid  9337] unlink("/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31.bc") = 0
[pid  9337] stat("/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31.repo-id", 0x7ffc71618430) = -1 ENOENT (No such file or directory)
[pid  9337] open("/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31.repo-id", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 12
[pid  9337] stat("/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31", {st_mode=S_IFREG|0644, st_size=285330, ...}) = 0
[pid  9337] stat("/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31", {st_mode=S_IFREG|0644, st_size=285330, ...}) = 0
[pid  9337] stat("/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31.repo-id", {st_mode=S_IFREG|0644, st_size=40, ...}) = 0
[pid  9337] stat("/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31.repo-id", {st_mode=S_IFREG|0644, st_size=40, ...}) = 0
[pid  9337] open("/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31.repo-id", O_RDONLY|O_CLOEXEC) = 12
[pid  9337] stat("/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31", {st_mode=S_IFREG|0644, st_size=285330, ...}) = 0
[pid  9337] open("/home/domi/.perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489325966.69235/A9/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31", O_RDONLY|O_CLOEXEC) = 12

The trace above was produced with:

$ rm -rf ~/.perl6
$ strace -f zef --help 2>&1 | grep grep A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31

Is this behavior expected ?

All the best

dod38fr commented 7 years ago

BTW, I think zef files should be installed in /usr/share/perl6/vendor and not directly in /usr/share/perl6...

Do you agree ?

ugexe commented 7 years ago

Yeah you would probably want to put zef in vendor.

Regarding precomp: see this discussion https://irclog.perlgeek.de/perl6-toolchain/2017-03-18#i_14287090

niner commented 7 years ago

dod38fr: I have spent loads of time thinking about how rakudo's precompilation can work with distribution packaging and have implemented it in a way so all your issues should actually be fixed already.

Rakudo precompiles modules at installation time. The resulting precomp files will be used by rakudo even when the user runs a program for the first time. The compunit repository where the modules are installed into (e.g. /usr/share/perl6/vendor) is designed so installation of a module will always only add new files. I.e. you as a packager can simply take those files, package them and on installation of the package just drop them in the right place. There are no pre- or post-scripts needed at all. Nor do you have to maintain any list of files other than the package meta data you already have in your package database.

When I'm done with my work on the toolchain, you won't need zef for packaging modules. Zef is a tool for the end user. It has much more functionality than you'd need for a simple job as packaging a module. This functionality might even be hurtful. When creating a distro package you will want installation to fail loudly if a dependency is missing, while zef will happily grab the dependency for you.

Rakudo already contains a minimal module installer meant explicitly for packagers (tools/install-dist.pl). There's still an issue with the generated precomp files which I'm working on and the open question of how to include custom build steps (like compiling the C-library for Inline::Perl5). Once those are solved, packaging modules should be downright trivial.

Please talk to me before wasting any more time on solving issues that already have a proper solution. My intention has always been to solve those issues for you. The way I envision packaging is:

# prepare
mkdir -p /tmp/buildroot/usr/share/perl6/vendor
tar -C /tmp -xf Inline::Perl5-0.123.tar /tmp
# install
perl6 install-dist.pl --from=/tmp/Inline-Perl5 --to=/tmp/buildroot/usr/share/perl6/vendor --for=vendor
# package
tar -C /tmp/buildroot -cf perl6-Inline-Perl5.tar usr/share/perl6/vendor

Regarding your issue with rakudo not using the precomp files created at installation time, please try again with RAKUDO_MODULE_DEBUG=1 in your environment. The debug output should tell us why it cannot use those files and instead compiles again.

dod38fr commented 7 years ago

Thanks for the RAKUDO_MODULE_DEBUG=1 hint. it shows:

  8 26747 RMD: Outdated precompiled /usr/share/perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1489847017.95032/C7/C712FE6969F786C9380D643DF17E85D06868219E
                mtime: Instant:1489846936
                since: Instant:1489846936

The times are identical whereas PrecompilationRepository test to compare mtime and since looks for different times (which is reasonable):

 if (not $since or $modified > $since)
                and self!load-dependencies($unit, $modified, @precomp-stores) { ...

I guess that the times are messed up when I copy files during rakudo packaging. I'll check this point.

I need to think more about the other points. I'll get back to you,

Thanks

dod38fr commented 7 years ago

Hello

I agree that using install-dist.pl is much simpler than using zef.

Shipping pre-compiled file is indeed simpler than compiling in post-install script. That would be my preferred choice.

But I wonder what happens when rakudo is updated ?

I guess that for a regular user, existing modules are precompiled when needed and the result is saved in ~/.perl6.

I'm more worried for the case where a Perl6 web server is running. In this case, pre-compiled modules would also be precompiled when needed and the result would be saved for the user that run the server (www-data on Debian). Even if the exploit might not be easy, this raise a security issue where the web server is able to write files that will be executed later.

Another problem arise for server running with a low privilege user (like user nobody): rakudo may fail because the low privilege user has no possibility to save the precompiled files.

All the best

niner commented 7 years ago

Shipping pre-compiled file is indeed simpler than compiling in post-install script. That would be my preferred choice.

But I wonder what happens when rakudo is updated ?

An upgrade of the rakudo package would have to trigger a re-build of the Perl 6 module packages in the distributor's build infrastructure. Thus an upgrade of rakudo would mean getting updated module packages for the user. These module packages contain the appropriate precomp files. This is also the way the user gets rid of the old, no longer useful precomp files.

AFAIK on e.g. the Open Build Service, simply stating the build time dependency in the meta data is enough to get an automatic re-build.

dod38fr commented 7 years ago

:open_mouth: I did not think about that.. May be I can (ab)use Debian transition mechanism to perform these regular rebuilds.

It looks like ocaml and haskell have that kind of setting. I'll ask for advices with my fellow Debian developers.

Thanks for the hint :-)

dod38fr commented 7 years ago

I think I've found why rakudo does not use the insalled pre-compiled files. As explained before, rakudo recompiles the files because the time stamp of the source is the same as the time stamp of the precompiled file.

This is due to a combination of smaller issues (or features):

to ensure reproducible builds, Debian packaging update the time stamps of the files to ensure that no file is more recent than the time of the last Debian changelog entry.
when building, rakudo copy the source file in its hexa file name (e.g. /usr/share/perl6/sources/51E302443A2C8FF185ABC10CA1E5520EFEE885A1 without preserving the time stamp of the original file)
since a Debian package is built after the debian changelog is updated, the files in /usr/share/perl6/sources/ and the precompilded files ends up with the exact same time-stamp

See for instance the time stamp of these files installed with a Debian package:

$ ls --full-time $(find /usr/share/perl6 -name 51E302443A2C8FF185ABC10CA1E5520EFEE885A1)
-rw-r--r-- 1 root root 90578 2017-03-18 15:21:39.000000000 +0100 /usr/share/perl6/precomp/ED429562BF693CB189D0808E8C54097E4896CD6C.1490464508.05962/51/51E302443A2C8FF185ABC10CA1E5520EFEE885A1
-rw-r--r-- 1 root root  7907 2017-03-18 15:21:39.000000000 +0100 /usr/share/perl6/sources/51E302443A2C8FF185ABC10CA1E5520EFEE885A1

This command provides a different result on a rakudo installed with rakudo-star (tested on rakudo-star 2017.01 docker image):

root@e60f2ddb7faf:/# ls -l --full-time $(find /usr/share/perl6 -name 51E302443A2C8FF185ABC10CA1E5520EFEE885A1)
-rw-r--r-- 1 root root 91610 2017-02-28 22:45:24.000000000 +0000 /usr/share/perl6/precomp/98EAEE29307DC37D4338EA65C19B9FA6D589C4AA.1488321898.28363/51/51E302443A2C8FF185ABC10CA1E5520EFEE885A1
-rw-r--r-- 1 root root  7907 2017-02-28 22:45:22.000000000 +0000 /usr/share/perl6/sources/51E302443A2C8FF185ABC10CA1E5520EFEE885A1

Nevertheless, a 2 second difference might be considered as a close shave...

Is it possible to change rakudo so that the time-stamp of the original source file is preserved when creating the source file in /usr/share/perl6/sources/ ?

All the best

ugexe commented 7 years ago

Yes it's possible. If you look inside an installed META6.json file in site/dist/$filename and look at the provides you'll notice it already records each provides file's last modified date (but not other files... yet?).

    "Zef::Utils::SystemInfo" : {
      "lib/Zef/Utils/SystemInfo.pm6" : {
        "cver" : "2017.03-24-g43e0902",
        "file" : "1D92EFF482C2A372BBC3C2C606FEAF258F9C155E",
        "time" : 1472511254.5901
      }
    },

(note it has taken the original meta6.json and turned the previous leaf node, lib/Zef/Utils/SystemInfo.pm6 and turned it into a node using the original value as the key of the node itself)

The problem is that you can't force CURI to accept just files. CURI takes a distribution interface, which could provide IO access from a file, socket, handle, whatever. And not all of these things have a .modified method that can be called by CURI when installing a distribution.

niner commented 7 years ago

Debian packaging update the time stamps of the files to ensure that no file is more recent than the time of the last Debian changelog entry.

I've always thought lying to software is an odd way to reach one's goal.

Is it possible to change rakudo so that the time-stamp of the original source file is preserved when creating the source file in /usr/share/perl6/sources/ ?

Well, at least not easily. MoarVM doesn't expose uv_fs_utime, so we'd have to add support to MoarVM, nqp and rakudo first. Though it's good to know, that we at least have the information available.

However, I wonder if we shouldn't instead make an effort to get rid of timestamp checking entirely. On file systems like FAT, we are faced with a granularity of 2 whole seconds (!), i.e. half an eternity in computer terms. Issues with time stamps were the reason why I already replaced much of the time stamp checking with sha hashes (e.g. precomp files themselves are hashed). If we record the source file's hash and compare that, we can be absolutely sure instead of having to be cautious. We already only check the dependencies' .modified if the identity (hash) of the whole repo chain changed.

For files in an Installation repository, we can even record the hash at installation time and add it to the short-name lookup file. Might even save us a couple of milliseconds when loading a module compared to the timestamp check.

dod38fr commented 7 years ago

However, I wonder if we shouldn't instead make an effort to get rid of timestamp checking entirely. On file systems like FAT, we are faced with a granularity of 2 whole seconds (!), i.e. half an eternity in computer terms.

From what I understand tar file (used inside of Debian package) have a granularity of one second. RPM packages are based on cpio. This program is quite old and I would not be surprised if cpio archives have the same time granularity.

Issues with time stamps were the reason why I already replaced much of the time stamp checking with sha hashes (e.g. precomp files themselves are hashed). If we record the source file's hash and compare that, we can be absolutely sure instead of having to be cautious.

Agreed. Basing build decision of file signature was the cornerstone of old cons build system. This was much more reliable that doing rebuild based on timestamp.

Anything that can reduce the number of Perl6 modules rebuilds will be most welcome by Debian project.

niner commented 7 years ago

First part just landed: https://github.com/rakudo/rakudo/commit/ca0a74398b9385053c79de62a42418258905c715 That takes care of a precomp file's dependencies.

The only modification time stamp check left is for the top-level module itself in CompUnit::PrecompilationRepository::load. That should be rather easy.

niner commented 7 years ago

Btw. this is Inline::Perl5's spec file for creating an RPM package: https://build.opensuse.org/package/view_file/home:niner9:branches:devel:languages:parrot/perl6-Inline-Perl5/perl6-Inline-Perl5.spec?expand=1 I guess most package files will look even simpler than that (cause they don't need to compile a helper library)

niner commented 7 years ago

And the second part is done: https://github.com/rakudo/rakudo/commit/ff4a034dd2cf37f6e56e34396851a0f3062a210c

We now no longer check any modification time stamps when loading precompilation files. We now fully rely on checksums. This should help with Debian's reproducible builds :)

dod38fr commented 7 years ago

Thanks for the updates. I'll check them out once rakudo 2017.05 is out.

All the best

gerd commented 7 years ago

I try to get zef and perl6 modules in Fedora.

At the URL

https://bugzilla.redhat.com/show_bug.cgi?id=1452985

is the review request for zef.

If zef is installed with 'tools/install-dist.pl', it seems not to work.

It would be nice if the script 'tools/install-dist.pl' would get some documentation. That it is for installing modules and the example how to use it like it is here in a above comment would be nice.

Is it possible to install zef by itself in two steps. First to build it and then to move|copy the files to its destination?

ugexe commented 7 years ago

@gerd

For me locally tools/install-dist.pl works:

nickl@li685-90:~/repos/rakudo$ install/bin/perl6 tools/install-dist.pl --from="~/repos/zef"

nickl@li685-90:~/repos/rakudo$ zef --help
# <help output>

The two step process you ask of works as well:

nickl@li685-90:~/repos/rakudo$ install/bin/perl6 tools/install-dist.pl --from="~/repos/zef" --to="tmp-repo" --for="site"

nickl@li685-90:~/repos/rakudo$ cp -R tmp-repo/* ~/repos/rakudo/install/share/perl6/site

nickl@li685-90:~/repos/rakudo$ zef --help
# <help output>

gerd commented 7 years ago

In the first version I did this:

%{__mkdir} temp-inst %{__perl6} -Ilib bin/zef --install-to=temp-inst install . %{__cp} -R temp-inst/* %{BR_PERL6_SITE}

But after removing the BUILD directory the package do not work any more.

In the second version I used the script 'tools/install-dist.pl': %perl6_mod_inst --to=%{BR_PERL6_VENDOR} --for=vendor

Using the option '--for=vendor' seems to be needed when installing zef to the buildroot destination specified with the option '--to'.

I hope someone will take the review and this will pass it. I will report this here.

niner commented 7 years ago

FWIW this is the example for how it's supposed to be used: https://build.opensuse.org/package/view_file/devel:languages:perl6/perl6-Inline-Perl5/perl6-Inline-Perl5.spec?expand=1 Note that the RAKUDO_RERESOLVE_DEPENDENCIES=0 is necessary!

AlexDaniel commented 6 years ago

Any update on this?

dod38fr commented 6 years ago

We've been sidetracked by issues we've faced to build rakudo on all architectures supported by Debian. This is now working.

We are resuming work on packaging modules for Perl6. @robertlemmen is testing some code to manage pre-compiled files in postinstall scripts.

All the best

AlexDaniel commented 6 years ago

@dod38fr thank you for keeping us updated!

AlexDaniel commented 6 years ago

@dod38fr @robertlemmen any news? Also, have you seen this packaging doc on fedora wiki (maybe there's something useful in it)?

robertlemmen commented 6 years ago

from my point of view: moar, nqp and rakudo seem to be working fine now, zef and tap-harness would be the first to "module" packages that we need. We already have packages for review, but not in the archive yet: https://salsa.debian.org/perl6-team/modules/perl6-tap-harness https://salsa.debian.org/perl6-team/modules/perl6-zef Zef needs some adjustments to make it possible to symlink installed executables in /usr/bin. @ugexe has provided input on how to achieve this, but so far I have not found the time to sort this out. After that we would start packaging "other" modules, most likely inspired by the package list used in rakudo star.

The fedora wiki is certainly interesting, I also looked at what arch does for more inspiration.

dod38fr commented 4 years ago

Hi We now have in Debian a fairly stable set of moar, nqp, rakudo and perl6-zef packages.

The only issue left (which is minor), is the lack of a good old man page for zef. Debian policy requires a man page for each executable installed on a Debian system. I would love to be able to generate a man page from zef documentation. So far the only doc I've seen is the output of zef --help which is ASCII format.

Do you have any plan to provide zef usage doc in a structured format that could be transformed in a man page ?

All the best

levitte commented 3 years ago

I've spent a few hours trying to figure out the perl6-zef package on sid. Basically, it doesn't install:

Setting up perl6-zef (0.8.5-2) ...
===SORRY!=== Error while compiling /usr/share/perl6/tools/install-dist.p6
Could not find CompUnit::Repository::Staging in:
    inst#/root/.raku
    inst#/usr/share/perl6/site
    inst#/usr/share/perl6/vendor
    inst#/usr/share/perl6/core
    ap#
    nqp#
    perl5#
at /usr/share/perl6/tools/install-dist.p6:35
"perl6 /usr/share/perl6/tools/install-dist.p6 --from=/usr/share/perl6/debian-sources/perl6-zef --to=/tmp/6vcCJVNlr9/build --for=vendor" unexpectedly returned exit value 1 at /usr/share/perl5/IPC/System/Simple.pm line 578.
    IPC::System::Simple::_check_exit("perl6 /usr/share/perl6/tools/install-dist.p6 --from=/usr/shar"..., 1, ARRAY(0x55f1515a06f0)) called at /usr/share/perl5/IPC/System/Simple.pm line 550
    IPC::System::Simple::_process_child_error(256, "perl6 /usr/share/perl6/tools/install-dist.p6 --from=/usr/shar"..., ARRAY(0x55f1515a06f0)) called at /usr/share/perl5/IPC/System/Simple.pm line 190
    IPC::System::Simple::run("perl6 /usr/share/perl6/tools/install-dist.p6 --from=/usr/shar"...) called at (eval 29) line 12
    eval {...} called at (eval 29) line 11
    Fatal::__ANON__("perl6 /usr/share/perl6/tools/install-dist.p6 --from=/usr/shar"...) called at /usr/share/perl6/rakudo-helper.pl line 47
    main::install("perl6-zef") called at /usr/share/perl6/rakudo-helper.pl line 151
 at /usr/share/perl6/rakudo-helper.pl line 47
dpkg: error processing package perl6-zef (--configure):
 installed perl6-zef package post-installation script subprocess returned error exit status 1

It turns out that while the rakudo package does indeed install install-dist.p6, it doesn't install CompUnit::Repository::Staging (the upstream source installs neither, as far as I can see, the installation of install-dist.p6 is a simple file copy, specified by debian/install in the rakudo package source)

niner commented 3 years ago

It turns out that while the rakudo package does indeed install install-dist.p6, it doesn't install CompUnit::Repository::Staging (the upstream source installs neither, as far as I can see, the installation of install-dist.p6 is a simple file copy, specified by debian/install in the rakudo package source)

CompUnit::Repository::Staging gets installed as part of the CORE dist into
inst#/usr/share/perl6/core during make install

levitte commented 3 years ago

Ah, interesting! It seems that the Debian package rakudo doesn't, though, as shown by the following (I installed a fresh checkout of https://github.com/rakudo/rakudo in ~/.private-installs/opt/rakudo):

: ~/gitwrk/github.com/rakudo/rakudo $ ~/.private-install/opt/rakudo/bin/raku ./tools/install-dist.p6 
Type check failed in binding to parameter '$meta-file'; expected IO::Path but got Any (Any)
  in sub MAIN at ./tools/install-dist.p6 line 59
  in block <unit> at ./tools/install-dist.p6 line 35

: ~/gitwrk/github.com/rakudo/rakudo $ /usr/bin/raku ./tools/install-dist.p6 
===SORRY!=== Error while compiling ./tools/install-dist.p6
Could not find CompUnit::Repository::Staging in:
    inst#/home/levitte/.raku
    inst#/usr/share/perl6/site
    inst#/usr/share/perl6/vendor
    inst#/usr/share/perl6/core
    ap#
    nqp#
    perl5#
at ./tools/install-dist.p6:35

dod38fr commented 3 years ago

Raku is broken on Debian/sid because raku is looking for modules in /usr/share/perl6 even though they are installed in /usr/lib/perl6.

This issue is tracked by Debian bug 969578 and https://github.com/rakudo/rakudo/issues/3537

On Debian, you should downgrade to testing version to get a working rakudo package.

All the best

levitte commented 3 years ago

Heh, I scrolled down enough on that report to see the path error just moments ago. Thanks, that explains it.

levitte commented 3 years ago

Another option is to observe that 2020.09 was release just a couple of hours ago :wink:. A quick uscan / uupdate of the moarvm, nqp and rakudo source packages, debuild -uc -us and dpkg -i gave a much better working installation on my machine.

crocket commented 2 years ago

I've used install-dist.raku successfully for gentoo linux.

dod38fr commented 2 years ago

Hello

After all, I've decided to ship pre-compiled files with Debian binary packages. So I'm now calling install-dist.raku at package build time.

This works for some package like raku-json-name which contains some files in /usr/lib/perl6/vendor/precomp/. On the other hand, raku-json-fast does not contain precomp files.

Here's a way to reproduce this problem with rakudo 2022.04 with JSON::Fast.

$ perl6 /usr/share/perl6/tools/install-dist.raku --to=/tmp/raku-test --for=vendor
 1 RMD: Requested for settings CORE.d
 1 RMD: Loading settings CORE.d
 1 RMD: Loading bytecode from CORE.d.setting.moarvm
 1 RMD: Loading bootstrap Perl6::BOOTSTRAP::v6d
 1 RMD: Requested for settings CORE.c
 1 RMD: Loading settings CORE.c
 1 RMD: Loading bytecode from CORE.c.setting.moarvm
 1 RMD: Loading bootstrap Perl6::BOOTSTRAP::v6c
 1 RMD: Settings CORE.c loaded
 1 RMD: Settings CORE.d loaded
 1 RMD: Attempting 'CompUnit::Repository::Staging' as a pragma
 1 RMD:   'CompUnit::Repository::Staging' is not a valid pragma
 1 RMD: Attempting to load 'CompUnit::Repository::Staging'
 1 RMD:   Late loading 'CompUnit::Repository::Staging'
 1 RMD: try-load source at /usr/lib/perl6/core/sources/70EBDA25F44EBFF8734F739F5779D64914083409
 1 RMD: Trying to load 70EBDA25F44EBFF8734F739F5779D64914083409
 1 RMD:   Loaded from /usr/lib/perl6/core/precomp
 1 RMD: Trying to load 70EBDA25F44EBFF8734F739F5779D64914083409.repo-id
 1 RMD:   Loaded from /home/domi/.raku/precomp
 1 RMD: Repo changed:
          0FD4AD54BC56EDA529A19DF2B8AA9FE9EB7B838E
          DCF27BF4000A187DC5AA00ADB4457A3D03FD1134
        Need to re-check dependencies.
 1 RMD: Loading precompiled
        /usr/lib/perl6/core/precomp/A234FA60C249CF61DEFEC1FD7D434DF2D0D597D5/70/70EBDA25F44EBFF8734F739F5779D64914083409
 1 RMD: Settings CORE.d already loaded
 1 RMD: Performing imports for 'CompUnit::Repository::Staging'
 1 RMD: Imports for 'CompUnit::Repository::Staging' done
 1 RMD: Settings CORE.d already loaded
 1 RMD: Precompiling /tmp/raku-test/sources/C1DA909DAD9BF713751A74EBF038C545A1EA6ECC into /home/domi/.raku/precomp/A234FA60C249CF61DEFEC1FD7D434DF2D0D597D5/C1/C1DA909DAD9BF713751A74EBF038C545A1EA6ECC.bc (   )
 2     RMD: Requested for settings CORE.d
 2     RMD: Loading settings CORE.d
 2     RMD: Loading bytecode from CORE.d.setting.moarvm
 2     RMD: Loading bootstrap Perl6::BOOTSTRAP::v6d
 2     RMD: Requested for settings CORE.c
 2     RMD: Loading settings CORE.c
 2     RMD: Loading bytecode from CORE.c.setting.moarvm
 2     RMD: Loading bootstrap Perl6::BOOTSTRAP::v6c
 2     RMD: Settings CORE.c loaded
 2     RMD: Settings CORE.d loaded
 2     RMD: Settings CORE.d already loaded
 2     RMD: Settings CORE.d already loaded
 2     RMD: Settings CORE.d already loaded
 2     RMD: Settings CORE.d already loaded
 2     RMD: Settings CORE.d already loaded
 2     RMD: Settings CORE.d already loaded
 2     RMD: Settings CORE.d already loaded
 2     RMD: Settings CORE.d already loaded
 2     RMD: Settings CORE.d already loaded
 2     RMD: Attempting 'nqp' as a pragma
 2     RMD: Successfully handled 'nqp' as a pragma
 1 RMD: Precompiled /tmp/raku-test/sources/C1DA909DAD9BF713751A74EBF038C545A1EA6ECC into /home/domi/.raku/precomp/A234FA60C249CF61DEFEC1FD7D434DF2D0D597D5/C1/C1DA909DAD9BF713751A74EBF038C545A1EA6ECC.bc
 1 RMD: Writing dependencies and byte code to /home/domi/.raku/precomp/A234FA60C249CF61DEFEC1FD7D434DF2D0D597D5/C1/C1DA909DAD9BF713751A74EBF038C545A1EA6ECC.tmp for source checksum: 3C60E5D5A51849AFCB01F30FFDD5E403E923D7C2
$ tree -a -s /tmp/raku-test/
[       4096]  /tmp/raku-test/
├── [       4096]  bin
├── [       4096]  dist
│   └── [        645]  5EB8EC94447A59889945785A70FE092DA9E66C78
├── [       4096]  precomp
├── [       4096]  resources
├── [       4096]  short
│   └── [       4096]  DF961EA9DD94E4FD7AAAB62E5362A67CC9E2D994
│       └── [        102]  5EB8EC94447A59889945785A70FE092DA9E66C78
└── [       4096]  sources
    └── [      35517]  C1DA909DAD9BF713751A74EBF038C545A1EA6ECC

7 directories, 3 files

The verbose output of install-dist show that the precomp file is written in ~/.raku/precomp.

How can I get this file in the directory specified with --to option ?

All the best

ugexe commented 2 years ago

Did this work on previous versions of rakudo? It sounds like this might be a regression

dod38fr commented 2 years ago

I can't say. That's the first time I've packaged JSON::Fast. For what it's worth, I get the same results with install-dist.p6.

ugexe commented 2 years ago

There have been a lot of changes across those areas of the rakudo code base in the last 6ish months, so it could very well not just be the original install script

dod38fr commented 2 years ago

Looks like I'm not the only one: https://github.com/rakudo/rakudo/issues/4907

Sorry, I forgot that this bug report is attached to zef. I will follow on rakudo bug tracker.

ugexe / zef